We Keep Coding

Node.js - Fastify: Connection closed during sleep (setTimeout)

I am having a problem with a Node.js server using Fastify.
At some point during the execution of a request, the server seems to be closing the connection and the client is getting a socket hang up error.
The logic in the server is:
Fastify client calling a service.
Service sending HTTP request using Axios to get certain information. The service implements a retry mechanism and after every retry, it waits for 15 seconds to make sure the information is available.
The code is as follows:
Fastify server:
fastify.post('/request', async (request, reply) => {
try {
const result = await service.performOperation(request.body);
return result;
} catch(error) {
console.error('Error during operation: %s', error.toString());
throw error;
}
})
fastify.addHook('onError', (request, reply, error, done) => {
console.error('onError hook: %o', error);
done();
})
Service:
async function performOperation(request) {
let attempt = 0;
let latestErrorMessage;
while(attempt++ < 5) {
try {
await waitBeforeAttempt();
return await getInfoFromServer(request);
} catch (error) {
latestErrorMessage = getErrorMessage(error);
if (attempt < 5) {
console.log(`Re-attempting after error: ${latestErrorMessage}`);
}
}
}
throw new Error(`Error after 5 attempts. Last error: ${latestErrorMessage}`);
}
function waitBeforeAttempt() {
return new Promise(resolve => setTimeout(resolve, 15000));
}
async function getInfoFromServer(request) {
const response = await axios.post('http://localhost:3000/service', request, {timeout: 120000});
return response.data.toString();
}
The problem is that the server seems to be closing the connection.
According to the logs, this is happening after waiting for 15 seconds and before the call via Axios, before finishing the first attempt.
You can see in the logs that after closing the connection, the logic continues and finishes with all the attempts with no problems whatsoever.
There is nothing in the logs as to why the connection is closed, not even from the Fastify onError hook declared.
Nothing from Axios either. I guess if there were any timeouts that would throw an exception and be logged.
Important note
Noted that connections are not dropped if I change the waitBeforeAttempt implementation to implement a busy wait instead of setTimeout ie:
function waitBeforeAttempt() {
const start = new Date();
let now;
while (true) {
now = new Date();
if (now - start >= 15000) {
break;
}
}
}
Is there anything I'm doing wrong that is causing the connections to be dropped? Perhaps the 15 seconds wait is too high? I have other setTimeout in the code via Puppetter (same implementation as mine) that don't seem to be causing the problem.

Just answering my own question. The problem turned out to be nothing related to the wait or timeouts.
This wasn't happening when the Node.js service was running locally, only happening intermittently when running on Kubernets + Nginx.
Nginx was just restarting without any apparent reason.
Nginx has been updated and the issue is not showing anymore.

Redis (ioredis) - Unable to catch connection error in order to handle them gracefully

I'm trying to gracefully handle redis errors, in order to bypass the error and do something else instead, instead of crashing my app.
But so far, I couldn't just catch the exception thrown by ioredis, which bypasses my try/catch and terminates the current process. This current behaviour doesn't allow me to gracefully handle the error and in order to fetch the data from an alternative system (instead of redis).
import { createLogger } from '#unly/utils-simple-logger';
import Redis from 'ioredis';
import epsagon from './epsagon';
const logger = createLogger({
label: 'Redis client',
});
/**
* Creates a redis client
*
* #param url Url of the redis client, must contain the port number and be of the form "localhost:6379"
* #param password Password of the redis client
* #param maxRetriesPerRequest By default, all pending commands will be flushed with an error every 20 retry attempts.
* That makes sure commands won't wait forever when the connection is down.
* Set to null to disable this behavior, and every command will wait forever until the connection is alive again.
* #return {Redis}
*/
export const getClient = (url = process.env.REDIS_URL, password = process.env.REDIS_PASSWORD, maxRetriesPerRequest = 20) => {
const client = new Redis(`redis://${url}`, {
password,
showFriendlyErrorStack: true, // See https://github.com/luin/ioredis#error-handling
lazyConnect: true, // XXX Don't attempt to connect when initializing the client, in order to properly handle connection failure on a use-case basis
maxRetriesPerRequest,
});
client.on('connect', function () {
logger.info('Connected to redis instance');
});
client.on('ready', function () {
logger.info('Redis instance is ready (data loaded from disk)');
});
// Handles redis connection temporarily going down without app crashing
// If an error is handled here, then redis will attempt to retry the request based on maxRetriesPerRequest
client.on('error', function (e) {
logger.error(`Error connecting to redis: "${e}"`);
epsagon.setError(e);
if (e.message === 'ERR invalid password') {
logger.error(`Fatal error occurred "${e.message}". Stopping server.`);
throw e; // Fatal error, don't attempt to fix
}
});
return client;
};
I'm simulating a bad password/url in order to see how redis reacts when misconfigured. I've set lazyConnect to true in order to handle errors on the caller.
But, when I define the url as localhoste:6379 (instead of localhost:6379), I get the following error:
server 2019-08-10T19:44:00.926Z [Redis client] error: Error connecting to redis: "Error: getaddrinfo ENOTFOUND localhoste localhoste:6379"
(x 20)
server 2019-08-10T19:44:11.450Z [Read cache] error: Reached the max retries per request limit (which is 20). Refer to "maxRetriesPerRequest" option for details.
Here is my code:
// Fetch a potential query result for the given query, if it exists in the cache already
let cachedItem;
try {
cachedItem = await redisClient.get(queryString); // This emit an error on the redis client, because it fails to connect (that's intended, to test the behaviour)
} catch (e) {
logger.error(e); // It never goes there, as the error isn't "thrown", but rather "emitted" and handled by redis its own way
epsagon.setError(e);
}
// If the query is cached, return the results from the cache
if (cachedItem) {
// return item
} else {} // fetch from another endpoint (fallback backup)
My understanding is that redis errors are handled through client.emit('error', error), which is async and the callee doesn't throw an error, which doesn't allow the caller to handle errors using try/catch.
Should redis errors be handled in a very particular way? Isn't it possible to catch them as we usually do with most errors?
Also, it seems redis retries 20 times to connect (by default) before throwing a fatal exception (process is stopped). But I'd like to handle any exception and deal with it my own way.
I've tested the redis client behaviour by providing bad connection data, which makes it impossible to connect as there is no redis instance available at that url, my goal is to ultimately catch all kinds of redis errors and handle them gracefully.

Connection errors are reported as an error event on the client Redis object.
According to the "Auto-reconnect" section of the docs, ioredis will automatically try to reconnect when the connection to Redis is lost (or, presumably, unable to be established in the first place). Only after maxRetriesPerRequest attempts will the pending commands "be flushed with an error", i.e. get to the catch here:
try {
cachedItem = await redisClient.get(queryString); // This emit an error on the redis client, because it fails to connect (that's intended, to test the behaviour)
} catch (e) {
logger.error(e); // It never goes there, as the error isn't "thrown", but rather "emitted" and handled by redis its own way
epsagon.setError(e);
}
Since you stop your program on the first error:
client.on('error', function (e) {
// ...
if (e.message === 'ERR invalid password') {
logger.error(`Fatal error occurred "${e.message}". Stopping server.`);
throw e; // Fatal error, don't attempt to fix
...the retries and the subsequent "flushing with an error" never get the chance to run.
Ignore the errors in client.on('error', and you should get the error returned from await redisClient.get().

Here is what my team has done with IORedis in a TypeScript project:
let redis;
const redisConfig: Redis.RedisOptions = {
port: parseInt(process.env.REDIS_PORT, 10),
host: process.env.REDIS_HOST,
autoResubscribe: false,
lazyConnect: true,
maxRetriesPerRequest: 0, // <-- this seems to prevent retries and allow for try/catch
};
try {
redis = new Redis(redisConfig);
const infoString = await redis.info();
console.log(infoString)
} catch (err) {
console.log(chalk.red('Redis Connection Failure '.padEnd(80, 'X')));
console.log(err);
console.log(chalk.red(' Redis Connection Failure'.padStart(80, 'X')));
// do nothing
} finally {
await redis.disconnect();
}

when using fetch, axios, etc.`catch` doesn't handle the error, instead your app crashes

To make sure that our request will be successful, first, we check the internet connection then send our request.
like this:
NetInfo.isConnected.fetch().then(async isConnected=> {
if(isConnected){
try {
let result = await fetch(MY_REMOTE_SERVER);
console.log("result: ", result)
} catch (error) {
console.error("error: ", error);
}
}
else ToastAndroid.show('No internet', ToastAndroid.SHORT);
});
Everything was fine, until I faced this issue: consider a situation in which access to a server for some countries is blocked.
So, although the internet connection is ok, each time I was getting network request failed error.
I couldn't find the problem because expected the catch to print the error, but my app was just crashing.
Now that I know the reason, I don't know how to solve it.
For example, when the connection can't be made I want to alert the user to use a VPN or leave the app because they are in an embargoed country!
On the other hand, what is the point of catch!? if it doesn't catch the error!
thanks.

Actually this is a mistake on our side, react-native will crash as it encounters console.error.
so by changing the above code to this version you will get rid of the red screen:
NetInfo.isConnected.fetch().then(async isConnected=> {
if(isConnected){
try {
let result = await fetch(MY_REMOTE_SERVER);
console.log("result: ", result)
} catch (error) {
// use "log" instead of "error"
console.log("error: ", error);
// or you may want to show a toast on error like
oastAndroid.show('No internet', ToastAndroid.SHORT)
}
}else ToastAndroid.show('No internet', ToastAndroid.SHORT);
});

Promise reject paused with an exception yet no error is log

Below is a Nodejs piece of code. I'm querying a Mongodb using mongoose. This piece of code gets paused at "Line A" with message "Paused on Exception". console shows no errors. Meanwhile this seem to happen only when I run in VS Code. Running the app from console raises no exception.
I'm inclined to say this a VS Code issue. Has anyone seen/faced same or something similar?
Board.findOne({ boardId: id }, function (err, data) {
if (!err) {
data ? resolve(data.children) : reject(data); //Line A
// line above pauses execution with this message: "Paused on Exception"
// No error logged in console. And this seem to happen only in VS code.
}else{
reject(err);
}
}

Mongoose already supports promises, so you can rewrite your code to this:
return Board.findOne({ boardId: id }).then(function(data) {
if (! data) throw new Error('data empty');
return data.children;
}).catch(function(err) {
console.log(err);
});
Perhaps this also solves your VS Code issue.

how to prevent node.js application from terminated for unhandled exception in setInterval?

Node.js application terminated for unhanded exception in setInterval. I try to fix it by process.on('uncaughtException', ..) and domain approaches (see below codes). Application still was terminated although the exception was handled.
function f () {
throw Error('have error')
}
process.on('uncaughtException', function(err){
console.log("process.on uncaughtException")
})
var d = require('domain').create();
d.on('error',function(err){
console.log("domain.on error")
})
d.run(function(){
setInterval(f, 1000)
})
// program terminated and output is: domain.on error

Program terminated because there was nothing else to process after setInterval(). In nodejs doc example, it creates the the server and bind the port to it. That's what keeps the application running. Here is the example from the doc:
var d = require('domain').create();
d.on('error', function(er) {
// The error won't crash the process, but what it does is worse!
// Though we've prevented abrupt process restarting, we are leaking
// resources like crazy if this ever happens.
// This is no better than process.on('uncaughtException')!
console.log('error, but oh well', er.message);
});
d.run(function() {
require('http').createServer(function(req, res) {
setInterval(f, 1000);
}).listen(8888);
});
Then if you point your browser to localhost:8888, the app is not terminated