We Keep Coding

Redis (ioredis) - Unable to catch connection error in order to handle them gracefully

I'm trying to gracefully handle redis errors, in order to bypass the error and do something else instead, instead of crashing my app.
But so far, I couldn't just catch the exception thrown by ioredis, which bypasses my try/catch and terminates the current process. This current behaviour doesn't allow me to gracefully handle the error and in order to fetch the data from an alternative system (instead of redis).
import { createLogger } from '#unly/utils-simple-logger';
import Redis from 'ioredis';
import epsagon from './epsagon';
const logger = createLogger({
label: 'Redis client',
});
/**
* Creates a redis client
*
* #param url Url of the redis client, must contain the port number and be of the form "localhost:6379"
* #param password Password of the redis client
* #param maxRetriesPerRequest By default, all pending commands will be flushed with an error every 20 retry attempts.
* That makes sure commands won't wait forever when the connection is down.
* Set to null to disable this behavior, and every command will wait forever until the connection is alive again.
* #return {Redis}
*/
export const getClient = (url = process.env.REDIS_URL, password = process.env.REDIS_PASSWORD, maxRetriesPerRequest = 20) => {
const client = new Redis(`redis://${url}`, {
password,
showFriendlyErrorStack: true, // See https://github.com/luin/ioredis#error-handling
lazyConnect: true, // XXX Don't attempt to connect when initializing the client, in order to properly handle connection failure on a use-case basis
maxRetriesPerRequest,
});
client.on('connect', function () {
logger.info('Connected to redis instance');
});
client.on('ready', function () {
logger.info('Redis instance is ready (data loaded from disk)');
});
// Handles redis connection temporarily going down without app crashing
// If an error is handled here, then redis will attempt to retry the request based on maxRetriesPerRequest
client.on('error', function (e) {
logger.error(`Error connecting to redis: "${e}"`);
epsagon.setError(e);
if (e.message === 'ERR invalid password') {
logger.error(`Fatal error occurred "${e.message}". Stopping server.`);
throw e; // Fatal error, don't attempt to fix
}
});
return client;
};
I'm simulating a bad password/url in order to see how redis reacts when misconfigured. I've set lazyConnect to true in order to handle errors on the caller.
But, when I define the url as localhoste:6379 (instead of localhost:6379), I get the following error:
server 2019-08-10T19:44:00.926Z [Redis client] error: Error connecting to redis: "Error: getaddrinfo ENOTFOUND localhoste localhoste:6379"
(x 20)
server 2019-08-10T19:44:11.450Z [Read cache] error: Reached the max retries per request limit (which is 20). Refer to "maxRetriesPerRequest" option for details.
Here is my code:
// Fetch a potential query result for the given query, if it exists in the cache already
let cachedItem;
try {
cachedItem = await redisClient.get(queryString); // This emit an error on the redis client, because it fails to connect (that's intended, to test the behaviour)
} catch (e) {
logger.error(e); // It never goes there, as the error isn't "thrown", but rather "emitted" and handled by redis its own way
epsagon.setError(e);
}
// If the query is cached, return the results from the cache
if (cachedItem) {
// return item
} else {} // fetch from another endpoint (fallback backup)
My understanding is that redis errors are handled through client.emit('error', error), which is async and the callee doesn't throw an error, which doesn't allow the caller to handle errors using try/catch.
Should redis errors be handled in a very particular way? Isn't it possible to catch them as we usually do with most errors?
Also, it seems redis retries 20 times to connect (by default) before throwing a fatal exception (process is stopped). But I'd like to handle any exception and deal with it my own way.
I've tested the redis client behaviour by providing bad connection data, which makes it impossible to connect as there is no redis instance available at that url, my goal is to ultimately catch all kinds of redis errors and handle them gracefully.

Connection errors are reported as an error event on the client Redis object.
According to the "Auto-reconnect" section of the docs, ioredis will automatically try to reconnect when the connection to Redis is lost (or, presumably, unable to be established in the first place). Only after maxRetriesPerRequest attempts will the pending commands "be flushed with an error", i.e. get to the catch here:
try {
cachedItem = await redisClient.get(queryString); // This emit an error on the redis client, because it fails to connect (that's intended, to test the behaviour)
} catch (e) {
logger.error(e); // It never goes there, as the error isn't "thrown", but rather "emitted" and handled by redis its own way
epsagon.setError(e);
}
Since you stop your program on the first error:
client.on('error', function (e) {
// ...
if (e.message === 'ERR invalid password') {
logger.error(`Fatal error occurred "${e.message}". Stopping server.`);
throw e; // Fatal error, don't attempt to fix
...the retries and the subsequent "flushing with an error" never get the chance to run.
Ignore the errors in client.on('error', and you should get the error returned from await redisClient.get().

Here is what my team has done with IORedis in a TypeScript project:
let redis;
const redisConfig: Redis.RedisOptions = {
port: parseInt(process.env.REDIS_PORT, 10),
host: process.env.REDIS_HOST,
autoResubscribe: false,
lazyConnect: true,
maxRetriesPerRequest: 0, // <-- this seems to prevent retries and allow for try/catch
};
try {
redis = new Redis(redisConfig);
const infoString = await redis.info();
console.log(infoString)
} catch (err) {
console.log(chalk.red('Redis Connection Failure '.padEnd(80, 'X')));
console.log(err);
console.log(chalk.red(' Redis Connection Failure'.padStart(80, 'X')));
// do nothing
} finally {
await redis.disconnect();
}

Suppress "WebSocket connection to 'xyz' failed"

I've written a web application that uses web-sockets. The idea is that my app tries to auto-connect to recently connected to hosts when it starts up. If it can't establish a connection to any of them, then it directs the user to the connection part and asks them to establish a connection manually.
All of this works. In summary, I try each known host in order, and if 200ms later it hasn't connected (`readyState != 1), it tries the next one. All these hosts should be on the LAN so 200ms works pretty reliably. If the last one on the list fails too, then the web opens up a modal directing the user to manually type in a host.
The problem is, by trying to auto-connect, I have to create websockets to my attempted hosts, which outputs error messages like the following to the console:
WebSocket connection to 'ws://lightmate:8080/' failed: Error in
connection establishment: net::ERR_NAME_NOT_RESOLVED
WebSocket connection to 'ws://localhost:8080/' failed: Error in
connection establishment: net::ERR_CONNECTION_REFUSED
While not a fatal flaw by any means, it's unsightly and gets in the way of my debugging.
I've tried to remove it by surrounding the calls to new WebSocket(address) with a try/catch block, and the errors still get through, and I've also tried to set an onerror handler, hoping that would suppress the error messages. Nothing's worked.
connect: function(){
var fulladdr = completeServerAddress(address);
try {
connection = new WebSocket(fulladdr);
connection.suppressErrorsBecauseOfAutoConnection = suppressErrorsBecauseOfAutoConnection; //Store this module-scoped variable in connection, so if the module changes suppression state, this connection won't.
} catch (e){
//Make sure we don't try to send anything down this dead websocket
connection = false;
return false;
}
connection.binaryType = "arraybuffer";
connection.onerror = function(){
if (connection !== false && !connection.suppressErrorsBecauseOfAutoConnection){
Announce.announceMessage("Connection failed with server");
}
connection = false;
};
connection.onmessage = function(m){
rxMessage(ConnectionProtocol.BaseMessage.parseChunk(m.data));
};
connection.onclose = function(){
hooks.swing("disconnected", "", 0);
if (connection !== false && !connection.suppressErrorsBecauseOfAutoConnection){
Announce.announceMessage("Connection lost with server");
}
};
connection.onopen = function(){
sendMessages(ConnectionProtocol.HandshakeMessage.create(name, sources, sinks));
while (idlingmessages.length){
websocketConnection.send(idlingmessages.splice(0,1)[0]);
}
hooks.swing("connected", "", 0);
};
},
Dupl Disclaimer:
This question is similar to this StackOverflow question, but that question is out of date by a year, and the consensus there was "you can't". I'm hoping things have changed since then.

There is no way to trap that error message, which occurs asynchronously to the code where the WebSocket object is created.
More details here: Javascript doesn't catch error in WebSocket instantiation

Node.js domain cluster worker disconnect

Looking at the example given at the nodejs domain doc page: http://nodejs.org/api/domain.html, the recommended way to restart a worker using cluster is to call first disconnect in the worker part, and listen to the disconnect event in the master part. However, if you just copy/paste the example given, you will notice that the disconnect() call does not shutdown the current worker:
What happens here is:
try {
var killtimer = setTimeout(function() {
process.exit(1);
}, 30000);
killtimer.unref();
server.close();
cluster.worker.disconnect();
res.statusCode = 500;
res.setHeader('content-type', 'text/plain');
res.end('Oops, there was a problem!\n');
} catch (er2) {
console.error('Error sending 500!', er2.stack);
}
I do a get request at /error
A timer is started: in 30s the process will be killed if not already
The http server is shut down
The worker is disconnected (but still alive)
The 500 page is displayed
I do a second get request at error (before 30s)
New timer started
Server is already closed => throw an error
The error is catched in the "catch" block and no result is sent back to the client, so on the client side, the page is waiting without any message.
In my opinion, it would be better to just kill the worker, and listen to the 'exit' event on the master part to fork again. This way, the 500 error is always sent during an error:
try {
var killtimer = setTimeout(function() {
process.exit(1);
}, 30000);
killtimer.unref();
server.close();
res.statusCode = 500;
res.setHeader('content-type', 'text/plain');
res.end('Oops, there was a problem!\n');
cluster.worker.kill();
} catch (er2) {
console.error('Error sending 500!', er2);
}
I'm not sure about the down side effects using kill instead of disconnect, but it seems disconnect is waiting the server to close, however it seems this is not working (at least not like it should)
I just would like some feedbacks about this. There could be a good reason this example is written this way that I've missed.
Thanks
EDIT:
I've just checked with curl, and it works well.
However I was previously testing with Chrome, and it seems that after sending back the 500 response, chrome does a second request BEFORE the server actually ends to close.
In this case, the server is closing and not closed (which means the worker is also disconnecting without being disconnected), causing the second request to be handled by the same worker as before so:
It prevents the server to finish to close
The second server.close(); line being evaluated, it triggers an exception because the server is not closed.
All following requests will trigger the same exception until the killtimer callback is called.

I figured it out, actually when the server is closing and receives a request at the same time, it stops its closing process.
So he still accepts connection, but cannot be closed anymore.
Even without cluster, this simple example illustrates this:
var PORT = 8080;
var domain = require('domain');
var server = require('http').createServer(function(req, res) {
var d = domain.create();
d.on('error', function(er) {
try {
var killtimer = setTimeout(function() {
process.exit(1);
}, 30000);
killtimer.unref();
console.log('Trying to close the server');
server.close(function() {
console.log('server is closed!');
});
console.log('The server should not now accepts new requests, it should be in "closing state"');
res.statusCode = 500;
res.setHeader('content-type', 'text/plain');
res.end('Oops, there was a problem!\n');
} catch (er2) {
console.error('Error sending 500!', er2);
}
});
d.add(req);
d.add(res);
d.run(function() {
console.log('New request at: %s', req.url);
// error
setTimeout(function() {
flerb.bark();
});
});
});
server.listen(PORT);
Just run:
curl http://127.0.0.1:8080/ http://127.0.0.1:8080/
Output:
New request at: /
Trying to close the server
The server should not now accepts new requests, it should be in "closing state"
New request at: /
Trying to close the server
Error sending 500! [Error: Not running]
Now single request:
curl http://127.0.0.1:8080/
Output:
New request at: /
Trying to close the server
The server should not now accepts new requests, it should be in "closing state"
server is closed!
So with chrome doing 1 more request for the favicon for example, the server is not able to shutdown.
For now I'll keep using worker.kill() which makes the worker not to wait for the server to stops.

I ran into the same problem around 6 months ago, sadly don't have any code to demonstrate as it was from my previous job. I solved it by explicitly sending a message to the worker and calling disconnect at the same time. Disconnect prevents the worker from taking on new work and in my case as i was tracking all work that the worker was doing (it was for an upload service that had long running uploads) i was able to wait until all of them are finished and then exit with 0.

How to catch socket.io errors and prevent them from showing up in the console?

I'm running socket.io on node.js and the socket.io client on an Apache website. If I don't start the node.js server and load the client page, the error event is triggered with an empty error message, which results in the following console output:
GET http://example.com:1337/socket.io/1/?t=1359731838906 socket.io.js:1659
Socket.handshake socket.io.js:1659
Socket.connect socket.io.js:1699
Socket socket.io.js:1551
io.connect socket.io.js:94
(anonymous function) general.js:7
(anonymous function)
What can I do to stop this error being written to the console?

You can not "catch" these errors in the sense that they are not appearing in the console but you can still act upon them by listening for 'connect_error', 'connect_failed' and 'disconnect' of the socket and then either handling them directly or redirecting them to a custom function like handleErrors():
const socket = io.connect(url, { reconnection: false })
socket.on('update', data => console.log(data))
socket.on('connect_error', err => handleErrors(err))
socket.on('connect_failed', err => handleErrors(err))
socket.on('disconnect', err => handleErrors(err))

The only way to hide that error is by never calling io.connect in the first place – which of course would mean your app's socket functions wouldn't work even if the server is up.
It's important to understand that the error message you're seeing is neither something placed there by socket.io itself (via console.error()) nor is it an uncaught JS Exception.
The error message is placed in your console by the browser's XHR object itself. It's telling you that a XHR request has failed (since your server isn't running). The stack trace is telling you what code initiated the XHR request; it isn't a trace of an actual Exception.
Since socket.io must make a request to the server, there's no way it (or you) could prevent that error message from appearing in the console if the server isn't responding.

try
socket.on('connect_failed', function(){
console.log('Connection Failed');
});
Not tested this. Found it here Node.js socket.io-client connect_failed / connect_error event

If you're using Socket.io v4 and need to capture middleware errors, try
// client-side
socket.on("connect_error", (err) => {
console.log(err.message); // prints the message associated with the error
});
source https://socket.io/docs/v4/middlewares/#handling-middleware-error

nodejs + WebSockets - reject the connection with a message

I want to provide a meaningful error to the client when too many users are connected or when they're connecting from an unsupported domain, so...
I wrote some WebSocket server code:
var http = require('http');
var httpServer = http.createServer(function (request, response)
{
// i see this if i hit http://localhost:8001/
response.end('go away');
});
httpServer.listen(8001);
// https://github.com/Worlize/WebSocket-Node/wiki/Documentation
var webSocket = require('websocket');
var webSocketServer = new webSocket.server({ 'httpServer': httpServer });
webSocketServer.on('request', function (request)
{
var connection = request.reject(102, 'gtfo');
});
And some WebSocket client code:
var connection = new WebSocket('ws://127.0.0.1:8001');
connection.onopen = function (openEvent)
{
alert('onopen');
console.log(openEvent);
};
connection.onclose = function (closeEvent)
{
alert('onclose');
console.log(closeEvent);
}
connection.onerror = function (errorEvent)
{
alert('onerror');
console.log(errorEvent);
};
connection.onmessage = function (messageEvent)
{
alert('onmessage');
console.log(messageEvent);
};
All I get is alert('onclose'); with a CloseEvent object logged to the console without any status code or message that I can find. When I connect via ws://localhost:8001 the httpServer callback doesn't come into play, so I can't catch it there. The RFC suggests I should be able to send any status code other than 101 when there's a problem, but Chrome throws an error in the console Unexpected response code: 102. If I call request.reject(101, 'gtfo'), implying it was successful I get a handshake error, as I'd expect.
Not really sure what else I can do. Is it just not possible right now to get the server response in Chrome's WebSocket implementation?
ETA: Here's a really nasty hack in the mean time, I hope that's not what I have to end up doing.
var connection = request.accept(null, request.origin);
connection.sendUTF('gtfo');
connection.close();

I'm the author of WebSocket-Node and I've also posted this response to the corresponding issue on GitHub: https://github.com/Worlize/WebSocket-Node/issues/46
Unfortunately, the WebSocket protocol does not provide any specific mechanism for providing a close code or reason at this stage when rejecting a client connection. The rejection is in the form of an HTTP response with an HTTP status of something like 40x or 50x. The spec allows for this but does not define a specific way that the client should attempt to divine any specific error messaging from such a response.
In reality, connections should be rejected at this stage only when you are rejecting a user from a disallowed origin (i.e. someone from another website is trying to connect users to your websocket server without permission) or when a user otherwise does not have permission to connect (i.e. they are not logged in). The latter case should be handled by other code on your site: a user should not be able to attempt to connect the websocket connection if they are not logged in.
The code and reason that WebSocket-Node allow you to specify here are an HTTP Status code (e.g. 404, 500, etc.) and a reason to include as a non-standard "X-WebSocket-Reject-Reason" HTTP header in the response. It is mostly useful when analyzing the connection with a packet sniffer, such as WireShark. No browser has any facility for providing rejection codes or reasons to the client-side JavaScript code when a connection is rejected in this way, because it's not provided for in the WebSocket specification.