I've built a notification system with node-postgres and socket.io. The system works alright, however, I am getting an error on startup.
remaining connection slots are reserved for non-replication superuser connections
I suspect this is due to not releasing the client back to the pool.
Pool.connect()
.then(client => {
return client.query('LISTEN "new_notification"')
.then(result => {
client.on('notification', data => {
// Handle the notification
});
// Should release here
})
.catch(e => {
// Should release here
this.log(e.message, 'error');
});
})
.catch(e => {
this.log(e.message, 'error')
});
However even after adding client.release() in the locations where // Should release here are marked I still receive the error message. The notification still works, though.
When the server starts it creates a single HeyListen object in which the Pool above is created as well.
Now, the above error usually happens when the server starts and is flooded with connections. There are 6 sites that handle connections with users. Each user spawns a new connection to the socket.io server and when the disconnecting event is triggered via socket.io they are removed from a list of connected users. Each time a user connects they trigger a query against postgres if they have outstanding notifications. Here's the query object:
const conn = this.getConnectionType(pool);
conn.connect()
.then(client => {
return client.query(query_string, params)
.then(result => {
client.release();
callback(null, result);
})
.catch(error => {
client.release();
callback(error, null);
});
})
.catch(error => {
callback(error, null);
});
If I run Select * From pg_stat_activity on my psql server I see:
I thought using client.release() was supposed to remove these connections? Rerunning the query above shows different query results so some are being removed. Is this an issue of not having enough max_connections available? If so is it a good idea to bump that number up for my use case?
While experimenting with my own pub/sub client functionality for Node.js and PostgreSQL, I've seen that error when I run out of usable connections. If I remember correctly, the problem you're running into is that with the node-postgres package, a connection being used with LISTEN is considered active and in-use by the underlying connection pool until the client stops listening with UNLISTEN.
However, if you stop listening on a given connection, you'll no longer receive those useful notification events. Bit of a dilemma.
If possible, I'd advise you to look into setting up a smaller number of dedicated connections specifically for listening to various channels through PostgreSQL in each of your six applications, and then use separate connections to execute your additional queries. I don't know how feasible or performant this would be with your current traffic load, but in theory it should reduce the likelihood of using all available connections from the pool.
Out of curiosity, what was your rationale for having each user's connection LISTEN for notifications all on its own? It's possible to pass a payload to listeners through NOTIFY, so if you have some means of identifying an individual user, you could send your six apps a payload that declares which user a notification is for, and each of your six apps could forward that information to the correct user connection.
Related
I want to connect to a MongoDB replica set (only one instance to works with change streams) while being able to be notified of connection lost/reconnect.
I followed what described here:
const { MongoClient } = require("mongodb");
// Replace the following with your MongoDB deployment's connection
// string.
const uri =
"mongodb+srv://<clusterUrl>/?replicaSet=rs&writeConcern=majority";
const client = new MongoClient(uri);
// Replace <event name> with the name of the event you are subscribing to.
const eventName = "<event name>";
client.on(eventName, event => {
console.log(`received ${eventName}: ${JSON.stringify(event, null, 2)}`);
});
async function run() {
try {
await client.connect();
// Establish and verify connection
await client.db("admin").command({ ping: 1 });
console.log("Connected successfully");
} finally {
// Ensures that the client will close when you finish/error
await client.close();
}
}
run().catch(console.dir);
I tried subscribing to events:
serverOpening and works fine
serverClosed and I can't understand why but it does not work!!!
No "reconnect" event, any solution?
You are mixing monitoring connections and application connections. Unfortunately the documentation you referenced doesn't talk about this and doesn't document CMAP events so the confusion is understandable. See the Ruby driver docs for a more in-depth explanation of the events that drivers publish (including the Node driver).
Monitoring connections are established by the driver to figure out what server(s) exist in the deployment it was instructed to work with. One (or two depending on driver and server version) such connection is established per known server. You don't control when these connections are established. These connections are NOT used for operations you initiate (inserts/finds etc.). They are only used internally by the driver.
The events published for monitoring connections are server opened, server closed, server heartbeat - the ones listed here. You are going to get these events when the client is instantiated (assuming a spec-compliant client, which the Node one is not in it default configuration as you are using it) without any operations being issued like creating a change stream.
Application connections are established by the driver to satisfy the application's operations like finds and inserts. One of these would be needed for your change stream. The events relevant to these connections are CMAP ones and start with "Connection" or "Pool", e.g. ConnectionCreated. These connections aren't established until you issue an operation, unless you have the min pool size on the client set to a value greater than zero.
If you want to "monitor connections", you can subscribe to either category of events or both.
With that said, both types of connections are managed internally by the driver. You don't get a say in when they are created or destroyed (other than setting min pool size and idle timeouts). So if your goal is to have a working, continuously-running, resuming change stream, you don't need any of this and instead you should be using the proper change stream consumption patterns like the one described here in Ruby syntax although all spec-compliant drivers should provide the equivalent interface.
Lastly, there isn't a "reconnect" event defined in any driver specification. If you have a question specifically about this event you should reference the driver documentation where it is described and read that documentation carefully to ascertain the implemented behavior.
I'm working on implementing a distributed hash table using WebRTC and IndexedDB, and I ran into an issue I can't quite find the correct solution for. I have a simple application I've been using to explore WebRTC, and I'm successfully sending data over a data channel.
It would be impossible to keep a connection alive for every saved node, but it would also be unwieldy to re-signal every time. Normally for this you'd just keep an IP, port, and ID saved, but WebRTC needs more than that; below is the code I'm using to try to retrieve the connection, with information from the previous session stored in the "session" object:
function retrieve() {
console.log("Retrieving peer connection");
pc = new RTCPeerConnection(pcConfig);
pc.localStream = session.dataChannel;
pc.onicecandidate = () => {};
pc.onremovestream = (e) => console.log("Remote stream removed.");
if(initiator) {
pc.createOffer().then((sessionDescription) => {
pc.setLocalDescription(sessionDescription);
});
} else {
pc.createAnswer().then((sessionDescription) => {
pc.setLocalDescription(sessionDescription);
});
}
pc.setRemoteDescription(session.remoteDescription);
cands = session.candidates;
cands.map((cand) => {
pc.addIceCandidate(new RTCIceCandidate({
sdpMLineIndex: candidate.label,
candidate: candidate.candidate,
}));
});
}
This actually works, but it causes a few errors which I worry may indicate problems. The first of these occurs on both ends:
InvalidStateError: Cannot set remote answer in state stable
The second only occurs on the offering side, in my tests:
InvalidStateError: Cannot add ICE candidate when there is no remote SDP
For some reason, the data stream even works fine if I don't make an offer or answer at all, and just reinstate the candidates; the error in that case is the latter error on both sides.
What's going on here? Do I need to worry about it, or can I forge on? And as a side question, is this a silly way of solving this problem, with an obvious missed alternative?
Edit: I've now also tried a bunch of ways of reordering these steps, according to these errors, and haven't gotten much in the way of results; however, this sort of approach does allow real connections to reconnect and doesn't cause problems with the signaling even when I change networks on my phone, so I'll carry on testing the limits of this approach. I'd still like to know exactly what causes these errors, though, since they don't necessarily seem to indicate what they appear to indicate.
I have a simple Node.JS HTTPS web server using socket.io. Users can send to the server lots of different information. In a few seconds, you may have the web server receive multiple one line objects to the web server from one user (i.e. { code: '124' }, but there may be multiple users doing this at once. Then the web server returns information to all users in that socket.io room.
As this information comes in, the web server intermittently saves this data to a simple MySQL database although I am limiting saves so that there aren't multiple small MySQL writes per second or anything like that.
My thought process is that as more users log onto and connect to the web server, the code may become blocked as Node.JS is single threaded, and this may cause a lag in information getting back to the users in real time, or it may cause problems with the latest data being saved to the database. I was thinking of doing something like this so that the database updates is handled by a separate web worker -
socket.on('data', function(msg) {
try {
const newWorker = new Worker('./src/worker.js');
newWorker.on('message', function(result) {
io.to(`${socketID}`).emit('newData', result);
});
newWorker.on('error', (err) => {
io.to(`${socketID}`).emit('newData', { error: err });
console.dir(err);
});
newWorker.postMessage(msg);
} catch(e) {
console.log(e);
io.to(`${socketID}`).emit('criticalError', "We ran into an error - try refreshing");
}
});
I know however, that async processes can be run on multiple threads in the background of Node.JS and Node.JS is generally quite performant.
My question is, given that there may be multiple database writes happening simultaneously, as well as multiple pieces of information coming in, which then need to be sent back to users in any given second, does it make sense for me to use web workers to make this kind of process mulithreaded? Or is Node.JS capable enough to handle all of this in the background on multiple threads without me needing to worry?
I'm running a node.js server. On my website I use different URLs for different pages. For instance:
mydomain.com/ -- Index
mydomain.com/register -- Register
mydomain.com/profile -- Profile
I am using socket.io to send chat messages and notifications to the client. However, whenever the user switches page or performs a POST-request the socket connection is disconnected and then reconnected.
I'm wondering what the complexity of socket.io's connect/disconnect functions are and wether it is durable that all clients reconnect their sockets each time they perform an action on my website? I've looked at the documentation for socket.io without finding the answer.
That's what I have found, but I would guess it would really depend on your code. If each page connects to the channel then the connection will be reestablished. I am also in the habit of storing messages on the server side to reestablish state - something to this effect:
const events = []
const onSomeEvent = function(myEvent) {
events.push(myEvent)
socket.emit("onSomeEvent", myEvent)
}
socket.on("connect", function(){
events.forEach(function(myEvent){
socket.emit("onSomeEvent", myEvent)
})
})
Today, I integrated Redis into my node.js application and am using it as a session store. Basically, upon successful authentication, I store the corresponding user object in Redis.
When I receive http requests after authentication, I attempt to retrieve the user object from Redis using a hash. If the retrieval was successful, that means the user is logged in and the request can be fulfilled.
The act of storing the user object in Redis and the retrieval happen in two different files, so I have one Redis client in each file.
Question 1:
Is it ok having two Redis clients, one in each file? Or should I instantiate only one client and use it across all areas of the application?
Question 2:
Does the node-redis library provide a method to show a list of connected clients? If it does, I will be able to iterate through the list, and call client.quit() for each of them when the server is shutting down.
By the way, this is how I'm implementing the "graceful shutdown" of the server:
//Gracefully shutdown and perform clean-up when kill signal is received
process.on('SIGINT', cleanup);
process.on('SIGTERM', cleanup);
function cleanup() {
server.stop(function() {
//todo: quit all connected redis clients
console.log('Server stopped.');
//exit the process
process.exit();
});
};
In terms of design and performance, it's best to create one client and use it across your application. This is pretty easy to do in node. I'm assuming you're using the redis npm package.
First, create a file named redis.js with the following contents:
const redis = require('redis');
const RedisClient = (function() {
return redis.createClient();
})();
module.exports = RedisClient
Then, say in a file set.js, you would use it as so:
const client = require('./redis');
client.set('key', 'value');
Then, in your index.js file, you can import it and close the connection on exit:
const client = require('./redis');
process.on('SIGINT', cleanup);
process.on('SIGTERM', cleanup);
function cleanup() {
client.quit(function() {
console.log('Redis client stopped.');
server.stop(function() {
console.log('Server stopped.');
process.exit();
});
});
};
Using multiple connections may be required by how the application uses Redis.
For instance, as soon as a connection is used the purpose of listening to a pub/sub channel, then it can only be used for this and nothing else. Per the documentation on SUBSCRIBE:
Once the client enters the subscribed state it is not supposed to issue any other commands, except for additional SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE and PUNSUBSCRIBE commands.
So if your application needs to subscribe to channels and use Redis as general value cache, then it needs two clients at a minimum: one for subscribing to channels and one for using Redis as a cache.
There are also Redis commands that are blocking like BLPOP. A busy web server normally replies to multiple requests at once. Suppose that for answering request A the server uses its Redis client to issue a blocking command. Then request B comes and the server needs to answer Redis with a non-blocking command but the client is still waiting for the blocking command issued for request A to finish. Now the response to request B is delayed by another request. This can be avoided by using a different client for the second request.
If you do not use any of the facilities that require more than one connection, then you can and should use just one connection.
If the way you use Redis is such that you need more than one connection, and you just need a list of connections but no sophisticated connection management, you could just create your own factory function: it would call redis.createClient() and save the client before returning it. Then at shutdown time, you could go over the list of saved clients and close them. Unfortunately, node-redis does not provide such functionality built-in.
If you need more sophisticated client management than the factory function described above, then the typical way to manage the multiple connections created is to use a connection pool but node-redis does not provide one. I usually access Redis through Python code so I don't have a recommendation for Node.js libraries, but an npm search shows quite a few candidates.