Socket.io disconnected unexpectedly - javascript

I have node.js service and angular client using socket.io to transport some message during long time http request.
Service:
export const socketArray: SocketIO.Socket[] = [];
export let socketMapping: {[socketId: string]: number} = {};
const socketRegister: hapi.Plugin<any> = {
register: (server) => {
const io: SocketIO.Server = socket(server.listener);
// Whenever a session connected to socket, create a socket object and add it to socket array
io.on("connection", (socket) => {
console.log(`socket ${socket.id} connected`);
logger.info(`socket ${socket.id} connected`);
// Only put socket object into array if init message received
socket.on("init", msg => {
logger.info(`socket ${socket.id} initialized`);
socketArray.push(socket);
socketMapping[socket.id] = msg;
});
// Remove socket object from socket array when disconnected
socket.on("disconnect", (reason) => {
console.log(`socket ${socket.id} disconnected because: ${reason}`)
logger.info(`socket ${socket.id} disconnected because: ${reason}`);
for(let i = 0; i < socketArray.length; i ++) {
if(socketArray[i] === socket) {
socketArray.splice(i, 1);
return;
}
}
});
});
},
name: "socketRegister",
version: "1.0"
}
export const socketSender = async (socketId: string, channel: string, content: SocketMessage) => {
try {
// Add message to db here
// await storeMessage(socketMapping[socketId], content);
// Find corresponding socket and send message
logger.info(`trying sending message to ${socketId}`);
for (let i = 0; i < socketArray.length; i ++) {
if (socketArray[i].id === socketId) {
socketArray[i].emit(channel, JSON.stringify(content));
logger.info(`socket ${socketId} send message to ${channel}`);
if (content.isFinal == true) {
// TODO: delete all messages of the process if isFinal is true
await deleteProcess(content.processId);
}
return;
}
}
} catch (err) {
logger.error("Socket sender error: ", err.message);
}
};
Client:
connectSocket() {
if (!this.socket) {
try {
this.socket = io(socketUrl);
this.socket.emit('init', 'some-data');
} catch (err) {
console.log(err);
}
} else if (this.socket.disconnected) {
this.socket.connect();
this.socket.emit('init', 'some-data');
}
this.socket.on('some-channel', (data) => {
// Do something
});
this.socket.on('disconnect', (data) => {
console.log(data);
});
}
They usually work fine but produce disconnection error randomly. From my log file, we can see this:
2018-07-21T00:20:28.209Z[x]INFO: socket 8jBh7YC4A1btDTo_AAAN connected
2018-07-21T00:20:28.324Z[x]INFO: socket 8jBh7YC4A1btDTo_AAAN initialized
2018-07-21T00:21:48.314Z[x]INFO: socket 8jBh7YC4A1btDTo_AAAN disconnected because: ping timeout
2018-07-21T00:21:50.849Z[x]INFO: socket C6O7Vq38ygNiwGHcAAAO connected
2018-07-21T00:23:09.345Z[x]INFO: trying sending message to C6O7Vq38ygNiwGHcAAAO
And at the same time of disconnect message, front-end also noticed a disconnect event which saying transport close.
From the log, we can get the work flow is this:
Front-end started a socket connection and sent an init message to back-end. It also save the socket.
Back-end detected the connection and received init message
Back-end put the socket to the array so that it can be used anytime anywhere
The first socket was disconnected unexpectedly and another connection is published without front-end's awareness so front-end never send a message to initialize it.
Since front-end's saved socket is not changed, it used the old socket id when made http request. As a result, back-end sent message with the old socket which was already removed from socket array.
The situation doesn't happen frequently. Does anyone know what could cause the disconnect and unknown connect issue?

It really depends what "long time http request" is doing. node.js runs your Javascript as a single thread. That means it can literally only do one thing at a time. But, since many things that servers do are I/O related (read from a database, get data from a file, get data from another server, etc...) and node.js uses event-driven asynchronous I/O, it can often have many balls in the air at the same time so it appears to be working on lots of requests at once.
But, if your complex http request is CPU-intensive, using lots of CPU, then it's hogging the single Javascript thread and nothing else can get done while it is hogging the CPU. That means that all incoming HTTP or socket.io requests have to wait in a queue until the one node.js Javascript thread is free so it can grab the next event from the event queue and start to process that incoming request.
We could only really help you more specifically if we could see the code for this "very complex http request".
The usual way around CPU-hogging things in node.js is to offload CPU-intensive stuff to other processes. If it's mostly just this one piece of code that causes the problem, you can spin up several child processes (perhaps as many as the number of CPUs you have in your server) and then feed them the CPU-intensive work and leave your main node.js process free to handle incoming (non-CPU-intensive) requests with very low latency.
If you have multiple operations that might hog the CPU, then you either have to farm them all out to child processes (probably via some sort of work queue) or you can deploy clustering. The challenge with clustering is that a given socket.io connection will be to one particular server in your cluster and if it's that process that just happens to be executing a CPU-hogging operation, then all the socket.io connections assigned to that server would have bad latency. So, regular clustering is probably not so good for this type of issue. The work-queue and multiple specialized child processes to handle CPU-intensive work are probably better because those processes won't have any outside socket.io connections that they are responsible for.
Also, you should know that if you're using synchronous file I/O, that blocks the entire node.js Javascript thread. node.js can not run any other Javascript during a synchronous file I/O operation. node.js gets its scalability and its ability to have many operations in flight at the same from its asynchronous I/O model. If you use synchronous I/O, you completely break that and ruin scalability and responsiveness.
Synchronous file I/O belongs only in server startup code or in a single purpose script (not a server). It should never be used while processing a request in a server.
Two ways to make asynchronous file I/O a little more tolerable are by using streams or by using async/await with promisified fs methods.

Related

SSE/Redis - how to recover messages sent when SSE goes offline

On a website I have a very simple Live chat setup that uses SSE/Redis and pub/sub structure.
The basic setup (without going into details) is:
Client-side using EventSource
Opens SSE connection and subscribes to live events sent by SSE daemon. Sends messages to an API endpoint
connect(hash, eventListener) {
const url = `${url}?client=$hash=${hash}`;
sseSource = new EventSource(url);
sseSource.onopen = function(e) {
reconnectFrequencySeconds = 1;
}
sseSource.onerror = err => {
this.closeSSEStream();
this.reconnectSSEStream(hash, eventListener);
};
sseSource.addEventListener('messages', event => {
const messages = JSON.parse(event.data);
eventListener(messages);
});
},
API endpoint
That stores message in the database and pushes it to a Redis channel.
Redis DB
That keeps and serves the messages.
Server-side SSE daemon
Subscribes client to a channel in a Redis DB and forwards messages to the subscribers using SSE stream.
const subscriber = redis.createClient();
subscriber.select(config.redisDatabase);
subscriber.on('message', function (channel, message) {
log(connectionId, 'Redis: new msg on channel: ' + channel, message);
let event = {
event: 'messages',
data: message
};
currentClient.connection.write(event);
});
The whole thing works pretty well, however, it is one tweak away from perfection.
During deploy we restart our workers (including SSE daemon) and while it goes offline users do not receive LIVE updates. It reconnects just fine but messages that have been sent during down time are lost (as daemon starts listening to messages on reconnect only).
My only idea for a workaround involves an overengineered solution where "lost" messages are collected with a separate API endpoint on reconnect and displayed to the user.
Is there an out-of-the-box way to receive messages that have been stored to Redis BEFORE subscribing to a channel? E.g. "pop" unprocessed messages or something like that?
when you have reconnected send request to check if you are new msg with time of last msg
and if you are newer msg send it in result msg to avoid new request

How to send WebSocketSubject messages only from client to server?

I created an observable for my websocket connection using WebSocketSubject from rxjs. So far so good the server-client communication is working. Now the thing is: I can't distinguish between the origins of the message in my client. I send messages by calling next() on the subject. All subscriptions on the client get those messages too. How can I send messages only to the server instead?
The implementation mainly stems from this article: https://medium.com/factory-mind/angular-websocket-node-31f421c753ff
My code:
socket$: WebSocketSubject<any>;
constructor() {
this.socket$ = WebSocketSubject.create(SOCKET_URL);
this.socket$.subscribe(
(message) => console.log('<-- ' + message),
(err) => console.error('Error on WebSocket:', err),
() => console.warn('Completed!')
);
}
send(message: SocketMessage) {
const tmp: any = {};
tmp.type = message.type;
tmp.payload = message.payload;
// This will be received by the server but also by client subscriptions = bad
this.socket$.next(JSON.stringify(tmp));
}
I found a solution trying to have your described behavior :
RxJS6's WebSocket's documentation states that
Calling next does not affect subscribers of WebSocketSubject - they have no information that something was sent to the server (unless of course the server responds somehow to a message).
Thus, by using RxJS 6 instead of RxJS 5 you should not have the described behavior.

Http Request can't send a response immediately in Nodejs server

I send JSON requests one by one to the nodejs server. After 6th request, server can't reply to the client immediately and then it takes a little while(15 seconds or little bit more and send back to me answer 200 ok) It occurs a writing json value into MongoDB and time is important option for me in terms with REST call. How can I find the error in this case? (which tool or script code can help me?) My server side code is like that
var controlPathDatabaseSave = "/save";
app.use('/', function(req, res) {
console.log("req body app use", req.body);
var str= req.path;
if(str.localeCompare(controlPathDatabaseSave) == 0)
{
console.log("controlPathDatabaseSave");
mongoDbHandleSave(req.body);
res.setHeader('Content-Type', 'application/json');
res.write('Message taken: \n');
res.write('Everything all right with database saving');
res.send("OK");
console.log("response body", res.body);
}
});
My client side code as below:
function saveDatabaseData()
{
console.log("saveDatabaseData");
var oReq = new XMLHttpRequest();
oReq.open("POST", "http://192.168.80.143:2800/save", true);
oReq.setRequestHeader("Content-type", "application/json;charset=UTF-8");
oReq.onreadystatechange = function() {//Call a function when the state changes.
if(oReq.readyState == 4 && oReq.status == 200) {
console.log("http responseText", oReq.responseText);
}
}
oReq.send(JSON.stringify({links: links, nodes: nodes}));
}
--Mongodb save code
function mongoDbHandleSave(reqParam){
//Connect to the db
MongoClient.connect(MongoDBURL, function(err, db)
{
if(!err)
{
console.log("We are connected in accordance with saving");
} else
{
return console.dir(err);
}
/*
db.createCollection('user', {strict:true}, function(err, collection) {
if(err)
return console.dir(err);
});
*/
var collection = db.collection('user');
//when saving into database only use req.body. Skip JSON.stringify() function
var doc = reqParam;
collection.update(doc, doc, {upsert:true});
});
}
You can see my REST call in google chrome developer editor. (First six call has 200 ok. Last one is in pending state)
--Client output
--Server output
Thanks in advance,
Since it looks like these are Ajax requests from a browser, each browser has a limit on the number of simultaneous connections it will allow to the same host. Browsers have varied that setting over time, but it is likely in the 4-6 range. So, if you are trying to run 6 simultaneous ajax calls to the same host, then you may be running into that limit. What the browser does is hold off on sending the latest ones until the first ones finish (thus avoiding sending too many at once).
The general idea here is to protect servers from getting beat up too much by one single client and thus allow the load to be shared across many clients more fairly. Of course, if your server has nothing else to do, it doesn't really need protecting from a few more connections, but this isn't an interactive system, it's just hard-wired to a limit.
If there are any other requests in process (loading images or scripts or CSS stylesheets) to the same origin, those will count to the limit too.
If you run this in Chrome and you open the network tab of the debugger, you could actually see on the timline exactly when a given request was sent and when its response was received. This should show you immediately whether the later requests are being held up at the browser or at the server.
Here's an article on the topic: Maximum concurrent connections to the same domain for browsers.
Also, keep in mind that, depending upon what your requests do on the server and how the server is structured, there may be a maximum number of server requests that can efficiently processed at once. For example, if you had a blocking, threaded server that was configured with one thread for each of four CPUs, then once the server has four requests going at once, it may have to queue the fifth request until the first one is done causing it to be delayed more than the others.

Detach node child process after receiving a first message

I'm building a system with node.js where the user is able to spawn worker processes from a web interface, and I'm having trouble combining these two criteria for the system:
The web server process and the worker process should run independently, so that if the web server crashes, the worker process isn't affected.
The mother process should be able to receive at least one initial message from the worker process. After that message, I want to log the output from the worker process to a file.
I've been able to accomplish both criteria by themselves, but not together. Here's what the relevant parts of my code currently looks like:
function spawnWorker (id, callback) {
var worker = spawn("node", [require.resolve("worker"), id], {
detached: true,
stdio: ["ignore", "pipe", "pipe"]
});
worker.unref();
var logfile = fs.createWriteStream("foo.log");
worker.stdout.once("data", function (data) {
worker.stdout.pipe(logfile);
worker.stderr.pipe(logfile);
callback(undefined, data.toString());
});
}
This meets criteria 2 (ie. it receives the first message from the worker and redirects all following output to a file), but if I kill the mother process, the worker process is killed as well, even though the output has been redirected.
My guess is that this is because the worker's stdout and stderr are piped to the mother process before they're sent to the file stream. If I were to pass in a stream when spawning the process, according to the docs, node would instead duplicate that stream in the worker process:
The stream's underlying file descriptor is duplicated in the child process[...]
Is there a way I can remove the connection between the mother process and the worker process after having received the first message? Or should I use some type of RPC to accomplish this, and if so, what would be a good choice for this one message once communication?
I solved this using node's net module to spawn a server on a random port that is passed to the worker when it's started (so it knows where to send a response when it's ready).
Here's an example of an implementation of my solution, to clarify, this meets the requirements of:
saving the log output of the worker process,
keeping the worker process detached from the mother process (so that if the latter crashes, it doesn't affect the worker)
server.js
var net = require("net"),
fs = require("fs"),
spawn = require("child_process").spawn,
randomPort = require("get-port");
randomPort(function (port) {
var server = net.createServer(function (connection) {
connection.on("data", function (data) {
server.close();
/**
* Response received from worker, this is where you would put
* your logic that returns a response to the client that initiated
* the request to the server.
*/
});
});
server.listen(port);
var logfile = fs.createWriteStream("log");
var worker = spawn("node", ["worker.js", port], {
detached: true,
stdio: ["ignore", logfile, logfile]
});
worker.unref();
worker.on("close", function (status, message) {
console.log("worker closed", status, message);
});
});
worker.js
var net = require("net");
var options = process.argv.slice(2),
rpcPort = Number(options[1]);
var client = net.connect({port: rpcPort}, function () {
client.end(JSON.stringify({started: true}}));
});

Send message to specific client with socket.io and node.js

I'm working with socket.io and node.js and until now it seems pretty good, but I don't know how to send a message from the server to an specific client, something like this:
client.send(message, receiverSessionId)
But neither the .send() nor the .broadcast() methods seem to supply my need.
What I have found as a possible solution, is that the .broadcast() method accepts as a second parameter an array of SessionIds to which not send the message, so I could pass an array with all the SessionIds connected at that moment to the server, except the one I wish send the message, but I feel there must be a better solution.
Any ideas?
Ivo Wetzel's answer doesn't seem to be valid in Socket.io 0.9 anymore.
In short you must now save the socket.id and use io.sockets.socket(savedSocketId).emit(...) to send messages to it.
This is how I got this working in clustered Node.js server:
First you need to set Redis store as the store so that messages can go cross processes:
var express = require("express");
var redis = require("redis");
var sio = require("socket.io");
var client = redis.createClient()
var app = express.createServer();
var io = sio.listen(app);
io.set("store", new sio.RedisStore);
// In this example we have one master client socket
// that receives messages from others.
io.sockets.on('connection', function(socket) {
// Promote this socket as master
socket.on("I'm the master", function() {
// Save the socket id to Redis so that all processes can access it.
client.set("mastersocket", socket.id, function(err) {
if (err) throw err;
console.log("Master socket is now" + socket.id);
});
});
socket.on("message to master", function(msg) {
// Fetch the socket id from Redis
client.get("mastersocket", function(err, socketId) {
if (err) throw err;
io.sockets.socket(socketId).emit(msg);
});
});
});
I omitted the clustering code here, because it makes this more cluttered, but it's trivial to add. Just add everything to the worker code. More docs here http://nodejs.org/api/cluster.html
each socket joins a room with a socket id for a name, so you can just
io.to('socket#id').emit('hey')
docs: http://socket.io/docs/rooms-and-namespaces/#default-room
The simplest, most elegant way
verified working with socket.io v3.1.1
It's as easy as:
client.emit("your message");
And that's it. Ok, but how does it work?
Minimal working example
Here's an example of a simple client-server interaction where each client regularly receives a message containing a sequence number. There is a unique sequence for each client and that's where the "I need to send a message to a particular client" comes into play.
Server
server.js
const
{Server} = require("socket.io"),
server = new Server(8000);
let
sequenceNumberByClient = new Map();
// event fired every time a new client connects:
server.on("connection", (socket) => {
console.info(`Client connected [id=${socket.id}]`);
// initialize this client's sequence number
sequenceNumberByClient.set(socket, 1);
// when socket disconnects, remove it from the list:
socket.on("disconnect", () => {
sequenceNumberByClient.delete(socket);
console.info(`Client gone [id=${socket.id}]`);
});
});
// sends each client its current sequence number
setInterval(() => {
for (const [client, sequenceNumber] of sequenceNumberByClient.entries()) {
client.emit("seq-num", sequenceNumber);
sequenceNumberByClient.set(client, sequenceNumber + 1);
}
}, 1000);
The server starts listening on port 8000 for incoming connections. As soon as a new connection is established, that client is added to a map that keeps track of its sequence number. The server also listens for the disconnect event to remove the client from the map when it leaves.
Each and every second, a timer is fired. When it does, the server walks through the map and sends a message to every client with their current sequence number, incrementing it right after. That's all that is to it. Easy peasy.
Client
The client part is even simpler. It just connects to the server and listens for the seq-num message, printing it to the console every time it arrives.
client.js
const
io = require("socket.io-client"),
ioClient = io.connect("http://localhost:8000");
ioClient.on("seq-num", (msg) => console.info(msg));
Running the example
Install the required libraries:
npm install socket.io#3.1.1 socket.io-client#3.1.1
Run the server:
node server
Open other terminal windows and spawn as many clients as you want by running:
node client
I have also prepared a gist with the full code here.
Well you have to grab the client for that (surprise), you can either go the simple way:
var io = io.listen(server);
io.clients[sessionID].send()
Which may break, I doubt it, but it's always a possibility that io.clients might get changed, so use the above with caution
Or you keep track of the clients yourself, therefore you add them to your own clients object in the connection listener and remove them in the disconnect listener.
I would use the latter one, since depending on your application you might want to have more state on the clients anyway, so something like clients[id] = {conn: clientConnect, data: {...}} might do the job.
You can use
//send message only to sender-client
socket.emit('message', 'check this');
//or you can send to all listeners including the sender
io.emit('message', 'check this');
//send to all listeners except the sender
socket.broadcast.emit('message', 'this is a message');
//or you can send it to a room
socket.broadcast.to('chatroom').emit('message', 'this is the message to all');
In 1.0 you should use:
io.sockets.connected[socketid].emit();
Whatever version we are using if we just console.log() the "io" object that we use in our server side nodejs code, [e.g. io.on('connection', function(socket) {...});], we can see that "io" is just an json object and there are many child objects where the socket id and socket objects are stored.
I am using socket.io version 1.3.5, btw.
If we look in the io object, it contains,
sockets:
{ name: '/',
server: [Circular],
sockets: [ [Object], [Object] ],
connected:
{ B5AC9w0sYmOGWe4fAAAA: [Object],
'hWzf97fmU-TIwwzWAAAB': [Object] },
here we can see the socketids "B5AC9w0sYmOGWe4fAAAA" etc. So, we can do,
io.sockets.connected[socketid].emit();
Again, on further inspection we can see segments like,
eio:
{ clients:
{ B5AC9w0sYmOGWe4fAAAA: [Object],
'hWzf97fmU-TIwwzWAAAB': [Object] },
So, we can retrieve a socket from here by doing
io.eio.clients[socketid].emit();
Also, under engine we have,
engine:
{ clients:
{ B5AC9w0sYmOGWe4fAAAA: [Object],
'hWzf97fmU-TIwwzWAAAB': [Object] },
So, we can also write,
io.engine.clients[socketid].emit();
So, I guess we can achieve our goal in any of the 3 ways I listed above,
io.sockets.connected[socketid].emit();
OR
io.eio.clients[socketid].emit();
OR
io.engine.clients[socketid].emit();
You can do this
On server.
global.io=require("socket.io")(server);
io.on("connection",function(client){
console.log("client is ",client.id);
//This is handle by current connected client
client.emit('messages',{hello:'world'})
//This is handle by every client
io.sockets.emit("data",{data:"This is handle by every client"})
app1.saveSession(client.id)
client.on("disconnect",function(){
app1.deleteSession(client.id)
console.log("client disconnected",client.id);
})
})
//And this is handle by particular client
var socketId=req.query.id
if(io.sockets.connected[socketId]!=null) {
io.sockets.connected[socketId].emit('particular User', {data: "Event response by particular user "});
}
And on client, it is very easy to handle.
var socket=io.connect("http://localhost:8080/")
socket.on("messages",function(data){
console.log("message is ",data);
//alert(data)
})
socket.on("data",function(data){
console.log("data is ",data);
//alert(data)
})
socket.on("particular User",function(data){
console.log("data from server ",data);
//alert(data)
})
As of version 1.4.5, be sure you provide a properly prefixed socketId in io.to().
I was taking the socketId the Client logged to debug and it was without prefix so I ended up searching forever till I found out! So you might have to do it like this if the Id you have is not prefixed:
io.to('/#' + socketId).emit('myevent', {foo: 'bar'});
io.sockets.sockets[socket.id].emit(...) worked for me in v0.9
Also you can keep clients refferences. But this makes your memmory busy.
Create an empty object and set your clients into it.
const myClientList = {};
server.on("connection", (socket) => {
console.info(`Client connected [id=${socket.id}]`);
myClientList[socket.id] = socket;
});
socket.on("disconnect", (socket) => {
delete myClientList[socket.id];
});
then call your specific client by id from the object
myClientList[specificId].emit("blabla","somedata");
Socket.IO allows you to “namespace” your sockets, which essentially means assigning different endpoints or paths.
This might help:
http://socket.io/docs/rooms-and-namespaces/

Categories