I have a node.js server, on which a socket.io is supposed to emit something every second. But at the client side, the clients get the data more than every 30 seconds. Anybody has any idea why this is happening and what's the solution?
var redis = require("redis"),
client = redis.createClient();
const nsp_page=io.of('/myPage');
nsp_page.on('connection', function(socket){
(function update_index() {
client.get('boxIndex', function (err, data) {
socket.emit('boxIndex',data);
});
setTimeout(update_index,1000);
})();
});
Thanks
But at the client side, the clients get the data more than every 30 seconds. Anybody has any idea why this is happening and what's the solution?
Because your code sends the data every second. This line of code:
setTimeout(update_index,1000);
Causes it to call udpate_index() every second, not every 30 seconds. If you want it to update every 30 seconds, you would change that to this:
setTimeout(update_index, 30 * 1000);
But, note that the way your code works is very inefficient because you're calling the exact same client.get() for every single connected client. That's really inefficient. It would make more sense to call client.get() once and then broadcast that data to every connected client:
const redis = require("redis");
const client = redis.createClient();
const nsp_page = io.of('/myPage');
setInterval(() => {
client.get('boxIndex', function (err, data) {
if (err) {
console.log("boxIndex not found", err);
} else {
// send to all connected clients
nsp_page.emit('boxIndex', data);
}
});
}, 30 * 1000)
When you have lots of connected clients, this will do only one query of your redis database every 30 seconds and use that result for all clients instead of calling redis separately for every single connected client.
Related
What I intend to do is a program that sends congratulatory emails for their birthday to several users, then the program will take today and execute a query to the database (it is an Excel file), in which it will take the date of the users and compare their date of birth with the current date, if the month and day coincide, mail will be sent. I think it can be done with a setInterval(), but I don't know if it affects the performance of the program. Since it will be uploaded on a windows server 2012 server of my company.
My code:
const express = require("express");
const app = express();
const excel = require('./public/scripts/readExcel.js');
const email = require('./services/email/sendEmail.js');
app.post('/send-email',(req, res)=>{
setInterval(() => {
email.sendEmail()
.then((result) => {
res.status(200).jsonp(req.body);;
console.log(result)
}).catch((err) => {
res.status(500).jsonp(req.body);;
console.log(err);
});
}, 3600000);//1 hour
});
app.listen(4000, ()=>{
console.log("Serven on -> http://localhost:4000");
})
Basically what it does is call the sendEmail function every hour which reads the Excel file or the database and extracts the date fields, compares with the current day, and sends the mail with Nodemailer to those who have birthdays. Also, the setInterval would go in the route of "/send-email" or how would the request be made?
For that, you can also run a cron job at every hour using npm package cron-job
using
var cron = require('node-cron');
cron.schedule('* * 1 * *', () => {
console.log('running a task every hour');
});
I'll break down my answer in two parts
What you need to do to make your solution work
How can you optimise the performance
1. What you need to do to make your solution work
There are two essential problems you need to resolve.
a. Your solution will work as it is, only thing you need to do is to call /send-email endpoint once after starting your server. BUT... this comes with side effects.
As setInterval will call the email.sendEmail... code block every hour, and this code block calls res.status(200).jsonp(req.body) every time. If you don't know this res.status.. sets the response for the request you receive. In this case, your request to /send-email. For the first time, it will work fine because you are returning the response to your above request. But when second time call to this code block kicks in, it has nothing to respond to because request has already been responded. Remember, HTTP protocol responds to a request once, then the request has been completed. So for this reason, your code block res.status... becomes invalid. So first thing, call res.status only once. So I'd remove this line out of the setInterval code block as follows
app.post('/send-email',(req, res)=>{
setInterval(() => {
email.sendEmail()
.then((result) => {
console.log(result)
}).catch((err) => {
console.log(err);
});
}, 3600000);//1 hour
res.status(200).jsonp(req.body);
})
b. Also I don't think you'd want the hastle of calling /send-email every time you start server, so I'd also make sure that this code block for birthday wishes gets kicked off every time you start server automatically. So I'd then just remove the line app.post('/send-email',(req, res)=>{. Also not that I'm not calling this for a request, I don't have any request to send response to so I can also remove the res.status.. line. And your code looks like this now
const express = require("express");
const app = express();
const email = require('./services/email/sendEmail.js');
(function(){
// Birthday wish email functionality
setInterval(() => {
email.sendEmail()
.then((result) => {
console.log(result)
}).catch((err) => {
console.log(err);
});
}, 3600000);//1 hour
})() // Immediately invoked function
That's it, your solution works now. Now to send birthday wish emails, you don't need to do anything else other than just starting your server.
Let's move on to second part now
2. How can you optimise the performance
a. Set interval to be 24hrs instead of 1 hr
Why do you need to check every hour for birthday? If you don't have a good answer here, I'd definitely change the interval to be 24hrs
b. Making the code more robust to deal with large data
As long as you have only 100s of entries in your excels and they are not going to grow much in future, I wouldn't go into making it more complex for performance.
But if your entries are destined to grow to 1000s and further. I'd suggest to use database(such as mongodb, postgres or mysql, etc.) to store your data and query only the entries with the birthday matching the particular date.
I'd also implement a queuing system to process query and send emails in batches instead of doing all of that at once.
I'm having trouble displaying the correct amount of online users. A question similar to this have shown me that I can count the users this way:
var count = 0
socket.on('connection', function(client) {
count++;
client.broadcast({count:count})
client.on('disconnect', function(){
count--;
})
})
The issue I'm constantly running into with that method is that when a user happens to reload the page too quickly, the counter pulls in too much it can throw out.
As you can see, on the right side of the image, a user spammed the reload and it caught more users online than there actually is. (There was only one user on the server at this time)
My question is is there a better or more reliable way to count the exact amount users online without the extra 'virtual users', without using the users++/users-- method?
If they're logging in as a user, then you should authenticate them to the socket. Use that authentication to see if they already have a session, and disconnect them decrementing the count, before you increment it again with the new session.
An example below. The clients objects stores the connected clients, with values being the sockets they're connected to.
var clients = {};
socket.on('connection', function(client) {
//Authenticate the client (Using query string parameters, auth tokens, etc...), and return the userID if the user.
var userId = authenticate(client);
if ( !userId ) {
//Bad authentication, disconnect them
client.disconnect();
return;
}
if (clients[userId]) {
//They already have a session, disconnect
clients[userId].disconnect();
}
//Set session here
clients[userId] = client;
client.broadcast({count: Object.keys(clients).length})
client.on('disconnect', function(){
delete clients[userId];
})
})
Could do this pretty cleanly with the Observable pattern (using RxJS v5 here):
const { Observable } = require('rxjs')
const connections = Observable.fromEvent(socket, 'connection').mapTo(1)
const disconnections = Observable.fromEvent(socket, 'disconnect').mapTo(-1)
// emit 1 for connection, -1 for disconnection
Observable.merge(connections, disconnections)
.scan((total, change) => total + change, 0) // emit total
.subscribe(count => client.broadcast({ count }))
You can use Server Sent Events for this purpose really.
Take a look at https://javascript.info/server-sent-events
I am using socket.io in my project. I turned on reconnect feature. I want to if user disconnects from server show an alert (Your internet connection loss. Trying reconnect). And if the user reconnects again I want to show one more alert (Don't worry, you are connected).
How can I do it?
To detect on the client you use
// CLIENT CODE
socket.on('disconnect', function(){
// Do stuff (probably some jQuery)
});
It's the exact same code as above for a node.js server too.
If you want for some reason to detect a user disconnecting and display it to the others, you will need to use the server one to detect it the person leaving and then emit back out a message to the others using something like:
socket.on('disconnect', function(){
socket.broadcast.to(roomName).emit('user_leave', {user_name: "johnjoe123"});
});
Hope this helps
socket.io has a disconnect event, put this inside your connect block:
socket.on('disconnect', function () {
//do stuff
});
I handled this problem this way. I've made an emit sender on client which is calling heartbeat on server.
socket.on('heartbeat', function() {
// console.log('heartbeat called!');
hbeat[socket.id] = Date.now();
setTimeout(function() {
var now = Date.now();
if (now - hbeat[socket.id] > 5000) {
console.log('this socket id will be closed ' + socket.id);
if (addedUser) {
--onlineUsers;
removeFromLobby(socket.id);
try {
// this is the most important part
io.sockets.connected[socket.id].disconnect();
} catch (error) {
console.log(error)
}
}
}
now = null;
}, 6000);
});
I found this code function to call:
io.sockets.connected[socket.id].disconnect();
inpired on invicibleTrain iv made a online checker also
SERVER SIDE
const sockets={online:{},admins:{}}
// the disconnect function you create the way you wish to, example:
const disconnect=()=>{
socket.emit('disconected')
io.in(socket.id).disconnectSockets()
delete sockets.online[socket.uid]
}
io.on('connection',socket=>{
socket.emit('connection',socket.id)
// the checker if user is online
let drop
const dropCheck=()=>{
if(!socket) return; // if user connects twice before the check, simply stops process
socket.emit('dropCheck')
drop = setTimeout(()=>disconnect(),4000) // 4 secs to recieve answer
}
const setDrop=()=>setTimeout(()=>dropCheck(),60000) // 60 secs to restart the process
socket.on('dropCheck',()=>{
clearTimeout(drop) // cancells actual drop coutdown (if any)
setDrop() // sets a new
})
//sets the user ID inside the socket used for comunication
socket.on('uid',uid=>{
socket.online[uid] = {status:'busy',socket:socket.id}
setDrop()
})
})
CLIENT SIDE
const sio = io(`...httpAddress...`,{transports:['websocket']})
sio.on('connection',socket=>{
sio.emit('uid','user Id here..') // sends the the user ID: uid
sio.on('dropCheck',()=>{ // responds to the checker
sio.emit('dropCheck')
})
})
Explanation:
The user logs in and after 60 seconds dropCheck is called
the dropCheck emits a ping and set a timmer of 4 seconds
the user handle the emited ping and responds with another ping
a: if the response comes within 4 secs the 4 secs timmer is cancelled and refreshes the 60 seconds timmer (restarting the
process)
b: if the response fails or delay too much the disconnect() function is called
I have a Node server which sends streaming tweets to clients as they connect, using Sockets.io and ntwitter. At the moment all the tweets (from different users) get sent to every client. But each client only requires a certain subset of tweets, and I'd like to the server to only send that subset (or category).
I think that having each category being like a room, in Sockets, would work, but I can't quite work out how to adapt my code to use them. Or, given that there's no communication between clients, maybe that's not the best solution?
The relevant, simplified, bits of current code...
Client:
var socket = io.connect(window.location.hostname);
socket.on('messages', function(messages_packet) {
$.each(messages_packet, function(idx, tweet) {
if (tweet_is_in_this_clients_category(tweet)) {
display_messages(messages_packet);
};
}
});
Server:
// [A function which, on start-up, fetches existing tweets and caches them.]
// Send cached messages when a client connects.
sockets.sockets.on('connection', function(socket) {
socket.emit('messages', cached_messages);
});
// Fetch tweets from the stream, and send new ones to clients.
twitter.stream('statuses/filter', {follow: [12345, 345678, etc]}, function(stream) {
stream.on('data', function(tweet) {
add_tweet_to_cache(tweet);
sockets.sockets.emit('messages', [tweet]);
}
});
So, the sockets.emit() part is currently sending every tweet to all the clients. And then the client decides whether to show that tweet, if it's in the client's category. It would obviously be more efficient if the server only sent tweets to the clients in the correct category. Given the server already knows which tweets are in which categories, how do I only emit them to those categories, rather than every client?
After some trial and error I seem to have it working. No idea if this is the best or correct way, but... the above code has been tweaked so it's now something like that shown below. There's one addition to the client code, and two to the server code.
Client:
var socket = io.connect(window.location.hostname);
// NEW CLIENT PART:
socket.on('connect', function(){
// room_name was created based on the URL this page was requested at:
socket.emit('subscribe', room_name);
};
socket.on('messages', function(messages_packet) {
$.each(messages_packet, function(idx, tweet) {
if (tweet_is_in_this_clients_category(tweet)) {
display_messages(messages_packet);
};
}
});
Server:
// [A function which, on start-up, fetches existing tweets and caches them.]
// Send cached messages when a client connects.
sockets.sockets.on('connection', function(socket) {
// NEW SERVER PART 1:
socket.on('subscribe', function(room_name) {
socket.join(room_name);
socket.emit('messages', cached_messages_for_this_room);
});
});
// Fetch tweets from the stream, and send new ones to clients.
twitter.stream('statuses/filter', {follow: [12345, 345678, etc]}, function(stream) {
stream.on('data', function(tweet) {
add_tweet_to_cache(tweet);
// NEW SERVER PART 2:
// Gets the array of room_names this twitter account is associated with:
var rooms = get_rooms_for_twitter_account(tweet.user.screen_name);
rooms.forEach(
function(room_name) {
sockets.sockets.in(room_name).emit('messages', [tweet]);
};
);
};
});
So, when a client connects it sends a 'subscribe' message, and the name of the room it wants to join.
When the server receives a 'subscribe' event it sends only the existing tweets associated with that room (the logic for getting that list of tweets is done elsewhere; not relevant to this particular issue).
And whenever a new tweet is picked up by the server, it finds the room(s) this tweet is associated with, and then emits the tweet to every client in each of them.
This seems to work... do point out anything that makes no sense, or could be done much better!
I'm making a webpage based around players being able to invite other players to parties, and other things a long the lines.
I have your basic send / receive / update of the chat/users in your party. The only thing is, what's to stop somebody from sitting there opening up a developer console and going
socket.emit('updateUsers', 'Weiner');
socket.emit('updateUsers', 'Idiot');
socket.emit('updateUsers', 'Bad word');
socket.emit('updateUsers', 'Other stupid malicious really long spam the chat name');
How can I prevent against this so that they can not do such things?
(Full JS Stack, Node.js)
Thanks!
I faced this problem aswell, This was my solution as far as spamming emits go (malicious socket use)..
var spamData = new Object();
var spamCheckFunctions = ["updateUsers","moreEmits"]; // anti-spam will check these socket emits
var antiSpam = 3000; // anti spam check per milliseconds
var antiSpamRemove = 3; // -spam points per antiSpam check
var maxSpam = 9; // Max spam points before disconnect is thrown to the socket
io.sockets.on('connection', function (socket) {
// Spam Check, this binds to all emits
var emit = socket.emit;
socket.emit = function() {
data = Array.prototype.slice.call(arguments);
if(spamCheckFunctions.contains(data[0])){
addSpam(socket);
};
emit.apply(socket, arguments);
};
var $emit = socket.$emit;
socket.$emit = function() {
data = Array.prototype.slice.call(arguments);
if(spamCheckFunctions.contains(data[0])){
addSpam(socket);
}
$emit.apply(socket, arguments);
};
});
function maxSpamCheck(socket){
if(spamData[socket.username].spamScore>=maxSpam && !socket.spamViolated){
socket.spamViolated = true;
socket.disconnect();
}
}
function checkSpam(){
for(user in spamData){
if(spamData[user].spamScore>=1) spamData[user].spamScore-=antiSpamRemove;
}
return;
}
setInterval(checkSpam,antiSpam);
function addSpam(socket){
if(socket.spamViolated) return;
spamData[socket.username].spamScore+=1;
maxSpamCheck(socket);
}
// Then add this where your user is authenticated
function authenticate(socket){
socket.username = username // here you define username
socket.spamViolated = false;
spamData[socket.username] = {
spamScore: 0
}
}
Array.prototype.contains = function(k) {
for(var p in this)
if(this[p] === k)
return true;
return false;
};
basically binds to all emits and checks if the emit name is contained in spamCheckFunctions if it is it will add a spam point, if a user exceeds a spam score amount (maxSpam); he will be disconnected. And for every milliseconds defined at antiSpam will minus the user spam score defined at antiSpamRemove
I'm sure there are cleaner solutions but this one worked out pretty good for me :)
Just make sure to verify/authenticate the users.
this is how I authenticate them (not using nodejs as a webserver, but had django):
io.configure(function(){
io.set('authorization', function(data, accept){
if(data.headers.cookie){
data.cookie = cookie_reader.parse(data.headers.cookie);
return accept(null, true);
}
return accept('error', false);
});
});
now you can access socket.handshake.cookie['sessionid'] (in my case this worked with django)
then match the socket.handshake.cookie['sessionid'] with a entry where your sessions are stored on the webserver
That's a difficult problem in general. Two things you could do:
1) Use self-invoking functions on the client side, i.e.
(function(w) {
// define your sockets here
var socket = ...;
})(window);
Obviously it is on the client side so this is not really secure. But it's not bad to have such wall.
2) On the server side keep track of the frequency of posting. For example if someone posts 5 times in a second, then you can assume that it is a spam and you could block that socket. It is especially effective if combined with authentication and complex registration (so people will have problem in creating new account).
Use an md5/sha key which behaves like a cookie.
Generate a key for a specific user and send it to the client and always check that incoming requests have the same key
It won't be completely secure since the hacker can always find your key in either the source code or localStorage but try to hide it through obfuscation of your code
A way to prevent spam is to either implement user authentication and/or a packet rate limiter. Add a middleware function which keeps track of the socketId and the amount of packets being sent through that socket. When it exceeds your limit disconnect the socket.
You can even add an extra function which keeps track of the IP address of that socket, if an IP address will be disconnected too often due spam you can ban that ip. Add a check on your connection event which IP addresses are allowed.
Use rate-limiter-flexible package for limiting number of events per second. Limiting by IP is the most simple, but would be better to limit by userId if possible.
const app = require('http').createServer();
const io = require('socket.io')(app);
const { RateLimiterMemory } = require('rate-limiter-flexible');
app.listen(3000);
const rateLimiter = new RateLimiterMemory(
{
points: 5, // 5 points
duration: 1, // per second
});
io.on('connection', (socket) => {
socket.on('bcast', async (data) => {
try {
await rateLimiter.consume(socket.handshake.address); // consume 1 point per event from IP
socket.emit('news', { 'data': data });
socket.broadcast.emit('news', { 'data': data });
} catch(rejRes) {
// no available points to consume
// emit error or warning message
socket.emit('blocked', { 'retry-ms': rejRes.msBeforeNext });
}
});
});
Read more in official docs