We've been trying to deploy a small NodeJS app using Socket.IO and have been running into a problem where while the heap size of the app remains fairly acceptable, the total memory used (rss) creeps up to over 2gb after around 2 hours, and continues to rise.
In an effort to make sure the problem wasn't in our code, we deployed a bare bones app with no custom logic apart from initializing Socket IO. We ran that against the same production traffic, and experienced the same issue.
Every 10 seconds we output the following data: rss memory usage, heap total, heap count, and connection count. Here's a sample of the output:
523898880 199490816 123040352 2001
537059328 209774080 163828336 2011
538578944 206714368 150879848 2031
535252992 199514880 156743280 2041
542162944 200522752 145077944 2039
539652096 195387136 129486792 2055
551006208 206726400 170918304 2070
553254912 205706496 156447496 2071
550584320 198482944 154005496 2076
564363264 209810176 140442920 2095
561176576 201578752 123214232 2118
562487296 200546816 110638376 2112
572096512 206714368 162713240 2133
569552896 200546816 147439016 2121
577777664 205682432 136653448 2115
582496256 207770368 121204056 2133
582909952 205706496 115449888 2153
597364736 215989760 164582600 2158
590491648 204686592 148962008 2158
598315008 209810176 137608840 2164
598249472 205718528 123472944 2188
607158272 211898112 160187496 2168
609869824 210866176 154986472 2161
618110976 214969856 142425488 2180
615014400 207782400 119745816 2188
623575040 214981888 163602944 2180
624717824 210842112 147051160 2189
627556352 210866176 142542800 2191
636477440 216013824 129968776 2203
643809280 221149440 162858408 2219
644407296 217057792 154994536 2224
642068480 211922176 141626008 2240
649084928 214969856 123126792 2267
662454272 224233216 166539024 2272
659439616 217045760 162742688 2258
662867968 217057792 137425392 2266
667013120 218065664 119616592 2261
673230848 220129536 172101080 2272
677904384 220129536 149771776 2267
676691968 217045760 129936448 2267
674639872 211898112 125941816 2277
689025024 223225344 163745856 2274
689991680 219109632 151478936 2282
698601472 225301248 137102712 2298
706170880 229428992 171321288 2306
705675264 224257280 160088496 2303
701198336 217033728 149326384 2313
701833216 216013824 129806072 2314
718053376 227365120 184078288 2335
718950400 223225344 157977312 2333
717037568 218065664 146137456 2354
714428416 210890240 136566344 2381
As you can see, in a fairly short amount of the time the total memory usage increased by 200mb, even though the connection count only increased by around 400. The heap usage remained roughly the same, just a bit higher to account for the higher connection count.
We're running on Debian Wheezy on 64bit. NodeJS version is 0.10.29, and Socket IO version is 1.0.6. The code we're using is:
var http = require('http'),
io = require('socket.io');
var app = http.createServer();
var server = io(app);
app.listen(80);
var connections = 0;
server.on('connection', function(socket) {
connections++;
socket.on('disconnect', function() {
connections--;
});
});
setInterval(function() {
var mem = process.memoryUsage();
console.log(mem.rss + ' ' + mem.heapTotal + ' ' + mem.heapUsed + ' ' + connections);
}, 10000);
Is there any way we can find out why Node is using so much memory in total, or any way to see what's happening outside of the heap to try and find the memory leak? We've already tried all of the usual tricks for checking heap usage and found nothing, but did not expect to since the problem doesn't seem to be with memory on the heap.
If you think a node.js module has a memory leak taking snapshots of the system memory from within the process will provide inaccurate results.
Your best bet it to use tools such as valgrind, gdb, prstat & dtrace. Joyent even provides a nice module to help you visualize the process in question.
Related
I am having difficulty retrieving heart rate data from the HumanActivityMonitorManager. Specifically, I am using a query with the readRecorderData function but not returning the proper results.
When I check the console, I see the error: "NotFoundError: Failed to read recorded data". According to the docs this means there is no data found.
Samsung Galaxy Watch. 46mm. Tizen Studio.
var query = {};
query.startTime =(new Date(2019, 6, 5)).getTime() / 1000;
query.endTime = (new Date(2019, 7, 31)).getTime() / 1000;
query.anchorTime = (new Date(2019, 7, 31, 0, 0)).getTime() / 1000;
query.interval = 1440; /* Day */
var type = "HRM";
try{
tizen.humanactivitymonitor.readRecorderData(type, query, onread, onerror);
}
catch (err){
console.log(err.name + ": " + err.message);
}
This should retrieve the HRM data. This works if I change the type to Pedometer or Pressure. I can get the data in real time but would love to be able to get the data already being grabbed by the device.
Any thoughts?
HRM data require a few seconds to get its initial value. Have you allowed enough time for the system to record the HRM by calling tizen.humanactivitymonitor.start('HRM', onchangedCB); and wait for onchangedCB? You need to do readRecorderData when humanactivitymonitor says it's ready to give you the data as in the reference.
Refer to https://developer.tizen.org/ko/development/guides/web-application/sensors/human-activity-monitor?langredirect=1 for more detail.
I experienced performance issues with requestAnimationFrame().
Consider the following code. It's a simple loop which prints the time since the last frame every time this time delta is larger than 20ms.
const glob_time_info = {delta_time: 0.0, last_frame: performance.now()};
var render = function (timestamp) {
glob_time_info.delta_time = timestamp - glob_time_info.last_frame;
glob_time_info.last_frame = timestamp;
if(glob_time_info.delta_time > 20)
console.log(glob_time_info.delta_time);
requestAnimationFrame(render);
};
render(performance.now());
As I understood requestAnimationFrame this snippet should never print anything, because it tries to run 60 times a second (60Hz as my monitor).
Therefore time delta should always be somewhat around 16-17ms.
But it prints times around 33ms every few seconds.
Why?
I experienced this on windows 10 with Chrome 54 and Firefox 49. I own an i5-6600
UPDATE
Here the output of Nit's script for windows and ubuntu. Windows, what are you doing?
Windows 10 (PC):
WIndows 8 (same netbook as below):
Ubuntu (same netbook as above):
It's easy to test your hypothesis that the issue is related to the platform you're running on - measure the performance.
Shortly put, run requestAnimationFrame a number of times similar to how you did and note down a timestamp on each run. After that simply visualize the results.
var times = [];
var measure = function() {
times.push(new Date().getTime());
if (times.length > 100) return draw();
requestAnimationFrame(measure);
};
var draw = function() {
var dataset = {
x: [],
y: [],
type: 'bar'
};
var layout = {
xaxis: {
title: 'measurement #'
},
yaxis: {
title: 'iteration duration (ms)'
},
height: 250
};
var options = {
displayModeBar: false
};
times.reduce(function(previous, current, i) {
dataset.x.push(i);
dataset.y.push(current - previous);
return current;
}, times.shift());
Plotly.newPlot('target', [dataset], layout, options);
}
measure();
#target {
margin-top: -50px;
}
<script src="https://cdn.plot.ly/plotly-1.2.0.min.js"></script>
<div id="target"></div>
You can run the same simulation on different operating systems and different browsers to see if you can narrow down the issue further.
As Travis J stated in the comments it's related to the operating system.
This performance issue doesn't appear on linux. So there is nothing I (we) can do about it.
I have 4 cores and ran this code according to this example :
var cluster = require('cluster');
var http = require('http');
var numCPUs = require('os').cpus().length;
var id = 0;
if (cluster.isWorker) {
id = cluster.worker.id;
}
var iterations = 1000000000;
console.time('Function #' + id);
for (var i = 0; i < iterations; i++) {
var test = 0;
}
console.timeEnd('Function #' + id);
if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
}
With 4 fork (the code above), I got :
Function #0: 1698.801ms
Function #1: 3282.679ms
Function #4: 3290.384ms
Function #3: 3425.090ms
Function #2: 3424.922ms
With 3 fork, I got :
Function #0: 1695.155ms
Function #2: 1822.867ms
Function #3: 2444.156ms
Function #1: 2606.680ms
With 2 fork, I got :
Function #0: 1684.929ms
Function #1: 1682.897ms
Function #2: 1686.123ms
I don't understand these results. Isn't 1 fork/core the optimal number ? Here I see that 4 fork is not better than 2 fork.
My guess is that your hardware actually only has 2 physical cores. However, because of hyper-threading (HT), the OS will say that there are 4 (logical) cores present.
The workers in your code keep a (physical) core entirely occupied, which is something that HT can't deal with very well, so the performance when keeping all 4 logical cores busy will be worse than when you keep only the 2 physical cores busy.
My hardware (quad core, so 4 physical and 8 logical cores) shows the same pattern:
8 workers:
Function #5: 926ms
Function #3: 916ms
Function #1: 928ms
Function #4: 895ms
Function #7: 934ms
Function #6: 905ms
Function #8: 928ms
Function #2: 928ms
4 workers:
Function #3: 467ms
Function #2: 467ms
Function #1: 473ms
Function #4: 472ms
That said, the rule of thumb of making the number of workers equivalent to the number of logical cores in your hardware still makes sense if your workers are I/O bound (which most Node apps are).
If you really want to perform heavy, blocking, calculations, count one physical core per worker.
Me and my friend are making a node.js game, and we have been testing cpu, but after profiling this process called zlib is sucking most of the CPU/RAM
3 clients connected to a game is fine, but when 12~13 players are connected it uses 58% where zlib is using about 30% of this cpu.
inclusive self name
ticks total ticks total
64775 58.5% 64775 58.5% /lib/x86_64-linux-gnu/libc-2.19.so
25001 22.6% 224 0.2% LazyCompile: *callback zlib.js:409
//this one is a different zlib
7435 6.7% 82 0.1% LazyCompile: ~callback zlib.js:409
Is there any way to decrease the cpu usage from this? or is there a reason why it is increasing so much.
I have done some reading and I am told it is from socket.io so here is our section of socket sending most of the data.
for (var i = 0; i < users.length; i++) {
if (u.room == users[i].room && users[i].x + users[i].radius >= u.x - u.screenWidth / 2 - 20 && users[i].x - users[i].radius <= u.x + u.screenWidth / 2 + 20 && users[i].y + users[i].radius >= u.y - u.screenHeight / 2 - 20 && users[i].y - users[i].radius <= u.y + u.screenHeight / 2 + 20) {
if (users[i].id == u.id) {
visiblePlayers.push({
x: users[i].x,
y: users[i].y,
angle: users[i].angle,
hue: users[i].hue,
radius: users[i].radius,
squeeze: users[i].squeeze,
name: users[i].name,
dead: users[i].dead,
isPlayer: true,
kills: users[i].kills
});
} else {
visiblePlayers.push({
x: users[i].x,
y: users[i].y,
angle: users[i].angle,
hue: users[i].hue,
radius: users[i].radius,
squeeze: users[i].squeeze,
name: users[i].name,
dead: users[i].dead
});
}
// SEND DYING INFO: (FOR OFFLINE ANIMATION):
if (users[i].dying) {
visiblePlayers[visiblePlayers.length - 1].dying = true;
}
}
}
var visibleEnergy = [];
for (var i = 0; i < energies.length; i++) {
if (u.firstSend || (energies[i].updated && energies[i].room == u.room)) {
var anim = energies[i].animate;
if (u.firstSend)
anim = true;
visibleEnergy.push({
x: energies[i].x,
y: energies[i].y,
radius: energies[i].radius,
index: i,
animate: anim,
hue: energies[i].hue,
room: energies[i].room
});
}
}
// SEND PLAYER UPDATES TO CLIENTS:
sockets[u.id].emit('serverTellPlayerMove', visiblePlayers,
visibleEnergy);
Zlib is one of out problems, but also if there are any other optimisation methods to decrease server CPU.
Extra: the formatting of checking users has been answered in StackExchange at https://codereview.stackexchange.com/questions/107922/node-js-cpu-issue. We just need help if there are any methods to decrease the zlib cpu usage.
Thanks.
zlib is a widely used compression library. The most likely reason you're seeing lots of CPU usage there is a lot of compressed network traffic. That's desirable because CPU is generally 'cheaper' than network traffic. Other areas that are likely candidates for heavy zlib usage would be compressed files on disk being loaded through node.js's zlib module, or perhaps if you're doing server-side decompression of PNG images which also commonly uses zlib.
If you find the zlib 'tax' is too much for your particular application, perhaps you could offload the decompression to another process. You could use a few node.js services in front of your main server whose purpose is purely to connect to clients. Then have them talk to your server in uncompressed, 'raw' format which your server has an easier time understanding.
You don't mention if this is actually causing you problems yet. I wouldn't necessarily worry about it yet. This kind of single process scalability is a downside of using node.js, and the only solution is to split up your service into smaller parts that can be spread across multiple cores or even multiple servers.
I have an object that contains data relative to the user's UI. For the moment, the data comes in the form of json, I run JSON.parse to evaluate the object and generate some HTML accordingly.
I'm thinking of storing the json string in the local session storage (not the local storage) and calling it each time the UI needs to be updated with new HTML. This way, if the user opens another page in the same browser, all the HTML of all the pages will be the same.
For now, the object is stored in memory and the interactions between the user's actions on the UI and the object's modifications run fast. If I serialize it and store in the session storage, will the browsers store the data in RAM or on the hard drive and slow down the page?
I did this small test FF 32, Chrome 37, IE 11. Just for fun.
console.clear();
var s = new Date();
for(var i=0; i < 100000; i++)
{
sessionStorage.item = i.toString();
}
var e = new Date();
console.log("session: " + (e - s) + " ms");
s = new Date();
var mem;
for(var i=0; i < 100000; i++)
{
mem = i.toString();
}
e = new Date();
console.log("mem: " + (e - s) + " ms");
s = new Date();
for(var i=0; i < 100000; i++)
{
localStorage.item = i.toString();
}
e = new Date();
console.log("local: " + (e - s) + " ms");
console.log('Done');
FF
session: 830 ms
mem: 92 ms
local: 1156 ms
Chrome
session: 2852 ms
mem: 147 ms
local: 2893 ms
IE
session: 977 ms
mem: 81 ms
local: 15062 ms
Safari (different device)
session: 380 ms
mem: 21 ms
local: 248 ms
After finishing test browser's window got frozen for few seconds and CPU + Disk activity increased (caused by localStorage).
Taken from the html5 spec:
The lifetime of a browsing context can be unrelated to the lifetime of the actual user agent process itself, as the user agent may support resuming sessions after a restart.
Which means browsers may store this information to disk.
Performance will be up to browser specific and OS specific implementations. However, fetching this information is very unlikely to be a bottleneck.