In one of our apps, we're finding that when usage spikes, RSS (as reported by top(1) for example) increases, hits a plateau and starts oscillating, but never comes back down. After a bit of a break, stressing the server again results in another higher plateau. This trend continues into the many gigabytes of RAM.
At first we suspected a memory leak on the Javascript side, and so started tracking the statistics returned by process.memoryUsage():
A few phases here:
After starting the server, we hit it with 100 concurrent long running requests for a few minutes and then left the server running for the evening. RSS stayed at around 320MB for the night; and heap usage just kept oscillating. We were emitting traces every minute so maybe this explained the movement in the heap.
At around 9a, we hit it with another round of concurrent requests and transitioned to a workflow involving 2 threads of rolling long running requests (downloads of 10MB of data). We kept these running until around 2p and stopped them. During this phase, after the initial bump in RSS, it again started to level off and stay between around 400 and 450MB.
The last bit is another round of concurrent requests. Here, RSS grew to between 480MB and 500MB
I'm wondering if there are ways to figure out what's going on outside the heap that could be consuming all of this memory and not releasing it. Short of that, are there any hidden options for limiting memory usage outside of the heap?
And I have to ask. Are we making too much of this? Is this about normal for Node?
Related
Ok, so the thing is, we have multiple NodeJS web servers which need to be online all the time. But they'll not be recieving many requests, approx. 100-200 requests a day. The tasks aren't CPU intensive either. We are provisioning EC2 instances for it. So, the question is, can we run multiple nodejs processes on a single core? If not, is it possible to run more low intensity NodeJS processes than number of cores present? What are the pros and cons? Any benchmarks available?
Yes, it is possible. The OS (or VM on top of the OS) will simply share the single CPU among the processes allocated to it. If, as you say, you don't have a lot of requests and those requests aren't very CPU hungry, then everything should work just fine and you probably won't even notice that you're sharing a single CPU among a couple server processes. The OS/VM will time slice among the processes using that CPU, but most of the time you won't even have more than one process asking to use the CPU anyway.
Pros/Cons - Really only that performance might momentarily slow down if both servers get CPU-busy at the same time.
Benchmarks - This is highly dependent upon how much CPU your servers are using and when they try to use it. With the small number of requests you're talking about and the fact that they aren't CPU intensive, it's unlikely a user of either server would even notice. Your CPU is going to be idle most of the time.
If you happen to run a request for each server at the exact same moment and that request would normally take 500ms to complete and most of that was not even CPU time, then perhaps each of these two requests might then take 750ms instead (slightly overlapping CPU time that must be shared). But, most of the time, you're not even going to encounter a request from each of your two servers running at the same time because there are so few requests anyway.
I'm trying to figure out some performance issues that our node.js app is having. We're on 8.11.1. I haven't dived into heap dumps yet, that's the next step but before I do, I thought I'd seek out another opinion or two. I spun up productions servers at the moments where that line is vertical and since then, it seems like multiple instances are constantly running at about ~60% memory utilization.
If this is not in fact a memory leak, what are some things that we can do to get overall memory utilization down to a lower level?
Thanks!
I'm working on a pretty simple web application that displays top users hitting a web page. I've written some code in d3 to convert the data to a bar graph and have written a simple php script to pull the data from a database. I use an ajax request to pull the data once every 5 seconds and update the bar graph. Without fail if I leave the page open in the background it will get to the point of the old
aw snap google chrome has run out of memory
I've gone through a bunch of sites on memory leaks and done what I could to prevent them but there is a decent enough chance I messed something up. The problem is other than that error coming without fail on this page (and I've written plenty of javascript applications where this doesn't happen) there is absolutely no evidence the leak is happening. The data I'm retrieving every 5 seconds is 212kb, and when I do a heap recording heap memory peaks at 25mb and doesn't really increase (it generally bounces between 10 and 25mb), so it seems like that 212kb is being garbage collected and is not accumulating in the heap. Similarly I've looked at the task manager and when I make that the only tab open there seems to be a good amount of fluctuation but definitely not a trend upwards. I've taken heap snapshots and they tend to be in the 10-15mb range and I really don't understand how to read the snapshot, but here's what it looks like after running for 15 minutes or so:
It's just getting extremely frustrating, I've spent 20-30 hours on this but it seems like a case of a watched pot never boils. If I look for evidence of it happening I can never find any, but if I leave it open while I leave my computer for a few hours, without fail the page crashes. It's almost like the garbage collector just waits for me to leave my computer before deciding to just stop running.
Any thoughts on what could be the culprit here or next steps to attempt to identify the culprit? This is my first experience with a memory leak and it's driving me crazy. Thanks.
Relative newbie to nodejs here trying to figure out a performance issue in a newly built application.
I am running a performance test on my node 0.12.7 app and I find the server hanging intermittently. It needs a restart upon reaching that state. After confirming there is no memory leak (the process heap does not exceed 500 MB whereas the default max-heap size is 1.4GB I understand), we moved to checking CPU profile. I have used this snippet of code with a dependency on v8-profiler to get regular profiles
Here is one of the charts that we encountered from jmeter (although the server didn't hang)
We plotted flame graphs in Chrome by loading the CPU profiles. I was expecting to find the JS stuck somewhere at this point, but I find that exactly in that time range, the node server is idle for a long time. Could anyone help me understand what could be the probable causes for the server to stay idle while being bombarded with client requests, and eventually recovering to continue operations after 10 minutes?
I unfortunately have lost the data to check if the responses between 16:48:10 and 16:57:40 are error or success, but it is very likely that they are error responses from the proxy since node didn't have a care in the world.
Here are the flame charts seen in Chrome
Before 16.47 hrs,
Around 16.47 hrs
A couple of minutes after 16.47 hrs
There could be multiple reasons around this.
Server is not accepting the requests. Do you see drop in throughput after you reach the peak?
Have you checked the server logs to see if any exceptions are logged?
Try plotting trends of response time and throughput for your test duration.
You may want to see any IO bound operations in your code.
Check the processor queue length. You should see it building up if processes are not getting enough CPU.
I'm getting some traffic to a server of mine and I'm not sure how to deal with this problem.
I've added the nodetime to my app, and here's the result of a heap snapshot. Retainers > Other is up to 88% from 78% (in a matter of a couple of minutes)
Overall system's free memory decrease:
It's slow, but definitely happens. The jump up around 21:20 is when I restarted the server.
The server itself is basically collecting logs: it saves incoming requests to MongoDB, reads from MongoDB once and occasionally sets Redis key. In other words, it's a pretty simple set-up.
How do I track down what what this buffer is? In addition, is there a list somewhere of basic do-not's that can cause this type of issue?
I should also mention that running stress tests with ab causes the server to consume proportionately more memory, so it's definitely a node.js issue and likely not another process that's eating up the memory.
Would it be helpful to dig through the code and rename as many anonymous functions as possible?