Doing a full garbage collection from Javascript in Electron (Chrome / nodejs) - javascript

I am developing a voxel engine in Javascript using Threejs, and I've run into some memory usage problems. The intended environment is within an Electron application, which means I can launch the program with flags (--expose-gc) that enable manual garbage collection with gc().
I noticed that when running the gc() function, much less memory was freed compared to when I clicked the "collect garbage" button in the Memory section of Chrome's developer tools. For example, if the program initially used 2000 MB of memory, after gc() it would free up to 300 MB, but after clicking the aforementioned button, it would free up to 600 MB more, making it about half of what it originally was.
After some digging, I also found out about the --gc-global flag. This would've been perfect, since it behaved in the same way as a manual gc did, but it ran multiple times per second, making my program extremely laggy. I tried controlling the gc rate with --gc-interval, but it didn't affect it at all.
I also found this answer that talked about ProfilerAgent.collectGarbage();, but it always threw a reference error. Additionally, the flag it provided (--debug-devtools-frontend) wasn't recognised by Chrome. That, and the fact that I could find very little info on it outside of the answer, lead me to believe that it's an obsolete feature.
I also profiled the memory usage from the developer tools, and surprisingly, it was much lower compared to the value that Task Manager showed me: it seemed to be unaffected by the leaking memory. This has made it nearly impossible to determine where the memory could be leaking from.
I should also make it clear that while Chrome does garbage-collect on its own, it's far too infrequent, and it rarely does a full gc, meaning that memory usage is about twice of what it could be.
In summary, is there a way to trigger a full garbage collection from either Chrome or nodejs/Electron? Obviously, I'm not concerned about cross-browser support, so any advice would be appreciated.

It turns out that self.gc only collects garbage in the thread/worker it was called from. Running the function in all of my workers freed as much memory as the developer tools garbage collector did.

Related

Our heap grows in size and then falls off. a lot. Looking at this screenshot, does this look like a memory leak?

This happens over a period of 90ish seconds. I'm trying to isolate the cause and I can't even begin to figure out where to start, and i'm at the point now where I'm questioning whether this is even a problem- this seems like Chrome is just good at handling performance, rather than we're doing something right. I'm trying to decrease our JS Heap size in general but I don't know even where to start.
In summary:
Does this look like a memory leak or performance problem?
I've read and watched a bunch of videos about FINDING memory leaks but have yet to find a good example of how to isolate and solve them. Any resources-- preferably google team ones-- would be super helpful
Without knowing anything about your application it is hard to tell, but in general do 100 MB of heap space used not particularly have to be a memory leak. Where the spike is going down is just the garbage collection of the Javascript Engine hitting and freeing all not longer used memory. We have a simple desktop application here in development that already uses 75 MB of heap space when it is just idling without doing any rerendering to hold all the states. For your comparison.
You can also check for sources like
https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/
and see if you do things that can cause memory leaks.
Check also:
Finding JavaScript memory leaks with Chrome

Is this js heap graph worrying? How can I fix it?

I have recorded the performances of an angular 4.4 app and I think that what the Chrome dev tools returned me about the js heap could be worrying, but I honestly lack on this subject.
I don't understand the straight drop at ~20000ms, the straight line soon after and the other drop at ~60000ms: what are they due to? Are those behaviours normal or do they means that something should be fixed?
The incline means that the page was allocating memory in the JS heap. This is normal.
The drops mean that the browser freed up memory in the JS heap that was no longer needed. This is called garbage collection. That's normal, too. Nothing alarming about that.
In general, if you see that the total amount of memory is progressively increasing after each garbage collection event, then that's a warning sign that you have a memory leak. The memory leak pattern usually looks like this:
Source
As you can see from the graph, if you leave the page running for long enough, eventually it will use up all of the computer's memory, causing the computer to run slowly, or crash.
See Fix Memory Problems for more techniques for analyzing memory usage.

Visual Profiler for Node.js

I've tried so many profilers for node I've lost count. I've never seen a profiler that gives you this:
This image shows second-by-second usage of CPU (top and center) and memory (bottom). I can click on a single "frame" (a dividend of a second) to see exactly which functions executed on that frame and what memory was allocated and deallocated (GC'd). This is Adobe Scout for Flash/AS3.
I need to find a ghost (a memory leak :), and I've successfully used the above interface hundreds of times to eliminate unwanted allocations and debug why memory doesn't get freed when it should.
How do I find which part of my app is allocating memory on a visual timeline? I need a timeline to see specifically which part of my app is allocating memory and why. Right now everything happens so fast I can't use the "objects currently in memory" panel to do anything useful. And comparing "heap snapshots" is harder than using a timeline. Web-based or app is fine. I use Windows 7.
I use pm2 as a process manager and they have a dashboard service keymetrics. You may have a look to see if match your need. :)

How to find JS Memory Leaks?

I have struggled around with the heap profiler in chrome but it is very confusing. Especially if there are minimized libraries inside. But even the DOMElements views, elements which may not be removed are very unclear presented.
Are there any tips how to use the heap dump in chrome to find JS code that leads to memory leaks, code that cannot be cleaned by GC... and how to find elements messing around even if removed from dom?
In other words, how to read heap dump in chrome correctly? Dominators View? Comparison?
Chrome now offers much better tools to find memory leaks, than at the time of most answers.
Here is to find memory leaks in javascript with a recent Chrome browser:
Press F12 to open the developer tools and go to the Memory Tab.
Pick a feature or a part of your app that you want to inspect for leaks. For example, when a dialog is opened and closed again, the memory used by it should be released.
Do the action (for example opening a dialog) you want to check for memory leaks once, so potential global services can be loaded. This prevents these objects, that are intentionally preserved from showing up as leaks.
Now select Record Allocation Timeline and press Start. Repeat the action you want to check for leaks a few times. So for example open a dialog, close it and repeat. While you do this Chrome draws the timeline with partially grey or blue bars. Usually you see a bar for each time you performed the action on your page. When the bar from several previous iterations of the action stays partially blue it usually means there is a memory leak. The blue part of the bar represents memory that was allocated at this time and has not yet been released again. Stop the recording by pressing the red dot on the top left of the developer tools.
When you see potential leaks you have to inspect this part of the timeline to find the source. Select a part of the timeline that is a few iterations of your actions in the past. And Chrome will show a list of object-types that are still present in the memory. The retained size column gives you an impression how much memory is still used. Browse into one of the object-types and select an object. If you do that, the list of retainers will appear below.
The list of retainers shows the "parent" objects that reference the selected object. Now you need to look at the retainers and your code to understand why the memory has not been released. For example in the image you see the object of the type scope. The second line says the scope is "context in initFormat()". The problem was that initFormat was an event listener that was not unbound after a dialog was left.
After you fixed your code check if the problem has been solved. Refresh the page and repeat the steps 3 to 6 again. If you have never checked for memory leaks before it is not unlikely that you find multiple problems.
Additional hints:
Sometimes there are caches that retain a part of the memory. Usually you can ignore them.
When you see the HTMLDivElement or other DOM elements in the list of object-types have a look. If the objects in this list are highlighted red it means they are no longer present in your page. This means they must be reference somewhere in the code. You may have forgotten to unbind an event listener.
Read about memory leaks in general, so you can identify them quicker in your code.
In Chrome developer tools, there is a Timeline - Memory tab:
We can watch the memory occupied by it.
There is also Profiles - Memory, where we can take a snapshot and see what’s inside. Snapshots can be compared to each other:
Most of time, it doesn’t tell you anything. But at least you can see which objects are piling up, and probably the structure of the leak.
Other way is using 'Task Manager'
here is an article regarding this:
http://www.javascriptkit.com/javatutors/closuresleak/
Google open sourced a tool for this purpose, leak-finder-for-javascript. They also proposed a method so-called the three snapshot technique (also see a brief description in this article).
First paragraph of the leak-finder link
Note: jsleakcheck is no longer supported! It was working against an unofficial and unstable Dev Tools protocol for taking heap snapshots.
The protocol is being worked on, and it is not stable enough so that I
could keep jsleakcheck working with various Chrome versions. In
addition, a lower level compatibility tool,
remote_inspector_client.py, which jsleakcheck was using to communicate
with Dev Tools, got removed.
I found this article very insightful:
http://addyosmani.com/blog/taming-the-unicorn-easing-javascript-memory-profiling-in-devtools/
It does cover the chrome developer tools extensively and explains very well how to go about when your application seems to have memory issues.
Quoted from JavaScript Kit:
If you are developing client-side re-usable scripting objects, sooner or later you will find yourself spotting out memory leaks. Chances are that your browser will suck memory like a sponge and you will hardly be able to find a reason why your lovely DHTML navigation's responsiveness decreases severely after visiting a couple of pages within your site.
A Microsoft developer Justin Rogers has described IE leak patterns in his excellent article (from web.archive.org).
In this article, we will review those patterns from a slightly different perspective and support it with diagrams and memory utilization graphs. We will also introduce several subtler leak scenarios. Before we begin, I strongly recommend you to read that article if you have not already read.
Why does the memory leak?
The problem of memory leakage is not just limited to Internet Explorer. Almost any browser (including but not limited to Mozilla, Netscape and Opera) will leak memory if you provide adequate conditions (and it is not that hard to do so, as we will see shortly). But (in my humble opinion, ymmv etc.) Internet Explorer is the king of leakers.
Don't get me wrong. I do not belong to the crowd yelling "Hey IE has memory leaks, checkout this new tool [link-to-tool] and see for yourself". Let us discuss how crappy Internet Explorer is and cover up all the flaws in other browsers".
Each browser has its own strengths and weaknesses. For instance, Mozilla consumes too much of memory at initial boot, it is not good in string and array operations; Opera may crash if you write a ridiculously complex DHTML script which confuses its rendering engine.
Although we will be focusing on the memory leaking situations in Internet Explorer, this discussion is equally applicable to other browsers.
continue reading...
Here is a very good post about how to find memory leaks using the Google Developper Tools: http://gent.ilcore.com/2011/08/finding-memory-leaks.html
Here is another good web page about that : http://javascript.crockford.com/memory/leak.html
I don't see mentioned window.performance.memory, which gives you the run-time ability to monitor and take action based on memory usage.
window.performance.memory:
MemoryInfo {
totalJSHeapSize: 7084834,
usedJSHeapSize: 6486770,
jsHeapSizeLimit: 4294705152
}
For a handy percentage, use this:
window.performance.memory.usedJSHeapSize / window.performance.memory.jsHeapSizeLimit
https://developer.mozilla.org/en-US/docs/Web/API/Performance/memory

How can I avoid causing memory leaks in Firefox?

It seems that there is a lot of information on memory leaks in IE and how web developers can avoid them, but I can't find much on avoiding leaks in FF. I've found lots of random tips on how end users can tweak their preferences, or tips for extension developers, but little on what I can do as a web developer to make sure my pages don't leak. Am I missing something? It seems lazy to just blame it on the user and say "you've got too many extensions". Or are the major patterns the same as in IE -- circular references and all that?
Also, if anyone knows of any tools to troubleshoot leaks in FF, that would be great. I found this:
https://addons.mozilla.org/en-US/firefox/addon/2490/
But it's apparently just for chrome and extension development.
Outside of the design patterns to favour the only truely safe way is to test your pages thoroughly. To monitor the memory usage of the browser Task Manager is alright but Process Explorer provides more accurate results.
JavaScript is one cause of memory leaks but be careful with flash movies on pages too. Our content team added a movie from our design department that used a thrid party transition affect and this swallowed 10Mb every 20s or so. Just watching the movie loop through it was obvious in TaskManager to see the memory jump when the affect occured and the it never quite release it all back.
You can force to run a Garbage Collector in FireFox. The Garbadge Collector will destroy & release objects that are not used anymore. The only possibility of "Leaking memory" with a Garbage Collector is not a "leak" but a reference which makes no sense: remove all references to objects that you don't want to use.
Read more on this page:
http://adblockplus.org/blog/different-ways-to-force-garbage-collection
A bunch of what you read about how to avoid memory leaks in browsers is about how to avoid things that cause the browser to fail to reclaim memory that it should reclaim.
However, a more substantial problem in many cases is about Web pages holding on to objects that they don't need anymore. It's only the browser's job to reclaim things that are no longer "reachable" -- that is, things that the script / page can't get to anymore. If you're accumulating objects in an array and not removing them when you're done with them, memory usage will go up as the array gets bigger, and there's nothing the browser can do about that.
To phrase that another way: that's an issue of a memory leak in the Web page rather than in the browser. And the tool you want for that is a memory profiling tool to examine the objects that are reachable in your page, so you can tell whether there's stuff in there that you should no longer be holding on to. Writing such a tool for Firefox has been on my list of things to do for a while, but I haven't gotten around to it yet. I think there might be some ongoing work on writing one that integrates into Firebug.
I don't know if there is specific information for Firefox, but the generic tips still apply.
I suggest that you closely examine all loops and recursive functions. Re-use existing objects over creating new ones, and ensure that temporary objects and primitives exit scope so that they can be freed.

Categories