Three.js: Managing memory when using a lot of textures - javascript

I am building a relatively simple Three.js Application.
Outline: You move with your camera along a path through a "world". The world is all made up from simple Sprites with SpriteMaterials with transparent textures on it. The textures are basically GIF images with alpha transparency.
The whole application works fine and also the performance is quite good. I reduced the camera depth as low as possible so objects are only rendered quite close.
My problem is: i have many different "objects" (all sprites) with different many textures. I reuse the Textures/Materials reference for the same type of elements that are used multiple times in the scene (like trees and rocks).
Still, i'm getting to the point where memory usage is going up too much (above 2GB) due to all the textures used.
Now, when moving through the world, not all objects are visible/displayed from the beginning, even though i add all the sprites to the scene from the very start. Checking the console, the objects not visible and its textures are only loaded when "moving" further into the world when new elements actually are visible in the frustum. The, also the memory usage goes gradually up and up.
I cannot really us "object pooling" for building the world due to its "layout" lets say.
To test, i added a function that removes objects from the scene and disposes their material.map as soon as the camera passed by. Sth like
this.env_sprites[i].material.map.dispose();
this.env_sprites[i].material.dispose();
this.env_sprites[i].geometry.dispose();
this.scene.remove(this.env_sprites[i]);
this.env_sprites.splice(i,1);
This works for the garbage collection and frees up memory again. My problem is then, when moving backwards with the camera, the Sprites would need to be readded to the scene and the materials/texture loaded again, which is quite heavy for performance and does not seem the right approach to me.
Is there a known technique on how to deal with such a setup in regards to memory management and "removing" and adding objects and textures again (in the same place)?
I hope i could explain the issue well enough.
This is how the "World" looks like to give you an impression:

Each sprite individually wastes a ton of memory as your sprite imagery will probably rarely be square and because of that there will be wasted space in each sprite, plus the mipmaps which are also needed for each sprite take a lot of space. Also, on lower end devices you might hit a limit on the total number of textures you can use -- but that may or may not be a problem for you.
In any case, the best way to limit the GPU memory usage with so many distinct sprites is to use a texture atlas. That means you have at most perhaps a handful of textures, each texture contains many of your sprites and you use distinct UV coordinates within the textures for each sprite. Even then you may still end up wasting memory over time due to defragmentation in the allocation and deallocation of sprites, but you would be running out of memory far less quickly. If you use texture atlases you might even be able to load all sprites at the same time without having to deallocate them.
If you want to try and tackle the problem programmatically, there's a library I wrote to manage sprite texture atlases dynamically. I use it to render text myself, but you can use any canvas functions to create your images. It currently uses a basic Knapsack algorithm (which is replaceable) to manage allocation of sprites across the textures. In my case it means I need only 2 1024x1024 textures rather than 130 or so individual sprite textures of wildly varying sizes and that really saves a lot of GPU memory.
Note that for offline use there are probably better tools out there that can generate a texture atlas and generate UV coordinates for each sprite, though it should be possible to use node.js together with my library to create the textures and UV data offline which can then be used in a three.js scene online. That would be an exercise left to the reader though.
Something thing you can also try is to pool all your most common sprites in your "main" texture atlases which are always loaded; load and unload the less commonly used ones on the fly.

Related

How to mass render differently scaled shape in webgl?

I'm trying to render a bunch of trees that are all different sizes composed of two cubes (one for the leaves and one for the trunk).
If I create a vertex buffer for each tree, bind the buffer, and make a call to draw for each individual tree, the program will slow down immensely.
I faced a similar problem for terrain and solved it as follows:
Generate all the terrain at the beginning, place all vertices in a huge position buffer, and send the entire buffer to be drawn at once. This made the program run way faster than when I was sending each triangle of the terrain individually.
However, this process can't really be applied to the trees, because the number of trees is continuously changing (some die and some are born in each turn. They also grow in size as they get older, from a sapling to a huge oak).
This means I would have to remake the huge position buffer each frame, which is impractical.
Is there a way I could use the same position buffer for each tree, but just scale it differently? Maybe in the shader (I'm woefully confused by the code in the shader)?
If such a thing is possible, could it also be used to translate and rotate the same shape many times? If so, then I could apply that to how my terrain is rendered, making terraforming possible (if I tried to do that now then I'd have to remake the terrain vertex position buffer each frame terraforming is taking place, which is way too slow).
One constraint is that I do not want to use three.js. I rewrote my whole program using three.js and it was much, much slower, and much uglier since the way I was previously using textures couldn't be used with three.js. I would really appreciate an answer that sticks with just the baseline webgl and javascript, since it leaves me with more control over optimizing my final code.

Slow performance with lots of images in threejs

I am working on a 3D environment in threejs and am running into some lagging/performance issues because I am loading too many images in my scene. I am loading about 300 500x500 .pngs (I know thats a lot) and rendering them all at once. Naturally it would seem that I would get much higher performance if didn't render all of the images in the scene at once, but rather, only when my character/camera were within a certain distance to an object that requires an image to be used in its material. My question is what method would be best to use?
Load all of the images and map them to materials that I save in an array. Then programmatically apply the correct materials to objects that I am close to from that array and apply default (inexpensive) materials to objects that I am far away from.
Use a relatively small maximum camera depth in association with fog to prevent the renderer from having to render all of the images at once.
My main goal is to create a system that allows for all of the images to be viewed when the character is near them while also being as inexpensive on the user's computer as possible.
Perhaps there is a better method then the two that I have suggested?
Thanks!

Processing.js improve draw performance

I use processing.js to draw some shapes on a canvas.
If I draw more than 100-200 circles which are constantly moving on the canvas the FPS drops significantly. If I remove the drawing the FPS goes back to 60 so the other tasks do not affect the FPS.
fill(mR,mG,mB);
ellipse(mX,mY,mRadius,mRadius);
I know, for example, that for complex shapes you can draw them onto another canvas in memory to cache them and then simply draw that canvas onto the main one.
Is there any way to improve this drawing performance?
Some links that may help:
http://www.html5rocks.com/en/tutorials/canvas/performance/
http://ramkulkarni.com/blog/improving-animation-performance-in-html5-canvas-part-ii/
But I couldn't use any of these in my context, or to integrate them with Processing.js.
EDIT:
Actually it looks like it can handle ~1000shapes before the fps start to drop. Tested on Google Chrome with GPU rendering enabled on an i5 processor and GTX660 gpu.
http://jsfiddle.net/Lfmfz/2/
I guess that's already pretty good, but is there any way to squeeze some extra FPS?
Make sure you're using for loops or something similar to render these shapes. If you don't, naturally, it will be slower.
If you can get away without calling fill every ellipse, that will improve runtime. If you can, try putting all the red shapes together, all the blue shapes together, et cetera.
Get rid of unneccessary computations. For instance, if you have the ellipse()s positioned by random() computations or new Random(1) and nextGaussian() together (which is extremely slow on the last one), then create local variables or use Math.random() alone (which seems to be faster as far as I can see).
You could use anti-aliased imageData, but that may not be supported on all devices. I use that, but I am not sure about runtime on HTML, and all those for loops could get rather... messy.
TL;DR
Don't use too many computations, don't do anything redundantly, and especially do not call ellipse over and over and over and over and over and over and over and over and over..... without iterating.

What's the fastest way to draw to an HTML 5 canvas?

I'm investigating the possibility of producing a game using only HTML's canvas as the display media. To take an example task I need to do, I need to construct the game environment from a number of isometric tiles. Of course, working in 2D means they by necessity come in rectangular packages so there's a large overlap between tiles.
I'm old enough that the natural solution to this problem is to call BitBltMasked. Oh wait, no, an HTML canvas doesn't have something as simple and as pleasing as BitBlt. It seems that the only way to dump pixel data in to a canvas is either with drawImage() which has no useful drawing modes that ignore the alpha channel or to use ImageData objects that have the image data in an array.. to which every. access. is. bounds. checked. and. therefore. dog. slow.
OK, that's more of a rant than a question (things the W3C like tend to provoke that from me), but what I really want to know is how to draw fast to a canvas? I'm finding it very difficult to ditch the feeling that doing 100s of drawImages() a second where every draw respects the alpha channel is inherently sinful and likely to make my application perform like arse in many browsers. On the other hand, the only way to implement BitBlt proper relies heavily on a browser using a hotspot-like execution technique to make it run fast.
Is there any way to draw fast across every possible implementation, or do I just have to forget about performance?
This is a really interesting problem, and there's a few interesting things you can do to solve it.
First, you should know that drawImage can accept a Canvas, not just an image. The "sub-Canvas"es don't even need to be in the DOM. This means that you can do some compositing on one canvas, then draw it to another. This opens a whole world of optimization opportunities, especially in the context of isometric tiles.
Let's say you have an area that's 50 tiles long by 50 tiles wide (I'll say meters for the sake of my own sanity). You might divide the area into 10x10m chunks. Each chunk is represented by its own Canvas. To draw the full scene, you'd simply draw each of the chunks' Canvas objects to the main canvas that's shown to the user. If only four chunks (a 20x20m area), you would only perform four drawImage operations.
Of course, each of those individual chunks will need to render its own Canvas. On game ticks where nothing happens in the chunk, you simply don't do anything: the Canvas will remain unchanged and will be drawn as you'd expect. When something does change, you can do one of a few things depending on your game:
If your tiles extend into the third dimension (i.e.: you have a Z-axis), you can draw each "layer" of the chunk into its own Canvas and only update the layers that need to be updated. For example, if each chunk contains ten layers of depth, you'd have ten Canvas objects. If something on layer 6 was updated, you would only need to re-paint layer 6's Canvas (probably one drawImage per square meter, which would be 100), then perform one drawImage operation per layer in the chunk (ten) to re-draw the chunk's Canvas. Decreasing or increasing the chunk size may increase or decrease performance depending on the number of update you make to the environment in your game. Further optimizations can be made to eliminate drawImage calls for obscured tiles and the like.
If you don't have a third dimension, you can simply perform one drawImage per square meter of a chunk. If two chunks are updated, that's only 200 drawImage calls per tick (plus one call per chunk visible on the screen). If your game involves very few updates, decreasing the chunk size will decrease the number of calls even further.
You can perform updates to the chunks in their own game loop. If you're using requestAnimationFrame (as you should be), you only need to paint the chunk Canvas objects to the screen. Independently, you can perform game logic in a setTimeout loop or the like. Then, each chunk could be updated in its own tick between frames without affecting performance. This can also be done in a web worker using getImageData and putImageData to send the rendered chunk back to the main thread whenever it needs to be updated, though making this work seamlessly will take a good deal of effort.
The other option that you have is to use a library like pixi.js to render the scene using WebGL. Even for 2D, it will increase performance by decreasing the amount of work that the CPU needs to do and shifting that over to the GPU. I'd highly recommend checking it out.
I know that GameJS has blit operations, and I certainly assume any other html5 game libraries do as well (gameQuery, LimeJS, etc etc). I don't know if these packages have addressed the specific array-bounds-checking concern that you had, but in practice their samples seem to work plenty fast on all platforms.
You should not make assumptions about what speedups make sense. For example, the GameJS developer reports that he was going to implement dirty rectangle tracking but it turned out that modern browsers do this automatically---link.
For this reason and others, I suggest to get something working before thinking about the speed. Also, make use of drawing libraries, as the authors have presumably spent some time optimizing performance.
I have no personal knowledge about this, but you can look into the appMobi "direct canvas" HTML element which is allegedly a much faster version of normal canvas, link. I'm confused about whether this works in all browsers or just webkit browsers or just appMobi's own special browser.
Again, you should not make assumptions about what speedups make sense without a very deep knowledge of web browser internal processes. That webpage about "direct canvas" mentions a bunch of things that slow down canvas-drawing: "Reflowing text, mapping hot spots, creating indexes for reference links, on and on." Alpha-blending and array-bounds-checking are not mentioned as prominent causes of slowness!
Unfortunately, there's no way around the alpha composition overhead. Clipping may be one solution, but I doubt there would be much, if any, performance gain. Not to mention how complicated such a route would be to implement on irregular shapes.
When you have to draw the entire display, you're going to have to deal with the performance hit. Although afterwards, you have a whole screen's worth of pre-calculated alpha imagery and you can draw this image data at an offset in one drawImage call. Then, you would only have to individually draw the new tiles that are scrolled into view.
But still, the browser is having to redraw each pixel at a different location in the canvas. Which is quite expensive. It would be nice if there was a method for just scrolling pixels, but no luck there either.
One idea that comes to mind is that you could implement multiple canvases, translating each individual canvas instead of redrawing the pixels. This would allow the browser to decide how to redraw those pixels, in a more native way, at least in theory anyway. Then you could render the newly visible tiles on a new, or used/cached, canvas element. Positioning it to match up with the last screen render.
But that's just my two blits... I mean bits... duh, I mean cents :]

Break the scene to several canvases or keep redrawing one?

Intent on creating a canvas-based game in javascript I stand before a choice:
Should I performance-wise keep all the stuff happening in the screen in one canvas (all the moving characters, sprites) and redraw it at constant rate of, say, 60 FPS or should I break the scene into several smaller canvases thus removing the need of redundant redrawing of the scene? I could even create separate canvas elements for the characters' limbs and then do most of the animation by simply manipulating the CSS of the given canvas element (rotation, positioning, opacity).
To me the latter sounds way more plausible and easier to implement, but is it also faster? Also, shouldn't I perhaps use SVG, keep the characters and sprites as elements inside of it and manipulate their XML and CSS properties directly?
So what do you think is the most fitting solution to a scene with severals sprites and characters:
One canvas object redrawn manually (and wastefully) at FPS rate
Several canvas elements, redrawn manually in a more reasonable fashion
Structured vector graphics document like SVG / VML manipulated via DOM
I am mainly concerned about the performance differences, but the legibility of the logical code behind is also of interest (I, having already worked with canvas before, am for example fairly sure that the redrawing function for the entire canvas would be one hard-to-maintain beast of a script).
DOM manipulations are slow in comparison to GPU-accelerated canvas operations, so I would stay away from SVG and VML.
As far as structuring your canvas code goes, it certainly doesn't make sense (especially for performance reasons) to clear and re-draw the entire canvas because the player moved or performed an action. Based on your description here, I'm guessing that your game will be 2D. These types of games lend themselves extremely well to layering unless you're doing something rather complex like Paper Mario. You should be looking at the issue from an object-oriented viewpoint and encapsulating your drawing procedures and objects together as appropriate.
For instance, create a player object that maintains a small canvas representing the character. All the logic needed to maintain the character's state is kept within the object and any changes made to it need not worry about other components of the game's visual representation. Likewise, do the same for the background, user interface, and anything else you can abstract into a layer (within reason). For example, if you're doing a parallax game, you might have a foreground, background, character, and user interface layer.
Ultimately you will need to maintain the state of the different components in your game individually. Player animations, background clouds moving, trees swaying, etc. will all be managed by appropriate objects. Since you will already have that type of code structure setup, it makes sense to just maintain major components on separate canvas elements and composite them together as needed for better performance. If the character moves in the bottom left corner of a scene with a static background, you don't need to re-draw the top right corner (or 95% of the scene, for that matter). If you're considering full-screen capabilities, this is definitely a performance concern.
There's a rule in programming, that you should never try and optimize something before you have a speed problem with it. You'll spend more time trying to figure out how to code something in the best way than to actually code, and will never finish anything.
Draw them all on your canvas at a fixed rate. That's how it's done. If you start creating a canvas for each limb and element and manipulate them using CSS, you're wasting the potential of canvas. Might as well just use images. That's how they did it before canvas. That's the problem canvas was made to solve.
If you ever encounter speed issues, then you can start hammering at them. Check out these slides for some tips (related video). This guy's blog also has some handy tips on canvas performance.

Categories