I have a library that renders models of specific formats.
These formats have animations, and constantly changing things like dynamic mesh colors (aka vertex color per mesh), meshes dynamically being visible or not, and many other things, all depending on the animations.
All was fine, and then I wanted to make the rendering actually fast and switched to instanced rendering.
Since then everything became a nightmare, and no matter how many designs I try, I simply cannot get it to work properly.
The code works in such a way that every mesh in every model owns a bucket object, which holds shared buffers for many instances of this mesh.
For example, every instance needs the mentioned dynamic vertex color, so the bucket holds one buffer with enough memory for some N vertex colors.
When a new instance is added, a free index into this buffer is given to it.
Every instance writes to this index as if it's a C pointer, and at the end the whole buffer is updated with the new values.
So far so good.
When an instance is removed, there's a hole in the buffer.
What I did was to "move" the last instance to the one that got deleted, essentially the same as changing pointers.
This isn't so nice, but it's fast.
Then the issues came in two ways - textures, and cameras.
Let's say I want one instance to use a different texture, this is impossible with instanced rendering directly.
So now it's split. A mesh holds an array of mesh "views", and each of these views hold the buckets.
A view can define its own texture overrides, and so every view is an additional N render calls.
This is so-so, but I couldn't think of anything else.
And finally, the actual problem I am now totally stuck on - cameras.
I want to be able to define the camera each instance uses.
It looked as if I could just add a matrix as an attribute, but that doesn't allow to change the viewport, which is kind of important.
I cannot for the life of me think of any way that actually works, that allows all of the above. Switching cameras, switching textures, and instanced rendering.
To make matters worse, I also can't think of any nice way to cull instances.
The only real way I can see to really support everything, is to dynamically keep re-creating buffers for each render call, and copying tons of data each frame, both on the CPU side and GPU side.
For example - sort all of the instances per-camera into different JS arrays (culling can be done here), create a typed array for each one, copy all of the instance data from all of the instances into this array, bind it to a buffer, and redo this for every group, for every "view", for every frame.
This is basically OOP rendering, switched to instanced rendering right at the final step, and I doubt it will be any kind of fast...
tl;dr instanced rendering is a hell to actually use. Are there any known proper techniques to use it with different textures/cameras/whatnot?
Related
So I was wondering how the .userData property worked in the Object3D class.
So this is what I understood :
It is an empty object that is not used by any internal logic of threejs, it is a safe way to park data that I want to have assigned to a given object. And that adding custom stuff to the object itself can cause problems in the event that there is a conflict.
But the only problem is I do not know when to use it or how I can implement this in my code? But can someone tell me if what I said is right?
According to the three.js documentation
.userData : Object
An object that can be used to store custom data about the Object3D. It should not hold references to functions as these will not be cloned.
and what is Object3D
This is the base class for most objects in three.js and provides a set of properties and methods for manipulating objects in 3D space.
so Object3D will be the base class for most of the objects in three.js including Cameras, Lights, Mesh, Materials etc.
Whenever you need to store some custom information about that perticular object that you might want to use for displaying or any other calculation, you will store that in .userData property.
A simple example could be that you created a geometry (a BoxGeometry), now you want to keep track of the count how many times user clicked on this geometry. This count can be kept in the .userData property. Later you can use it however you want, either to display it somewhere or do further calculations.
I am building a relatively simple Three.js Application.
Outline: You move with your camera along a path through a "world". The world is all made up from simple Sprites with SpriteMaterials with transparent textures on it. The textures are basically GIF images with alpha transparency.
The whole application works fine and also the performance is quite good. I reduced the camera depth as low as possible so objects are only rendered quite close.
My problem is: i have many different "objects" (all sprites) with different many textures. I reuse the Textures/Materials reference for the same type of elements that are used multiple times in the scene (like trees and rocks).
Still, i'm getting to the point where memory usage is going up too much (above 2GB) due to all the textures used.
Now, when moving through the world, not all objects are visible/displayed from the beginning, even though i add all the sprites to the scene from the very start. Checking the console, the objects not visible and its textures are only loaded when "moving" further into the world when new elements actually are visible in the frustum. The, also the memory usage goes gradually up and up.
I cannot really us "object pooling" for building the world due to its "layout" lets say.
To test, i added a function that removes objects from the scene and disposes their material.map as soon as the camera passed by. Sth like
this.env_sprites[i].material.map.dispose();
this.env_sprites[i].material.dispose();
this.env_sprites[i].geometry.dispose();
this.scene.remove(this.env_sprites[i]);
this.env_sprites.splice(i,1);
This works for the garbage collection and frees up memory again. My problem is then, when moving backwards with the camera, the Sprites would need to be readded to the scene and the materials/texture loaded again, which is quite heavy for performance and does not seem the right approach to me.
Is there a known technique on how to deal with such a setup in regards to memory management and "removing" and adding objects and textures again (in the same place)?
I hope i could explain the issue well enough.
This is how the "World" looks like to give you an impression:
Each sprite individually wastes a ton of memory as your sprite imagery will probably rarely be square and because of that there will be wasted space in each sprite, plus the mipmaps which are also needed for each sprite take a lot of space. Also, on lower end devices you might hit a limit on the total number of textures you can use -- but that may or may not be a problem for you.
In any case, the best way to limit the GPU memory usage with so many distinct sprites is to use a texture atlas. That means you have at most perhaps a handful of textures, each texture contains many of your sprites and you use distinct UV coordinates within the textures for each sprite. Even then you may still end up wasting memory over time due to defragmentation in the allocation and deallocation of sprites, but you would be running out of memory far less quickly. If you use texture atlases you might even be able to load all sprites at the same time without having to deallocate them.
If you want to try and tackle the problem programmatically, there's a library I wrote to manage sprite texture atlases dynamically. I use it to render text myself, but you can use any canvas functions to create your images. It currently uses a basic Knapsack algorithm (which is replaceable) to manage allocation of sprites across the textures. In my case it means I need only 2 1024x1024 textures rather than 130 or so individual sprite textures of wildly varying sizes and that really saves a lot of GPU memory.
Note that for offline use there are probably better tools out there that can generate a texture atlas and generate UV coordinates for each sprite, though it should be possible to use node.js together with my library to create the textures and UV data offline which can then be used in a three.js scene online. That would be an exercise left to the reader though.
Something thing you can also try is to pool all your most common sprites in your "main" texture atlases which are always loaded; load and unload the less commonly used ones on the fly.
I am using WebGL to do hardware skinning, however updating my model node hierarchies is causing a huge hit for performance.
Every node needs to query its current location/rotation/scale keyframes, construct a local matrix with them, and multiply it with its parent's world matrix if it has a parent.
The matrix math itself is as optimized as it gets (special variants of matrix construction based on gl-matrix).
Still, if I update many models, with tens of nodes each (some even with hundreds, sadly), this hogs all of the execution time of the browser.
I have tried using a dirty state for when nodes don't actually need updating, but simply checking if their local data changed (mostly just checking if the location or rotation changed) actually causes the same amount of processing as just calculating the matrices.
WebCL would have been ideal, but that seems to go nowhere since 2014.
I am starting to think of running it all in a shader, but I can't quite wrap my head on how to design it (e.g. storing the keyframes, which are a map of frame->data, or how to write the data back).
Another way is to cache all of the animation transformations in a texture, but this doesn't scale well. For models with a low enough amount of keyframes, this is ok, but for ones with long animations, this turns to hundreds of megabytes very fast.
This is mostly because I can't think of any way to store sparse data. If that were possible, then I could store the same amount of transformations as there are keyframes, which would not take a lot of memory (right now, I store the transformations for every single frame).
Granted, this would require to do matrix interpolation, and I am not sure how reliable that is.
Does anyone have any ideas?
I dont think its practical to offload the entire node hierarchy calculations to the GPU. The best you can do is upload the 2 absolute world transformations keyframes to GPU and let GPU interpolate between. But I am not sure if the interpolated world transformation is same as if you actually calculated via node hierarchy. If that is possible, then that would be a feasible solution. Note that you cannot interpolate between matrices either. You need to convert it in a form that support interpolation, such as with quaternions + additional data.
I actually ran into this problem as well for my project. I found the updating transformation calculation to be the most time consuming operation, despite having a full collision system + response going as well.
I solved the problem by reducing the amount of times this updating transformation need to be called. For example, if you can conclude that the entire model is out of your view frustum, then you dont need to calculate the transformations at all. This may reduce the amount of calculation you need to do to between ~1/6 to ~1/4.
Secondly, for distant objects, you dont need to update their transformation every frame. Just update their transforms every few frames or so. Remember, there's games that are shipped with only 30FPS and thus a few skipped frame for distant objects may not be noticeable.
Finally, and this may not work for Javascript at all so I didnt do (yet), is that you should store and access data in a cache coherent manner. See these slides. Could have a 10x performance increase. But again, may not work for Javascript because well, Javascript arrays are not guaranteed to pack their data sequentially.
say you have a simple raycaster. When you mouse over, the model will light up.
However, in this implementation, this model would be broken into parts, which would be parts of the model but still their own separate "models". For example, say your model happened to be a car. When you mouse over the hood, it lights up. When you mouse over the door, it lights up, etc.
I did not find any such instance of what I was speaking of in the threejs examples.
is there a way to break a full .obj model into several, individual but connected, models in three.js?
I have been working with threejs for quite a time but I do not think that this is possible, at least not with threejs itself (maybe WebGL has some tools which would help you to achieve this, but still, if it is a complex model such as car, the result would still be pretty terrible). But there are several workarounds.
In treejs, create your model from multiple smaller ones. For simple objects this is possible (example: instead of sphere, create two hemispheres and place them next to each other).
Instead of using raycaster, use point light. This will cast the light on the area not the object, therefore if you target one big object, you will end up lighting up only the part of the object, based on the intensity and distance of the point light.
If you have complex model such as car. Load it with some 3D modelling program (Blender, ...) and break it into smaller ones and then save each of them separatelly. And in your threejs code, load each one separatelly and position them the way it will look like single object. (I guess this is the only reasonable way in this case)
I am working on a 3D environment in threejs and am running into some lagging/performance issues because I am loading too many images in my scene. I am loading about 300 500x500 .pngs (I know thats a lot) and rendering them all at once. Naturally it would seem that I would get much higher performance if didn't render all of the images in the scene at once, but rather, only when my character/camera were within a certain distance to an object that requires an image to be used in its material. My question is what method would be best to use?
Load all of the images and map them to materials that I save in an array. Then programmatically apply the correct materials to objects that I am close to from that array and apply default (inexpensive) materials to objects that I am far away from.
Use a relatively small maximum camera depth in association with fog to prevent the renderer from having to render all of the images at once.
My main goal is to create a system that allows for all of the images to be viewed when the character is near them while also being as inexpensive on the user's computer as possible.
Perhaps there is a better method then the two that I have suggested?
Thanks!