I need to scale images in array form in a Web Worker. If I was outside a web worker I could use a canvas and drawImage to copy certain parts of an image or scale it.
Look like in a web worker I can't use a canvas so, what can I do? Is there any pure Javascript library that can help me?
Thanks a lot in advance.
Scaling can be done in various ways, but they all boil down to either removing or creating pixels from the image. Since images are essentially matrices (resized as arrays) of pixel values, you can look at scaling up images as enlarging that array and filling in the blanks and scaling down images as shrinking the array by leaving values out.
That being said, it is typically not that difficult to write your own scale function in JavaScript that works on arrays. Since I understand that you already have the images in the form of a JavaScript array, you can pass that array in a message to the Web Worker, scale it your scale function and send the scaled array back to the main thread.
In terms of representation I would advise you to use the Uint8ClampedArray which was designed for RGBA (color, with alpha channel) encoded images and is more efficient than normal JavaScript arrays. You can also easily send Uint8ClampedArray objects in messages to your Web Worker, so that won't be a problem. Another benefit is that a Uint8ClampedArray is used in the ImageData datatype (after replacing CanvasPixelArray) of the Canvas API. This means that it quite easy to draw your scaled image back on a Canvas (if that was what you wanted), simply by getting the current ImageData of the canvas' 2D context using ctx.getImageData() and changing its data attribute to your scaled Uint8ClampedArray object.
By the way, if you don't have your images as arrays yet you can use the same method. First draw the image on the canvas and then use the data attribute of the current ImageData object to retrieve the image in a Uint8ClampedArray.
Regarding scaling methods to upscale an image, there are basically two components that you need to implement. The first one is to divide the known pixels (i.e. the pixels from the image you are scaling) over the larger new array that you have created. An obvious way is to evenly divide all the pixels over the space. For example, if you are making the width of an image twice as wide, you want simply skip a position after each pixel leaving blanks in between.
The second component is then to fill in those blanks, which can be slightly less straightforward. However, there are several that are fairly easy. (On the other hand, if you have some knowledge of Computer Vision or Image Processing you might want to look at some more advanced methods.) An easy and somewhat obvious method is to interpolate each unknown pixel position using its nearest neighbor (i.e. the closest pixel value that is known) by duplicate the known pixel's color. This does typically result in the effect of bigger pixels (larger blocks of the same color) when you scale the images too much. Instead of duplicating the color of the closest pixel, you can also take the average of several known pixels that are nearby. Possibly even combined with weights were you make closer pixels count more in the average than pixels that are farther away. Other methods include blurring the image using Gaussians. If you want to find out what method is the best for your application, look at some pages about image interpolation. Of course, remember that scaling up always means filling in stuff that isn't really there. Which will always look bad if you do it too much.
As far as scaling down is concerned, one typically just removes pixels by transferring only a selection of pixels from the current array to the smaller array. For example if you would want to make the with of an image twice as small, you roughly iterate through the current array with steps of 2 (This depends a bit on the dimensions of the image, even or odd, and the representation that you are using). There are methods that do this even better by removing those pixels that could be missed the most. But I don't know enough about them.
By the way, all of this is practically unrelated to web workers. You would do it in exactly the same way if you wanted to scale images in JavaScript on the main thread. Or in any other language for that matter. Web Workers are however a very nice way to do these calculations on a separate thread instead of on the UI thread, which means that the website itself does not seem unresponsive. However, like you said, everything that involves the canvas element needs to be done on the main thread, but scaling arrays can be done anywhere.
Also, I'm sure there are JavaScript libraries that can do this for you and depending on their methods you can also load them in your Web Worker using importScripts. But I would say that in this case it might just be easier and a lot more fun to try to write it yourself and make it tailor-made for your purpose.
And depending on how advanced your programming skills are and the speed at which you need to scale you can always try to do this on the GPU instead of on the CPU using WebGL. But that does seem a slight overkill in this case. Also, you can try to chop your image in several pieces and try to scale the separate parts on several Web Workers making it multi-threaded. Although it is certainly not trivial to combine the parts later. Perhaps multi-threaded makes more sense when you have a lot of images that need to be scaled on the client side.
It all really depends on your application, the images and your own skills and desires.
Anyway, I hope that roughly answers your question.
I feel some specifics on mslatour's answer are needed, since I just spent 6 hours trying to figure out how to "…simply… change its data attribute to your scaled Uint8ClampedArray object". To do this:
① Send your array back from the web-worker. Use the form:
self.postMessage(bufferToReturn, [bufferToReturn]);
to pass your buffer to and from the web worker without making a copy of it, if you don't want to. (It's faster this way.) (There is some MDN documentation, but I can't link to it as I'm out of rep. Sorry.) Anyway, you can also put the first bufferToReturn inside lists or maps, like this:
self.postMessage({buffer:bufferToReturn, width:500, height:500}, [bufferToReturn]);
You use something like
webWorker.addEventListener('message', function(event) {your code here})
to listen for a posted message. (In this case, the events being posted are from the web worker and the event doing the listening is in your normal JS code. It works the same other way, just switch the 'self' and 'webWorker' variables around.)
② In your browser-side Javascript (as opposed to worker-side), you can use imageData.data.set() to "simply" change the data attribute and put it back in the canvas.
var imageData = context2d.createImageData(width, height);
imageData.data.set(new Uint8ClampedArray(bufferToReturn));
context2d.putImageData(imageData, x_offset, y_offset);
I would like to thank hacks.mozilla.org for alerting me to the existence of the data.set() method.
p.s. I don't know of any libraries to help with this… yet. Sorry.
I have yet to test it out myself, but there is a pure JS library that might be of use here:
https://github.com/taisel/JS-Image-Resizer
Related
I'm working on a 2d world editor for html/js, and trying to find the best way to split up a chip set (as seen below) into multiple little squares (chips).
Currently, I'm using a method similar to the default css sprite sheets method, of using background position to move the background of many little <div> elements until each displays one square/chip of the chip set.
It is working fine, with no big performance issues, but it seems like an overall clunky way to do it.
Other ways I've thought of doing it would be to slice the chip set into many temporary images and make an <image> element for each, or using <canvas> instead of <div>'s or <image>'s
Anyways, I'm looking for advice on the subject:
What is a high performance way of splitting up Sprite Sheets/ Chip Sets in JS?
example of a chip set
The way you're doing it right now should be pretty fast. You could essentially reimplement that on the canvas level by storing just the one image then using drawImage to only draw sections from it. To the best of my knowledge, this is the most common way and is very fast. At least that was my experience when I wrote a game engine using canvas.
Using the canvas is most likely faster and more memory efficient since you don't have the additional DOM overhead but you'd have to measure it for your specific use case and verify that.
Creating a bunch of individual images, as you suggested, would be very slow since it'd require copying portions of the image and creating an additional DOM element for each image.
In short: using the canvas will always be faster (for a comparable implementation) since you're almost directly interfacing with the GPU. Drawing from one image rather than copying it into multiple sub images will always be faster since you won't have duplicate memory sitting around and as long as you're drawing from the same sheet, the GPU doesn't have to switch out the texture.
This is more of a generic question to be honest, just wondering if anyone has done any sort of research on the subject.
Basically I am adding event support to a small game engine I am creating for my own personal use. I would like pixel perfect hover over 2d object event support and am just thinking of the best way of doing it. Realistically it would be faster for me personally just to invoke a draw of my objects onto a transparent canvas and checking if the mouse x y is over a transparent pixel or not since I dont have to make a set of points defining the outside of an object. This would also allow me to have holes in my object and it would still correctly know if I hovered over or not.
What I am wondering is using methods shown here: How can I determine whether a 2D Point is within a Polygon?
How much slower would my method be to the methods shown there?
Im currently still learning so its not easy for me to implement all of this and just test it myself since it would probably take me ages to get to work correctly and test the speeds.
Side note: I would still have a basic bounding box to save it from redrawing and testing every single time.
Checking if a point is in a polygon will 99.999999% of the time be vastly, vastly faster.
To be slower the polygon would need to be extremely complex.
To do the other method you need to use getImageData, and getting image data on the canvas is very slow.
Point in polygon algorithms do properly account for holes. Make sure you have one that obeys the non-zero winding number rule, because that is what canvas uses (as opposed to the even odd rule) and you may want compatibility with paths constructed in the canvas (either now or later).
I'm trying to visualize some data on an HTML canvas and I'm facing an issue similar to this one. That is, the size of my data doesn't exactly match the size of my canvas.
In one instance I'd like to plot a 1024 point signal on a canvas that's 100px wide. (E.g., an audio waveform.)
In another instance I'd like to show a 1024 by 5000 point matrix on a canvas that's 100 px high by by 500 px wide. (E.g., an audio spectrogram.)
In both cases, I'll need to resample my data so that it fits on the canvas. Does anyone know of a library/toolkit/function in Javascript that can do this?
** EDIT **
I'm aware that there are many techniques I could use here. One possibility is to simply discard or duplicate data points. This would do in a pinch, but discarding/duplication is known to produce results that tend to look "jagged" or "blocky" (see here and here). I'd prefer to use a slightly more sophisticated algorithm that outputs smoother images such as Lanczos, bilinear or bicubic resampling. Any of these would meet my needs.
My question isn't about which algorithm to use, though, it's about whether any of them have been implemented in open-source javascript libraries. Surprisingly, I haven't been able to find much in JS. Coding my own resampling function is obviously an option, but I wanted to check with the SO community first to make sure I wasn't re-inventing the wheel.
(This answer gives a code listing that's very close to what I want, except that it operates directly on the canvas objects rather than the data arrays, and it forces the aspect ratios of the input and output to be the same. If nothing else is available, I can definitely work with this, but I was hoping for a solution that's a bit more general and flexible, along the lines of Matlab's resample.)
use canvas scale
ctx.scale(xscale,yscale);
you can determine the scaling by calculating the rate between your canvas and the data
ctx.scale(canvas_x/data_x,canvas_y/data_y)
I'm investigating the possibility of producing a game using only HTML's canvas as the display media. To take an example task I need to do, I need to construct the game environment from a number of isometric tiles. Of course, working in 2D means they by necessity come in rectangular packages so there's a large overlap between tiles.
I'm old enough that the natural solution to this problem is to call BitBltMasked. Oh wait, no, an HTML canvas doesn't have something as simple and as pleasing as BitBlt. It seems that the only way to dump pixel data in to a canvas is either with drawImage() which has no useful drawing modes that ignore the alpha channel or to use ImageData objects that have the image data in an array.. to which every. access. is. bounds. checked. and. therefore. dog. slow.
OK, that's more of a rant than a question (things the W3C like tend to provoke that from me), but what I really want to know is how to draw fast to a canvas? I'm finding it very difficult to ditch the feeling that doing 100s of drawImages() a second where every draw respects the alpha channel is inherently sinful and likely to make my application perform like arse in many browsers. On the other hand, the only way to implement BitBlt proper relies heavily on a browser using a hotspot-like execution technique to make it run fast.
Is there any way to draw fast across every possible implementation, or do I just have to forget about performance?
This is a really interesting problem, and there's a few interesting things you can do to solve it.
First, you should know that drawImage can accept a Canvas, not just an image. The "sub-Canvas"es don't even need to be in the DOM. This means that you can do some compositing on one canvas, then draw it to another. This opens a whole world of optimization opportunities, especially in the context of isometric tiles.
Let's say you have an area that's 50 tiles long by 50 tiles wide (I'll say meters for the sake of my own sanity). You might divide the area into 10x10m chunks. Each chunk is represented by its own Canvas. To draw the full scene, you'd simply draw each of the chunks' Canvas objects to the main canvas that's shown to the user. If only four chunks (a 20x20m area), you would only perform four drawImage operations.
Of course, each of those individual chunks will need to render its own Canvas. On game ticks where nothing happens in the chunk, you simply don't do anything: the Canvas will remain unchanged and will be drawn as you'd expect. When something does change, you can do one of a few things depending on your game:
If your tiles extend into the third dimension (i.e.: you have a Z-axis), you can draw each "layer" of the chunk into its own Canvas and only update the layers that need to be updated. For example, if each chunk contains ten layers of depth, you'd have ten Canvas objects. If something on layer 6 was updated, you would only need to re-paint layer 6's Canvas (probably one drawImage per square meter, which would be 100), then perform one drawImage operation per layer in the chunk (ten) to re-draw the chunk's Canvas. Decreasing or increasing the chunk size may increase or decrease performance depending on the number of update you make to the environment in your game. Further optimizations can be made to eliminate drawImage calls for obscured tiles and the like.
If you don't have a third dimension, you can simply perform one drawImage per square meter of a chunk. If two chunks are updated, that's only 200 drawImage calls per tick (plus one call per chunk visible on the screen). If your game involves very few updates, decreasing the chunk size will decrease the number of calls even further.
You can perform updates to the chunks in their own game loop. If you're using requestAnimationFrame (as you should be), you only need to paint the chunk Canvas objects to the screen. Independently, you can perform game logic in a setTimeout loop or the like. Then, each chunk could be updated in its own tick between frames without affecting performance. This can also be done in a web worker using getImageData and putImageData to send the rendered chunk back to the main thread whenever it needs to be updated, though making this work seamlessly will take a good deal of effort.
The other option that you have is to use a library like pixi.js to render the scene using WebGL. Even for 2D, it will increase performance by decreasing the amount of work that the CPU needs to do and shifting that over to the GPU. I'd highly recommend checking it out.
I know that GameJS has blit operations, and I certainly assume any other html5 game libraries do as well (gameQuery, LimeJS, etc etc). I don't know if these packages have addressed the specific array-bounds-checking concern that you had, but in practice their samples seem to work plenty fast on all platforms.
You should not make assumptions about what speedups make sense. For example, the GameJS developer reports that he was going to implement dirty rectangle tracking but it turned out that modern browsers do this automatically---link.
For this reason and others, I suggest to get something working before thinking about the speed. Also, make use of drawing libraries, as the authors have presumably spent some time optimizing performance.
I have no personal knowledge about this, but you can look into the appMobi "direct canvas" HTML element which is allegedly a much faster version of normal canvas, link. I'm confused about whether this works in all browsers or just webkit browsers or just appMobi's own special browser.
Again, you should not make assumptions about what speedups make sense without a very deep knowledge of web browser internal processes. That webpage about "direct canvas" mentions a bunch of things that slow down canvas-drawing: "Reflowing text, mapping hot spots, creating indexes for reference links, on and on." Alpha-blending and array-bounds-checking are not mentioned as prominent causes of slowness!
Unfortunately, there's no way around the alpha composition overhead. Clipping may be one solution, but I doubt there would be much, if any, performance gain. Not to mention how complicated such a route would be to implement on irregular shapes.
When you have to draw the entire display, you're going to have to deal with the performance hit. Although afterwards, you have a whole screen's worth of pre-calculated alpha imagery and you can draw this image data at an offset in one drawImage call. Then, you would only have to individually draw the new tiles that are scrolled into view.
But still, the browser is having to redraw each pixel at a different location in the canvas. Which is quite expensive. It would be nice if there was a method for just scrolling pixels, but no luck there either.
One idea that comes to mind is that you could implement multiple canvases, translating each individual canvas instead of redrawing the pixels. This would allow the browser to decide how to redraw those pixels, in a more native way, at least in theory anyway. Then you could render the newly visible tiles on a new, or used/cached, canvas element. Positioning it to match up with the last screen render.
But that's just my two blits... I mean bits... duh, I mean cents :]
1.) I found a canvas API called EaselJS, it does an amazing job of creating a display list for each elements you draw. They essentially become individually recognizable objects on the canvas (on one single canvas)
2.) Then I saw on http://simonsarris.com/ about this tutorial that can do drag and drop, it makes use of a hidden canvas concept for selection.
3.) And the third approach, a working approach, http://www.lucidchart.com/ , which is exactly what I'm trying to achieve, basically have every single shape on a different canvas, and use to position them. There's a huge amount of canvas.
The question is, what is the easiest way to achieve interactive network diagram as seen on http://www.lucidchart.com/
A side question is, is it better to get text input through positioning on canvas or using multiple canvas (one for rendering text) as in LucidChart
I'm the person who made the tutorials in 2. There's a lot going on here, so I'll try to explain a bit.
I use a hidden canvas for selection simply because it is easy to learn and will work for ANY kind of object (text, complex paths, rectangles, semi-transparent images). In the real diagramming library that I am writing, I don't do anything of the sort, instead I use a lot of math to determine selection. The hidden-canvas method is fine for less than 1000 objects, but eventually performance starts to suffer.
Lucidchart actually uses more than one canvas per object. And it doesn't just have them in memory, they are all there the DOM. This is an organizational choice on their part, a pretty weird one in my opinion. SVG might have made their work a lot easier if thats what they are going to do, as if seems they are doing a lot of footwork just to be able to emulate how SVG works a bit. There aren't too many good reasons to have so many canvases in the DOM if you can avoid it.
It seems to me that the advantage of them doing it that way is that if they have 10,000 objects, when you click, you only have to look at the one (small) canvas that is clicked for selection testing, instead of the entire canvas. So they did it to make their selection code a little shorter. I'd much rather only have one canvas in the DOM; their way seems needlessly messy. The point of canvas is to have a fast rendering surface instead of a thousand divs representing objects. But they just made a thousand canvases.
Anyway, to answer your question, the "easiest" way to achieve interactive network diagrams like lucidchart is to either use a library or use SVG (or an SVG library). Unfortunately there aren't too many yet. Getting all the functionality yourself in Canvas is hard but certainly doable, and will afford you better performance than SVG, especially if you plan on having more than 5,000 objects in your diagrams. Starting with EaselJS for now isn't too bad of an idea, though you'll probably find yourself modifying more and more of it as you get deeper into your project.
I am making one such interactive canvas diagramming library for Northwoods Software, but it won't be done for a few more months.
To answer the question that is sort-of in your title: The fastest method of doing interactiveness such as hit-testing is using math. Any high-performance canvas library with the features to support a lot of different types of objects will end up implementing functions like getNearestIntersectionPoint, getIntersectionsOnRect, pathContainsPoint, and so on.
As for your side question, it is my opinion that creating a text field on top of the canvas when a user wants to change text and then destroying it when the user is done entering text is the most intuitive-feeling way to get text input. Of course you need to make sure the field is positioned correctly over the text you are editing and that the font and font sizes are the same for a consistent feel.
Best of luck with your project. Let me know how it goes.
Using SVG (and maybe libraries as Raphael)!!
Then any element can receive mouse events.