Share Array Reference between JavaScript and ActionScript - javascript

I have been working with the WebcamJS library to stream video from the camera in the browser, but I have run into a major performance bottleneck. Since I am using Internet Explorer 11 (and cannot switch to a different browser), this library reverts to a Flash fallback for accessing the camera.
The ActionScript callback that returns the image is prohibitively slow, due to its many steps. When it returns the image, it first encodes its byte array as a PNG or JPG, and then to a base 64 string. This string is then passed using ExternalInterface to JavaScript, which decodes the image through a data URI. Given that all I need is the byte array in JavaScript, these extra steps seem wasteful.
I have had to tackle a similar problem before, in C++/Python. Rather than repeatedly pass the array data back and forth between the two languages, I used Python to pass a NumPy array reference at the start of the program. Then, they could both access the same data from then on without any extra communication.
Now that you understand my situation, here is the question: is it possible to pass a JavaScript Array or ArrayBuffer by reference to ActionScript? In that case, I could have ActionScript modify the JavaScript array directly, rather than waste time converting, encoding, and decoding the image for each frame.
(WebcamJS: https://github.com/jhuckaby/webcamjs)

Just for completeness, SharedObjects in flash store data, serialised with the AMF protocol, on the file system (in a very specific, sandboxed and locked place) where Javascript has no way to access to read the data.
Have you tried to simply call the ExternalInterface method and pass an array of bytes as an argument? it would be passed by value, automatically converted from the Actionscript data structure to the Javascript one, but you'd skip all the encoding steps and it should be fast enough ...

Related

What is best practice to pass binary data between WebWorker and main thread?

To my knowledge you can only pass a string or a object which can be seralize as JSON.
Then what is the best way to pass some image files between workers and main page if I am using WebWorker to download them in the background?
you can only pass a string or a object which can be seralize as JSON.
Your premise is wrong. You can pass every kind of object that is supported by the structured clone algorithm, this can for example be circular-linked data which cannot be represented as JSON. You also can pass ArrayBuffers, ArrayBufferViews, PixelDataArrays, Blobs etc, all the data types that are not known to JSON.
The second parameter of the postMessage method of Workers does even allow you to transfer binary data (such as ArrayBuffers) directly to the worker - it won't even copy anything, though it will neuter the reference that you have.

how to pass large data to web workers

I am working on web workers and I am passing large amount of data to web worker, which takes a lot of time. I want to know the efficient way to send the data.
I have tried the following code:
var worker = new Worker('js2.js');
worker.postMessage( buffer,[ buffer]);
worker.postMessage(obj,[obj.mat2]);
if (buffer.byteLength) {
alert('Transferables are not supported in your browser!');
}
UPDATE
Modern versions of Chrome, Edge, and Firefox now support SharedArrayBuffers (though not safari at the time of this writing see SharedArrayBuffers on MDN), so that would be another possibility for a fast transfer of data with a different set of trade offs compared to a transferrable (you can see MDN for all the trade offs and requirements of SharedArrayBuffers).
UPDATE:
According to Mozilla the SharedArrayBuffer has been disabled in all major browsers, thus the option described in the following EDIT does no longer apply.
Note that SharedArrayBuffer was disabled by default in all major
browsers on 5 January, 2018 in response to Spectre.
EDIT: There is now another option and it is sending a sharedArray buffer. This is part of ES2017 under shared memory and atomics and is now supported in FireFox 54 Nightly. If you want to read about it you can look here. I will probably write up something some time and add it to my answer. I will try and add to the performance benchmark as well.
To answer the original question:
I am working on web workers and I am passing large amount of data to
web worker, which takes a lot of time. I want to know the efficient
way to send the data.
The alternative to #MichaelDibbets answer, his sends a copy of the object to the webworker, is using a transferrable object which is zero-copy.
It shows that you were intending to make your data transferrable, but I'm guessing it didn't work out. So I will explain what it means for some data to be transferrable for you and future readers.
Transferring objects "by reference" (although that isn't the perfect term for it as explained in the next quote) doesn't just work on any JavaScript Object. It has to be a transferrable data-type.
[With Web Workers] Most browsers implement the structured cloning
algorithm, which allows you to pass more complex types in/out of
Workers such as File, Blob, ArrayBuffer, and JSON objects. However,
when passing these types of data using postMessage(), a copy is still
made. Therefore, if you're passing a large 50MB file (for example),
there's a noticeable overhead in getting that file between the worker
and the main thread.
Structured cloning is great, but a copy can take hundreds of
milliseconds. To combat the perf hit, you can use Transferable
Objects.
With Transferable Objects, data is transferred from one context to
another. It is zero-copy, which vastly improves the performance of
sending data to a Worker. Think of it as pass-by-reference if you're
from the C/C++ world. However, unlike pass-by-reference, the 'version'
from the calling context is no longer available once transferred to
the new context. For example, when transferring an ArrayBuffer from
your main app to Worker, the original ArrayBuffer is cleared and no
longer usable. Its contents are (quiet literally) transferred to the
Worker context.
- Eric Bidelman Developer at Google, source: html5rocks
The only problem is there are only two things that are transferrable as of now. ArrayBuffer, and MessagePort. (Canvas Proxies are hopefully coming later). ArrayBuffers cannot be manipulated directly through their API and should be used to create a typed array object or a DataView to give a particular view into the buffer and be able to read and write to it.
From the html5rocks link
To use transferrable objects, use a slightly different signature of
postMessage():
worker.postMessage(arrayBuffer, [arrayBuffer]);
window.postMessage(arrayBuffer, targetOrigin, [arrayBuffer]);
The worker case, the first argument is the data and the second is the
list of items that should be transferred. The first argument doesn't
have to be an ArrayBuffer by the way. For example, it can be a JSON
object:
worker.postMessage({data: int8View, moreData: anotherBuffer}, [int8View.buffer, anotherBuffer]);
So according to that your
var worker = new Worker('js2.js');
worker.postMessage(buffer, [ buffer]);
worker.postMessage(obj, [obj.mat2]);
should be performing at great speeds and should be being transferred zero-copy. The only problem would be if your buffer or obj.mat2 is not an ArrayBuffer or transferrable. You may be confusing ArrayBuffers with a view of a typed array instead of what you should be using its buffer.
So if you have this ArrayBuffer and it's Int32 representation. (though the variable is titled view it is not a DataView, but DataView's do have a property buffer just as typed arrays do. Also at the time this was written the MDN use the name 'view' for the result of calling a typed arrays constructor so I assumed it was a good way to define it.)
var buffer = new ArrayBuffer(90000000);
var view = new Int32Array(buffer);
for(var c=0;c<view.length;c++) {
view[c]=42;
}
This is what you should not do (send the view)
worker.postMessage(view);
This is what you should do (send the ArrayBuffer)
worker.postMessage(buffer, [buffer]);
These are the results after running this test on plnkr.
Average for sending views is 144.12690000608563
Average for sending ArrayBuffers is 0.3522000042721629
EDIT: As stated by #Bergi in the comments you don't need the buffer variable at all if you have the view, because you can just send view.buffer like so
worker.postMessage(view.buffer, [view.buffer]);
Just as a side note to future readers just sending an ArrayBuffer without the last argument specifying what the ArrayBuffers are you will not send the ArrayBuffer transferrably
In other words when sending transferrables you want this:
worker.postMessage(buffer, [buffer]);
Not this:
worker.postMessage(buffer);
EDIT: And one last note since you are sending a buffer don't forget to turn your buffer back into a view once it's received by the webworker. Once it's a view you can manipulate it (read and write from it) again.
And for the bounty:
I am also interested in official size limits for firefox/chrome (not
only time limit). However answer the original question qualifies for
the bounty (;
As to a webbrowsers limit to send something of a certain size I am not completeley sure, but from that quote that entry on html5rocks by Eric Bidelman when talking about workers he did bring up a 50 mb file being transferred without using a transferrable data-type in hundreds of milliseconds and as shown through my test in a only around a millisecond using a transferrable data-type. Which 50 mb is honestly pretty large.
Purely my own opinion, but I don't believe there to be a limit on the size of the file you send on a transferrable or non-transferrable data-type other than the limits of the data type itself. Of course your biggest worry would probably be for the browser stopping long running scripts if it has to copy the whole thing and is not zero-copy and transferrable.
Hope this post helps. Honestly I knew nothing about transferrables before this, but it was fun figuring out them through some tests and through that blog post by Eric Bidelman.
I had issues with webworkers too, until I just passed a single argument to the webworker.
So instead of
worker.postMessage( buffer,[ buffer]);
worker.postMessage(obj,[obj.mat2]);
Try
var myobj = {buffer:buffer,obj:obj};
worker.postMessage(myobj);
This way I found it gets passed by reference and its insanely fast. I post back and forth over 20.000 dataelements in a single push per 5 seconds without me noticing the datatransfer.
I've been exclusively working with chrome though, so I don't know how it'll hold up in other browsers.
Update
I've done some testing for some stats.
tmp = new ArrayBuffer(90000000);
test = new Int32Array(tmp);
for(c=0;c<test.length;c++) {
test[c]=42;
}
for(c=0;c<4;c++) {
window.setTimeout(function(){
// Cloning the Array. "We" will have lost the array once its sent to the webworker.
// This is to make sure we dont have to repopulate it.
testsend = new Int32Array(test);
// marking time. sister mark is in webworker
console.log("sending at at "+window.performance.now());
// post the clone to the thread.
FieldValueCommunicator.worker.postMessage(testsend);
},1000*c);
}
results of the tests. I don't know if this falls in your category of slow or not since you did not define "slow"
sending at at 28837.418999988586
recieved at 28923.06199995801
86 ms
sending at at 212387.9840001464
recieved at 212504.72499988973
117 ms
sending at at 247635.6210000813
recieved at 247760.1259998046
125 ms
sending at at 288194.15999995545
recieved at 288304.4079998508
110 ms
It depends on how large the data is
I found this article that says, the better strategy is to pass large data to a web worker and back in small bits. In addition, it also discourages the use of ArrayBuffers.
Please have a look: https://developers.redhat.com/blog/2014/05/20/communicating-large-objects-with-web-workers-in-javascript

Flash - passing audio data ByteArray to javascript

I'm able to record sound with a Flash application embedded in my website, this audio is saved to a ByteArray, which I need to pass to Javascript in order to post to my server along with other required data.
I know I can use AS3 ExternalInterface class to communicate with Flash from Javascript, but what would be the appropriate format or variable type in javascript to hold the ByteArray, and how can I ensure that I won't lose much audio data when doing so?
This code worked for me (rec is ByteArray):
rec.position = 0;
rec.compress();
var b64:Base64Encoder = new Base64Encoder();
b64.encodeBytes(rec);
ExternalInterface.call('soundRecorded', b64.toString());
It sucessfully handles about 6 megabytes of data (I didn't try more).
Maybe it's not possible for you for some reason I'm not aware of, but if that's not the case, I'd post the data directly from Actionscript (you can send binary data).
Anyway, if you have to relay the data to JS, the safest way would be base64-encoding the ByteArray. After that you have a string that will not have any control (read: problematic) characters.
Size is of course something to take into account. I don't know what are the limitations of ExternalInterface (I've only ever used it to pass small ammounts of data), but you'll most likely hit a hard limit there. LocalConnection objects have a limit of 100 kb or so if I recall correctly (these are not related to the ExternalInterface api -at least not directly- but I mention it just as a remainder of the possible limitations). If you're working with raw audio, your data will be rather big, so you'd have to figure out how to compress it (and decompress it in the JS end or in the server) and also, probably, how to send it in chunks, as sending it all at once will likely be impossible if the data is too big.
Again, if possible, I'd post directly from Actionscript and would use at least the ByteArray's compress method before sending the data.

JavaScript: Web Worker and Typed Arrays

I have a web worker (started with new Worker()) that does some processing and is supposed to return a Float32Array.
It seems however that after the worker postMessage()s the data, it goes through serialization and desirialization to JSON and what I end up with when receiving the message is a plain javascript Array (with all of the properties the original typed array had)
A trivial work around would be to just recreate the typed array from the javascript array but that's wasteful and takes up time and memory.
Is there a better way to do this? Some kind of way to tell the JSON deserialization to instantiate a Float32Array instead of a javascript array? or a way to otherwise transfer the binary data?
All browsers that support workers (except IE10) support what's called transferable objects which means that if you have an array buffer (ie take your the .buffer property of your typed array) you can as a second parameter of postMessage include a list of array buffers you want to transfer ownership of back. This is much much faster than copying it.
update: this seems to be a Chrome bug at the moment:
http://code.google.com/p/chromium/issues/detail?id=73313
typed array are preserved in Firefox 4.

Passing an ActionScript JPG Byte Array to Javascript (and eventually to PHP)

Our web application has a feature which uses Flash (AS3) to take photos using the user's web cam, then passes the resulting byte array to PHP where it is reconstructed and saved on the server.
However, we need to be able to take this web application offline, and we have chosen Gears to do so. The user takes the app offline, performs his tasks, then when he's reconnected to the server, we "sync" the data back with our central database.
We don't have PHP to interact with Flash anymore, but we still need to allow users to take and save photos. We don't know how to save a JPG that Flash creates in a local database. Our hope was that we could save the byte array, a serialized string, or somehow actually persist the object itself, then pass it back to either PHP or Flash (and then PHP) to recreate the JPG.
We have tried:
- passing the byte array to Javascript instead of PHP, but javascript doesn't seem to be able to do anything with it (the object seems to be stripped of its methods)
- stringifying the byte array in Flash, and then passing it to Javascript, but we always get the same string:
ÿØÿà
Now we are thinking of serializing the string in Flash, passing it to Javascript, then on the return route, passing that string back to Flash which will then pass it to PHP to be reconstructed as a JPG. (whew). Since no one on our team has extensive Flash background, we're a bit lost.
Is serialization the way to go? Is there a more realistic way to do this? Does anyone have any experience with this sort of thing? Perhaps we can build a javascript class that is the same as the byte array class in AS?
I'm not sure why you would want to use Javascript here. Anyway, the string you pasted looks like the beginning of a JPG header. The problem is that a JPG will for sure contain NULs (characters with 0 as its value). This will most likely truncate the string (as it seems to be the case with the sample you posted). If you want to "stringify" the JPG, the standard approach is encoding it as Base 64.
If you want to persist data locally, however, there's a way to do it in Flash. It's simple, but it has some limitations.
You can use a local Shared Object for this. By default, there's a 100 Kb limit, which is rather inadequate for image files; you could ask the user to allot more space to your app, though. In any case, I'd try to store the image as JPG, not the raw pixels, since the difference in size is very significative.
Shared Objects will handle serialization / deserialization for you transparently. There are some caveats: not every object can really be serialized; for starters, it has to have a parameterless constructor; DisplayObjects such as Sprites, MovieClips, etc, won't work. It's possible to serialize a ByteArray, however, so you could save your JPGs locally (if the user allows for the extra space). You should use AMF3 as the encoding scheme (which is the default, I think); also, you should map the class you're serializing with registerClassAlias to preserve the type of serialized the object (otherwise it will be treated as an Object object). You only need to do it once in the app life cycle, but it must be done before any read / write to the Shared Object.
Something along the lines of:
registerClassAlias("flash.utils.ByteArray",ByteArray);
I'd use Shared Objects rather than Javascript. Just keep in mind that you'll most likely have to ask the user to give you more space for storing the images (which seems reasonable enough if you're allowing them to work offline), and that the user could delete the data at any time (just like he could delete their browser's cookies).
Edit
I realize I didn't really pay much attention the "we have chosen Gears to do so" part of your question.
In that case, you could give the base 64 approach a try to pass the data to JS. From the Actionscript side it's easy (grab one of the many available Base64 encoders/decoders out there), and I assume the Gear's API must have an encoder / decoder available already (or at least it shouldn't be hard to find one). At that point you'll probably have to turn that into a Blob and store it to disk (maybe using the BlobAPI, but I'm not sure as I don't have experience with Gears).

Categories