We Keep Coding

How to read audio data from a 'MediaStream' object in a C++ addon

After sweating blood and tears I've finally managed to set up a Node C++ addon and shove a web-platform standard MediaStream object into one of its C++ methods for good. For compatibility across different V8 and Node.js versions, I'm using Native Abstractions for Node.js (nan):
addon.cc
NAN_METHOD(SetStream)
{
Nan::HandleScope scope;
v8::Local<v8::Object> mediaStream = info[0]->ToObject();
}
addon.js
setStream(new MediaStream());
For what it's worth, this works correctly (i.e. it does not obliterate the renderer process on sight), and I can verify the presence of the MediaStream object, e.g. by returning its constructor name from the C++ method:
addon.cc
info.GetReturnValue().Set(mediaStream->GetConstructorName());
When called from JavaScript through setStream, this would return the string MediaStream, so the object is definitely there. I can also return the mediaStream object itself and everything will work correctly, so it's indeed the object I need.
So, how would I read audio data (i.e. audio samples) from this MediaStream object in C++? As a sidenote, the actual data read (and processing) would be done in a separate std::thread.
Bounty Update
I understand this would be sort of easier/possible if I were compiling Electron and/or Chromium myself, but I'd rather not get involved in that maintenance hell.
I was wondering if it would be possible without doing that, and as far as my research goes, I'm convinced I need 2 things to get this done:
The relevant header files, for which I believe blink public should be adequate
A chromium/blink library file (?), to resolve external symbols, similarly to the node.dylib file
Also, as I said, I believe I could compile chromium/blink myself and then I would have this lib file, but that would be a maintenance hell with Electron. With this in mind, I believe this question ultimately comes down to a C++ linking question. Is there any other approach to do what I'm looking for?
Edit
ScriptProcessorNode is not an option in my case, as its performance makes it nearly unusable in production. This would require to process audio samples on the ui/main thread, which is absolutely insane.
Edit 2
AudioWorklets have been available in Electron for some time now, which, unlike the ScriptProcessorNode (or worse, the AnalyzerNode), is low-latency and very reliable for true C++ backed audio processing even in real time.
If someone wants to go ahead and write an AudioWorklet-based answer, I'll gladly accept, but beware: it's a very advanced field and a very deep rabbit hole, with countless obstacles to get through even before a very simple, general-purpose pass-through prototype (especially so because currently in Electron, Atomics-synced, buffered cross-thread audio processing is required to pull this off because https://github.com/electron/electron/issues/22503 -- although getting a native C++ addon into one audio renderer thread, let alone multiple threads at the same time, is probably equally as challenging).

The MediaStream header is part of Blink's renderer modules, and it's not obvious to me how you could retrieve this from nan plugin.
So, instead let's look at what you do have, namely a v8::Object. I believe that v8::Object exposes all the functionality you need, it has:
GetPropertyNames()
Get(context, index)
Set(context, key, value)
Has(context, key)
Unless you really need a strictly defined interface, why not avoid the issue altogether and just use the dynamic type that you already have?
For getting audio data out specifically, you would need to call getAudioTracks() on the v8::Object, which probably looks something like this?
Note: I don't think you need a context, v8 seems to be happy with it being empty: v8/src/api/api.cc
Should look something like this, plus some massaging of types in and out of v8.
v8::MaybeLocal<v8::Value> get_audio_tracks = mediaStream->Get("getAudioTracks");
// Maybe needs to be v8::Object or array?
if (!get_audio_tracks.IsEmpty()) {
v8::Local<v8::Value> audio_tracks = get_audio_tracks.ToLocalChecked()();
}

How to use PixiJS in web worker

I need to display a quite complex 2D shape in a canvas using PixiJS and to do so I'd like to create and define all graphic elements in a separate thread (web worker) to not block the rest of the UI.
The problem is that when I import PixiJS in the web worker file like this
importScripts('https://cdnjs.cloudflare.com/ajax/libs/pixi.js/4.5.2/pixi.js');
it gives me an error because it accesses DOM elements (like window and document) and this is not allowed in web workers. How can I solve this?
This is the error:
Uncaught ReferenceError: window is not defined
at Object.179../Math.sign (Object.assign.js:3)
at s (_prelude.js:1)
at _prelude.js:1
at Object.<anonymous> (WebGLPrepare.js:101)
at Object.187../accessibility (index.js:44)
at s (_prelude.js:1)
at e (_prelude.js:1)
at _prelude.js:1
at _prelude.js:1
at _prelude.js:1

Well I guess you cannot. Webworkers have their own DedicatedWorkerGlobalScope with no access to the DOM, window, etc. If you have heavy computations, for instance to calculate animations, all you can do is let the webworker do the number crunching and the main thread do the rendering.
The Worker and the main thread cannot share objects. Even if the following explanation is not 100% technically correct, you can imagine that if you:
var obj = { a: { b: 100 } };
worker.postMessage(obj);
It will be more like:
//really dirty object clone here
worker.postMessage(JSON.parse(JSON.stringify(obj)));
With that I want to point out, that you cannot share objects with the worker and vice verca.
Find a tecnically correct explaintion here:
https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers#Transferring_data_to_and_from_workers_further_details

Philipp is right on his answer, I just want to elaborate this a bit further, from the games and web workers point-of-view.
In my experience, it is ofter hard to take advantage of web workers in games. Especially if you use a library to handle canvas / webGL, since that does most of the number crunching anyway. Web workers are designed for number crunching as philipp mentioned. And passing data between web worker and main thread is quite expensive, so passing data and doing tiny operations on them, will not be beneficial.
Without providing a direct link, I read already long while ago that some game engines (I think construct) used web workers for their pathfinding module. So nothing related to direct graphic handling, but numerical operations.
Also there is a proposal for possibility to use canvases in web worker contexts, so it is clearly an issue with others too. If my memory serves right, I believe it was this: https://developer.mozilla.org/fi/docs/Web/API/OffscreenCanvas

You can include PixiJS in your Web Worker, although you can not perform graphics operations.
Although the other two answers are technically correct, there are legitimate use cases for including Pixi's code in your Worker's scope without actually rendering graphics. An example is using Pixi's Point and Rectangle classes. In my case, I wrote several modules that rely on definitions exported by Pixi and perform heavy calculations based on them.
The errors you had were because Pixi relies on the globals window and document to define some graphics-related constants. In order to allow Pixi to finish loading you can provide mock values for window and document with empty methods with the same names as the ones used by Pixi.
As an example, while using PixiJS 4.8.6 the following code worked for me:
window = self
document = {
createElement () {
return {
getContext () {
return {
fillRect () {},
drawImage () {},
getImageData () {},
}
},
}
},
}
importScripts('pixi-4-8-6.js');
/* Web Worker code follows */
// ...
Depending on Pixi's version you may need to adjust this boilerplate. An alternative would be to try one of those packages that mock document and Canvas for headless browsers. I haven't tried that since the code above worked well enough.
Pixi 5 came out recently and it may have changed this completely. I skimmed the code and it looks like they removed the constant definition that was causing problems so it's also possible that this new version just works out of the box. I know they have been looking into letting people use Workers more easily and they have been exploring OffscreenCanvas like Hachi said.

pixi team is working on adding headless environments support like web workers; see https://github.com/pixijs/pixijs/issues/7123

Is there a way to create out of DOM elements in Web Worker?

Context:
I have a web application that processes and shows huge log files. They're usually only about 100k lines long, but it can be up to 4 million lines or more. To be able to scroll through that log file (both user initiated and via JavaScript) and filter the lines with decent performance I create a DOM element for each line as soon as the data arrives (in JSON via ajax). I found this better for performance then constructing the HTML at the back-end. Afterwards I save the elements in an array and I only show the lines that are visible.
For max 100k lines this takes only about a few seconds, but anything more takes up to one minute for 500k lines (not including the download). I wanted to improve the performance even more, so I tried using HTML5 Web Workers. The problem now is that I can't create elements in a Web Worker, not even outside the DOM. So I ended up doing only the json to HTML conversion in the Web Workers and send the result to the main thread. There it is created and stored in an array. Unfortunately this worsened the performance and now it takes at least 30 seconds more.
Question: So is there any way, that I'm not aware of, to create DOM elements, outside the DOM tree, in a Web Worker? If not, why not? It seems to me that this can't create concurrency problems, as creating the elements could happen in parallel without problems.

Alright, I did some more research with the information #Bergi provided and found the following discussion on W3C mailing list:
http://w3-org.9356.n7.nabble.com/Limited-DOM-in-Web-Workers-td44284.html
And the excerpt that answers why there is no access to the XML parser or DOM parser in the Web Worker:
You're assuming that none of the DOM implementation code uses any sort
of non-DOM objects, ever, or that if it does those objects are fully
threadsafe. That's just not not the case, at least in Gecko.
The issue in this case is not the same DOM object being touched on
multiple threads. The issue is two DOM objects on different threads
both touching some global third object.
For example, the XML parser has to do some things that in Gecko can
only be done on the main thread (DTD loading, offhand; there are a
few others that I've seen before but don't recall offhand).
There is however also a workaround mentioned, which is using a third-party implementation of the parsers, of which jsdom is an example. With this you even have access to your own separate Document.

So is there any way, that I'm not aware of, to create DOM elements, outside the DOM tree, in a Web Worker?
No.
Why not? It seems to me that this can't create concurrency problems, as creating the elements could happen in parallel without problems.
Not for creating them, you're right. But for appending them to the main document - they would need to be sent to a different memory (like it's possible for blobs) so that they're inaccessible from the worker thereafter. However, there's absolutely no Document handling available in WebWorkers.
I create a DOM element for each line as soon as the data arrives (in JSON via ajax). Afterwards I save the elements in an array and I only show the lines that are visible.
Constructing over 500k DOM elements is the heavy task. Try to create DOM elements only for the lines that are visible. To improve performance and showing the first few lines faster, you also might chunk their processing into smaller units and use timeouts in between. See How to stop intense Javascript loop from freezing the browser

You have to understand the nature of a webworker. Programming with threads is hard, especially if you're sharing memory; weird things can happen. JavaScript is not equipped to deal with any kind of thread-like interleaving.
The approach of webworkers is that there is no shared memory. This obviously leads to the conclusion that you can't access the DOM.

There is no direct way to access the DOM through Web Workers.
I recently released #cycle/sandbox, it is still WIP, but it proves with the Cycle JS architecture it is fairly straight forward to declare UI behaviour in the Web Worker. The actual DOM is only touched in the main thread, but event listeners, and DOM updates are indirectly declared in the worker, and a synthesized event object is sent when something happens on those listeners. Furthermore it is straight forward to mount these sandboxed Cycle Components side-by-side regular Cycle Components.
http://github.com/aronallen/-cycle-sandbox/

I don't see any reason why you can't construct html strings using web-workers. But I also don't think there would be much of a performance boost.
This isn't related to Web-Workers, but it relates to the problem you're trying to solve. Here are some thing that might help speed things up:
Use DocumentFragments. Add elements to them as the data comes in, and add the fragments to the DOM at an interval (like once a second). This way you don't have to touch the DOM (and incur a redraw) every time a line of text is loaded.
Do loading in the background, and only parse the lines as the user hits the bottom of the scroll area.

According to https://developer.mozilla.org/en-US/docs/Web/Guide/Performance/Using_web_workers there's no access to the DOM from a web worker unfortunately.

You have a couple of anti-patterns in your design:
Creating a DOM object has considerable overhead, and you are
creating potentially millions of them at once.
Trying to get a web worker to manage the DOM is exactly what web
workers are not for. They do everything else so the DOM event loop stays responsive.
You can use a cursor pattern to scroll through arbitrarily large sets of data.
DOM posts a message to worker with start position and number of lines requested (cursor).
Web worker random accesses logs, posts back the fetched lines (cursor data).
DOM updates an element with the async cursor response event.
This way, the heavy lifting is done by the worker, whose event loop is blocked during the fetch instead of the DOM, resulting in happy non-blocked users marvelling at how smooth all your animations are.

Update for 2022 (actually available in chrome since 2018):
If you are ok with displaying your logs in a canvas element, you could use the new OffscreenCanvas api.
The OffscreenCanvas interface provides a canvas that can be rendered off screen. It is available in both the window and worker contexts.
You could then asynchronously display frames produced in the Worker back to a canvas element on the main thread.
More examples here.

So you can't directly create DOM in a webworker - however, there may be another option to do a fair bit of your processing outside the main thread.
Check out this jsPerf I just created: http://jsperf.com/dom-construction-obj-vs-str
Essentially, you could be emitting POJSO's that have all the same values you get from a DOM, and convert it to DOM objects after receiving the message (this is what you're doing when you get HTML back, after all; POJSOs are just lower overhead, by virtue of not requiring further string processing). In this way you could even do things like emit event listeners and such (by, say, prefixing the event name with '!', and having the value map to some template-supplied view argument).
Meanwhile, without the DOM parser available, you'll need your own thing to convert a template as-needed, or to compile one to a format that's fast.

No you can't create DOM elements in a web worker, but you can create a function that accepts the post message from that web worker, that does create the DOM elements. I think the deign that your looking for is called array chucking. And you would need to mix that with the web worker design pattern.

How to do worker-to-worker communication?

I'm experimenting with web workers, and was wondering how well they would deal with embarassingly parallell problems. I therefore implemented Connaway's Game of Life. (To have a bit more fun than doing a blur, or something. The problems would be the same in that case however.)
At the moment I have one web worker performing iterations and posting back new ImageData for the UI thread to place in my canvas. Works nicely.
My experiment doesn't end there however, cause I have several CPU's available and would like to parallellize my application.
So, to start off simply I split my data in two, down the middle, and make two workers each dealing with a half each. The problem is of course the split. Worker A needs one column of pixels from worker B and vice versa. Now, I can clearly fix this by letting my UI-thread give that column down to the workers, but it would be much better if my threads could pass them to eachother directly.
When splitting further, each worker would only have to keep track of it's neighbouring workers, and the UI thread would only be responsible for updating the UI (as it should be).
My problem is, I don't see how I can achieve this worker-to-worker communication. I tried handing the neighbours to eachother by way of an initialization postMessage, but that would copy my worker rather than hand down a reference, which luckily chrome warned me about being impossible.
Uncaught Error: DATA_CLONE_ERR: DOM Exception 25
Finally I see that there's something called a SharedWorker. Is this what I should look into, or is there a way to use the Worker that would solve my problem?

You should be able to use channel messaging:
var channel = new MessageChannel();
worker1.postMessage({code:"port"}, [channel.port1]);
worker2.postMessage({code:"port"}, [channel.port2]);
Then in your worker threads:
var xWorkerPort;
onmessage = function(event) {
if (event.data.code == "port") {
xWorkerPort = event.ports[0];
xWorkerPort.onmessage = function(event) { /* do stuff */ };
}
}
There's not much documentation around, but you could try this MS summary to get started.

multi-core programming using JavaScript?

So I have this seriously recursive function that I would like to use with my code. The issue is it doesn't really take advantage of dual core machines because js is single threaded. I have tried using webworkers but don't really know much about multicore programming. Would someone point me to some material that could explain how it is done. I googled to find this sample link but its not really much help without documentation! =/
I would be glad if someone could show me how this could be done without webworkers though! That would be just awesome! =)
I came across this link on whatwg. This is really weird because it explains how to use multicore programming in webworkers etc, but on executing on my chrome browser it throws errors. Same goes with other browsers.
Error: 9Uncaught ReferenceError: Worker is not defined in worker.js

UPDATE (2018-06-21): For people coming here in search of multi-core programming in JavaScript, not necessarily browser JavaScript (for that, the answer still applies as-is): Node.js now supports multi-threading behind a feature flag (--experimental-workers): release info, relevant issue.
Writing this off the top of my head, no guarantees for source code. Please go easy on me.
As far as I know, you cannot really program in threads with JavaScript. Webworkers are a form of multi-programming; yet JavaScript is by its nature single-threaded (based on an event loop).
A webworker is seperate thread of execution in the sense that it doesn't share anything with the script that started it; there is no reference to the script's global object (typically called "window" in the browser), and no reference to any of your main script's variables other than data you send to the thread.
Think as the web worker as a little "server" that gets asked a question and provides an answer. You can only send strings to that server, and it can only parse the string and send back what it has computed.
// in the main script, one starts a worker by passing the file name of the
// script containing the worker to the constructor.
var w = new Worker("myworker.js");
// you want to react to the "message" event, if your worker wants to inform
// you of a result. The function typically gets the event as an argument.
w.addEventListener("message",
function (evt) {
// process evt.data, which is the message from the
// worker thread
alert("The answer from the worker is " + evt.data);
});
You can then send a message (a String) to this thread using its postMessage()-Method:
w.postMessage("Hello, this is my message!");
A sample worker script (an "echo" server) can be:
// this is another script file, like "myworker.js"
self.addEventListener("message",
function (evt) {
var data = JSON.parse(evt.data);
/* as an echo server, we send this right back */
self.postMessage(JSON.stringify(data))
})
whatever you post to that thread will be decoded, re-encoded, and sent back. of course you can do whatever processing you would want to do in between. That worker will stay active; you can call terminate() on it (in your main script; that'd be w.terminate()) to end it or calling self.close() in your worker.
To summarize: what you can do is you zip up your function parameters into a JSON string which gets sent using postMessage, decoded, and processed "on the other side" (in the worker). The computation result gets sent back to your "main" script.
To explain why this is not easier: More interaction is not really possible, and that limitation is intentional. Because shared resources (an object visible to both the worker and the main script) would be subject to two threads interfering with them at the same time, you would need to manage access (i.e., locking) to that resource in order to prevent race conditions.
The message-passing, shared-nothing approach is not that well-known mainly because most other programming languages (C and Java for example) use threads that operate on the same address space (while others, like Erlang, for instance, don't). Consider this:
It is really hard to code a larger project with mutexes (a mutual exclusion mechanism) because of the associated deadlock/race condition complexities. This is stuff that can make grown men cry!
It is really easy in comparison to do message-passing, shared-nothing semantics. The code is isolated; you know exactly what goes into your worker and what comes out of your worker. Deadlocks and race conditions are impossible to achieve!
Just try it out; it is capable of doing interesting things, probably all you want. Bear in mind that it is still implementation defined whether it takes advantage of multicore as far as I know.
NB. I just got informed that at least some implementations will handle JSON encoding of messages for you.
So, to give an answer to your question (it's all above; tl;dr version): No, you cannot do this without web workers. But there is nothing really wrong about web workers aside from browser support, as is the case with HTML5 in general.

As far as I remember this is only possible with the new HTML5 standard. The keyword is "Web-Worker"
See also:
HTML5: JavaScript Web Workers
JavaScript Threading With HTML5 Web Workers

Web workers are the answer to the client side. For NodeJS there are many approaches. Most popular - spawn several processes with pm2 or similar tool. Run single process and spawn/fork child processes. You can google around these and will find a lot of samples and tactics.
Web workers are already well supported by all browsers. https://caniuse.com/#feat=webworkers
API & samples: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers