Please consider a Scala.js class that contains a large JavaScript typed array called xArr.
A process called p(xArr) consumes xArr as input but takes a long time to complete. In order to avoid script timeout warnings, p(xArr) runs in a Web Worker.
Recall these constraints on communication between the main thread and the Web Worker thread:
Communication in either direction takes the form of message passing.
Message data must conform to the requirements of JavaScript's structured clone algorithm.
Unless specified in the optional transfer list, message data gets duplicated instead of transfered to/from the main and Worker threads.
To transfer message data instead of copying it to/from the worker thread, the data must implement the Transferable interface and the transfer list must contain a reference to the transferable data.
If a transferable object transfers between threads, the sending thread loses access to it.
Because of xArr's size, sending a copy of it to the worker thread will incur severe memory costs, but becasue of p(xArr)'s run time, it can not run in the main thread.
Fortunately, typed arrays implement the Transferable interface, so to save compute and memory resources, the program invokes p(xArr) by transferring xArr to the WebWorker which invokes p(xArr) then transfers xArr back to the main thread.
Unfortunately, other asynchronous methods in the main thread must access xArr which may have transferred to the worker's scope at invocation time.
What Scala language features could govern access to xArr so that method calls execute immediately when the main thread owns xArr but wait for it to return to scope when the worker owns xArr?
In other words: How would you handle a class variable that continuously alternates between defined and undefined over time?
Would you suggest locks? Promise/Callback queues? Would you approach the problem in an entirely different way? If so, how?
Remember that this is a Scala.js library, so we must disqualify JVM specific features.
I understand your very real pain here. This used to work with SharedArrayBuffer but it is currently disabled in Chrome. Sadly there is no alternative for shared memory:
Note that SharedArrayBuffer was disabled by default in all major browsers on 5 January, 2018 in response to Spectre.
There are plans to re-add SharedArrayBuffer after proper security auditing will be complete. I guess we'll have to wait.
If you were running your code in Node - this would be hard but possible.
Thanks to all who considered this issue. A solution exists as of 19 May 2018; hopefully a better one can replace it soon.
The current version works as follows:
Problem 1: How can we associate function calls from the main thread with function definitions in the worker thread?
S1: A map of Promise objects: Map[Long, PromiseWrapper]() associates a method invocation ID with a promise that can process the result. This simple multiplexing mechanism evolved from another Stack Overflow question. Thanks again to Justin du Coeur.
Problem 2: How can we invoke functions in the worker thread from the main thread?
S1: Pass a text representation of the function to the worker, then parse it with eval and invoke the resulting function. Unfortunately, eval comes with security risks. Besides, having to write pure JavaScript code in string values defeats most of the advantages of Scala.js, namely type safety and Scala syntax.
S2: Storing function definitions in a lookup table in worker scope and invoking the functions by passing the keys. This could work, but feels clunky in Scala because different functions take parameters that vary in number and type.
S3: Wrap the functions into serializable case classes, then send the serialized bytes from the main scope to the worker scope and invoke the function there. You can think of these case classes as message classes. The current solution uses this approach. It relies on BooPickle by Otto Chrons. The serialized class wraps the method call and any trivial function parameters, e.g. numbers, short strings, and simple case classes. Large data, like the TypedArray values featured in this question transfer from the main thread to the worker thread through a mechanism discussed later. Unfortunately, this approach means that all operations on the TypedArray values must be defined before compile time because BooPickle relies on macros, not reflection, to serialize and deserialize classes.
Problem 3: How can we pass the values of the TypedArray class variable, xArr to and from the worker thread without duplicating it?
S1: Because xArr conforms to the Transferrable interface, it can transfer wholly between the main and worker scopes. At the same time, the serialized classes that wrap the function calls conform to a trait that specifies an apply method with this signature:
def apply(parameters: js.Array[Transferable]): js.Array[Transferable]
By convention, the parameters array contains a serialized version of the message case class in index 0. Subsequent indices contain the TypedArray values. Each message class has its own unique implementation of this apply method.
Problem 4: How can we pass the result of the computation back to the promise that waits for it in the main thread?
S1: The apply methods mentioned in Problem 3.S1 return a new array of Transferrable objects with another serialized message class at its head. That message class wraps the return value from the computation: p(xArr) and, with an apply method of its own, instructs the main thread on how to interpret the array. In cases where p(xArr) returns large objects like other TypedArray values, those occupy subsequent positions in the array.
Problem 5: What if statements in the main thread try to access xArr when it has transferred to the worker thread?
S1. Now, any code in the main thread can only access xArr through a checkOut method and must restore it by calling a checkIn method. The checkOut method returns a Future that completes when xArr returns from the worker thread. Concurrent calls to checkOut get pushed onto a queue of promises. Any code that calls checkOut must call checkIn to pass control of xArr on to the next Promise waiting in the queue. Unfortunately, this design burdens the programmer with the responsibility of restoring xArr to its encompassing class. Unfortunately, schemes like this resemble classical concurrency models with locks and memory allocation methods like malloc and free, and tend toward buggy code that freezes or crashes.
Problem 5: After p(xArr) executes in the worker thread, how can xArr return to the class that encapsulated it in the main thread?
S1. Message case classes meant to invoke p(xArr) now inherit from a trait called Boomerang. As the name implies, these messages transfer from the main thread to the worker thread, invoke p(xArr) while there, then return, unchanged, to the main thread. Once returned to the main thread, Boomerang objects call relevant checkIn methods to restore xArr values to their original encapsulating objects.
For simplicity, this answer leaves out details about different types of Transferrable parameters, operations that mutate xArr instead of simply reading it and restoring it, operations that don't take any parameters but still yield large TypedArray responses, and operations that take multiple large TypedArray parameters, but minor modifications to the five solutions articulated above met those objectives.
With this as a baseline, can we:
Simplify this design?
Incorporate user defined operations?
Find safer alternatives to the checkOut, checkIn methods?
Related
I would like to create a worker thread in a node.js app and pass the current context to the new thread, so I would be able to access my variables and functions within the new thread, Is there is a library to support that? And if not can I a least pass an anonymous function between them?
There is no way to share a context with a worker thread. This isn't "an ideology of the Node.js team", instead it's a limitation of the JavaScript language, which doesn't allow concurrency (such as concurrent access to objects from a worker thread).
The one exception is that you can share numerical data between multiple threads by using a SharedArrayBuffer.
Aside from that, the way to send data to or receive it from a worker thread is to use postMessage. See also Node's full worker threads documentation.
For completeness: there is an early-stage proposal to add a new kind of cross-thread-shareable object to JavaScript. As with all early-stage proposals, there's no guarantee that it'll be finalized at all or how long that might take, but it does indicate that there's some interest in this space.
I am writing a multi-threaded program. The main thread is constantly receiving network data, and the amount of data is relatively large, so sub-threads are used to process the data.
The received data is a 100-byte packet. Each time I receive a packet, I create a 100-byte SharedArrayBuffer and send it to the child thread via postMessage(). But the main thread receives the data very fast, so it is necessary to frequently call postMessage to notify the sub-thread, which leads to high CPU usage...affecting the response speed of the main thread
So I was thinking, if SharedArraybuffer can grow dynamically, the received data is constantly appended at the end of the SharedArrayBuffer, I only notify the child thread once, so that the child thread can also access the data.
I would like to ask how to dynamically increase the length of SharedArrayBuffer. I have tried to implement it in a chained way, storing a SharedArrayBuffer object in another SharedArrayBuffer object, but the browser does not allow this.
I would like to ask how to dynamically increase the length of SharedArrayBuffer.
From MDN web docs (emphasis mine).
"The SharedArrayBuffer object is used to represent a generic, fixed-length raw binary data buffer, similar to the ArrayBuffer object, but in a way that they can be used to create views on shared memory."
Fixed-length means you can't resize it...so it's not surprising it doesn't have a resize() method.
(Note: One thing that does cross my mind though is I believe there is a very new ability for SharedArrayBuffer to be used in WebAssembly as "linear memory" which has a grow_memory operator. I would imagine taking advantage of this would be very difficult, if it is possible at all, and likely would not be supported in many browsers if it was.)
I have tried to implement it in a chained way, storing a SharedArrayBuffer object in another SharedArrayBuffer object, but the browser does not allow this.
Nope. You can only write numbers.
It might seem that you could use a number to index into a table of SharedArrayBuffers, and link them that way. But then you have to worry about how to share that table between threads--same problem.
So no matter what you do, whichever thread makes the decision to update the shared buffering structure will need to notify the others of the update somehow. For that notification to be able to transfer SharedArrayBuffers, it will have to use postMessage to do it.
Have you considered experimenting with allocating a larger SharedArrayBuffer to start with, and treat it like a circular buffer so that the main thread reads out of the writes the sub threads are doing, in a "producer/consumer" pattern?
If you insist on implementing resizes, you might consider having some portion of the buffer hold an indicator that it is "stale" and a new one must be requested from the thread that resized it. You'll have to control that with synchronization. If you make a small sample that does this, it would probably make a good technical article...and if you have trouble with the small sample, it would be a good basis for further questions here.
There is no way to resize, only copy via a typed array.
But no RAM is actually allocated, until the ram is actually used. Under Node.js (v14.14.0) you can see how the ram usage gradually increases as the buffer is filled or how it is basically instantly used if array.fill is used.
const sharedBuffer = new SharedArrayBuffer(512 * 1024 * 1024)
const array = new Uint8Array(sharedBuffer)
// array.fill(1) // Causes ram to be allocated right away
I have a fuzzysearch function. I have a list of 52k words. I'm running the function against each word. it takes about 30ms to finish.
i tried splitting it up into 8 web worker threads by sending 1/8th of the list to each worker (i have 8 threads on my cpu) using myWorker.postMessage({targets:slice, search}). but this is much slower, around 100ms.
my question is: is it possible for multithreaded to be faster here? or is it simply too much data to copy around to finish in under 30ms threaded? is it possible to not copy the memory and have some kind of shared memory?
(it seems like just simply sending the data to the workers is slower than me actually searching all the data in 1 thread!)
is it possible to not copy the memory and have some kind of shared
memory?
You can use second parameter of Worker.postMessage() to transfer the created object from Worker thread to main thread, or from main thread to worker.
// transfer data to `Worker` instance
worker.postMessage(data.buffer, [data.buffer]) // where `data` is an `ArrayBuffer`
// transfer data from `Worker` instance
self.postMessage(data.buffer, [data.buffer]) // where `data` is an `ArrayBuffer`
Passing data by transferring ownership (transferable objects)
Google Chrome 17+ and Firefox 18+ contain an additional way to pass
certain types of objects (transferable objects, that is objects
implementing the Transferable interface) to or from a worker with
high performance. Transferable objects are transferred from one
context to another with a zero-copy operation, which results in a vast
performance improvement when sending large data sets.
In JavaScript: Understanding the Weird Parts the instructor explains that memory for variables is set up during a so-called creation phase (and that undefined is assigned); then the execution phase happens. But why is this useful when we don't know what value(s) the variable will later point to?
Clearly variables can point to many different things -from e.g. a short string all the way to a deeply nested object structure -and I assume that they can vary wildly in the amount of memory they need.
If line-by-line execution -including variable assignment -happens only in the later, execution phase, how can the initial creation phase know how to set up memory? Or, is memory set aside only for the name in each variable name/value pair, with memory for the value being managed differently?
The instructor is referring to Google Chrome's V8 engine (as is evidenced by his use of it in the video).
The V8 engine uses several optimization approaches in order to facilitate memory management. At first, it will compile the JavaScript code and during compilation it will determine how many variables (hidden classes, more later) it needs to create. These will determine the amount of memory originally allocated.
V8 compiles JavaScript source code directly into machine code when it is first executed. There are no intermediate byte codes, no interpreter. Property access is handled by inline cache code that may be patched with other machine instructions as V8 executes. 1
The first set is created by navigating the JavaScript code to determine how many different object "shapes" there are. Anything without a prototype is considered to be a "Transitioning object shape"
The main way objects are encoded is by separating the hidden class (description) from the object (content). When new objects are instantiated, they are created using the same initial hidden class as previous objects from the same constructor. As properties are added, objects transition from hidden class to hidden class, typically following previous transitions in the so-called “transition tree”. 2
Conversely, if the object does have a prototype then it will have its particular shape tracked separately.
Prototypes have 2 main phases: setup and use. Prototypes in the setup phase are encoded as dictionary objects. Any direct access to the prototype, or access through a prototype chain, will transition it to use state, making sure that all such accesses from now on are fast. 2
The compiler will essentially read all possible variables as being one of these two possible shapes and then allocate the amount of memory necessary to facilitate instantiating those shapes.
Once the first set of shapes is setup, V8 will then take advantage of what they call "fast property access" in order to build on the first set of variables (hidden classes) that were setup during the build.
To reduce the time required to access JavaScript properties V8 dynamically creates hidden classes behind the scenes 3
There are two advantages to using hidden classes: property access does not require a dictionary lookup, and they enable V8 to use the classic class-based optimization, inline caching 3
As a result, not all memory use is known during compilation, only how much to allocate for the core set of hidden classes. This allocation will grow as the code is executed, from things like assignment, inline cache misses, and conversion into dictionary mode (which happens when too many properties are assigned to an object, and several other nuanced factors).
1. Dynamic machine code generation, https://github.com/v8/v8/wiki/Design%20Elements#dynamic-machine-code-generation
2. Setting up prototypes in V8, https://medium.com/#tverwaes/setting-up-prototypes-in-v8-ec9c9491dfe2
3. Fast Property Access, https://github.com/v8/v8/wiki/Design%20Elements#fast-property-access
Javascript is a bit of a problem you know at least in my opinion
in Javascript there are specifications of the language made by ecmascript
and there are implementation made by developers
you have to understand that what is been taught about Javascript what's so called "under the hood" are the specifications of Javascript
you might have heard the terms Execution Context Lexical Environment.
They are only spec on how the language should work they give idea to the Developers how to build their JS engine in a way that it will behave like the spec.
An execution context is purely a specification mechanism and need not correspond to any particular artefact of an ECMAScript implementation. It is impossible for ECMAScript code to directly access or observe an execution context. ECMAScript
Every Javascript engine is implemented differently and they supposed to behave like the spec of ECMAScript
there is a concept that everytime when an execution context is created.
it has stages creation phase and execution phase.
everytime when creation phase begin it's allocating memory for the variables for that execution context.
in reality it doesn't work like that at all
there is no really a "Creation Phase" at least in V8 engine(Google Chrome JS Engine) in the way that you think.
i can give you a hint and tell you that eveytime when you call a function that
doesn't have another function inside it.
the variables inside that function are basically replacing some "block" in memory
i will give you a basic example let's say V8 engine uses some address in memory
let's say the address is 0x61FF1C
function setNum(){ var num = 5; }
everytime when i will call the setNum function the value for example of num will get stored at address 0x61FF1C EVERYTIME when i will call the function the value of num 5it will get stored at 0x61FF1C so it's overwriting the content that were before inside 0x61FF1C.
that's just how the v8 engine works in that scenario now the example i gave is just to get the idea i know it's sounds a little vague.
there is much more to the V8 engine which i'm not going to discuss because it's huge
by the way i'm not a V8 developer so i dont know everything about that engine
i'm not even close to that level but i know some things.
anyway i think that every JS Developer should think in the spec way but also remember that many engines behave like the spec but it doesn't mean that they work exactly like the spec
When any program executes, time of execution we call running time. During running time or processing time, processor process code and communicate with memory. Procesor takes code from memory, process it, result give back to memory, take other code. All the running time, some space in memory get grater; some space gets zero, some variables get the new value, some variables have been deleting, etc. Volume of working memory in running time is changing all the time.
Read in a node.js related web document that it is a single threaded server. So it confuses me whether all data structures by default be thread-safe in a node server!
I have multiple call-backs accessing a global object like this :
callback1{
global_var['key'] = val;
}
callback2{
globalv_var['key'] = val;
}
'key' may be same at times and may be different as well. Will the global_var be thread-safe ?
callbacks, as intended gets called back as and when something is done, in no particular order.
Node.JS contains a "dispatcher." It accepts web requests and hands them off for asynchronous processing. That dispatcher is single threaded. But the dispatcher spins up a new thread for each task, and quickly hands off the task to the new thread, freeing the dispatcher's thread for servicing a new request.
To the extent that those task threads are kept separate (i.e. they don't modify each other's state), yes, they are threadsafe.
All of the javascript you write for your node.js applocation executes as if it were running in a single thread.
Any multithreading occurs behind the scenes, in the I/O code and in other native modules. So there's no need to worry about the thread safety of any application code, regardless.