I'd like to build a layer of abstraction over the WebWorker API that would allow (1) executing an arbitrary function over a webworker, and (2) wrapping the interaction in a Promise. At a high level, this would look something like this:
function bake() {
... // expensive calculation
return 'mmmm, pizza'
}
async function handlePizzaButtonClick() {
const pizza = await workIt(bake)
eat(pizza)
}
(Obviously, methods with arguments could be added without much difficulty.)
My first cut at workIt looks like this:
async function workIt<T>(f: () => T): Promise<T> {
const worker: Worker = new Worker('./unicorn.js') // no such worker, yet
worker.postMessage(f)
return new Promise<T>((resolve, reject) => {
worker.onmessage = ({data}: MessageEvent) => resolve(data)
worker.onerror = ({error}: ErrorEvent) => reject(error)
})
}
This fails because functions are not structured-cloneable and thus can't be passed in worker messages. (The Promise wrapper part works fine.)
There are various options for serializing Javascript functions, some scarier than others. But before I go that route, am I missing something here? Is there another way to leverage a WebWorker (or anything that executes in a separate thread) to run arbitrary Javascript?
I thought an example would be useful in addition to my comment, so here's a basic (no error handling, etc.), self-contained example which loads the worker from an object URL:
Meta: I'm not posting it in a runnable code snippet view because the rendered iframe runs at a different origin (https://stacksnippets.net at the time I write this answer — see snippet output), which prevents success: in Chrome, I receive the error message Refused to cross-origin redirects of the top-level worker script..
Anyway, you can just copy the text contents, paste it into your dev tools JS console right on this page, and execute it to see that it works. And, of course, it will work in a normal module in a same-origin context.
console.log(new URL(window.location.href).origin);
// Example candidate function:
// - pure
// - uses only syntax which is legal in worker module scope
async function get100LesserRandoms () {
// If `getRandomAsync` were defined outside the function,
// then this function would no longer be pure (it would be a closure)
// and `getRandomAsync` would need to be a function accessible from
// the scope of the `message` event handler within the worker
// else a `ReferenceError` would be thrown upon invocation
const getRandomAsync = () => Promise.resolve(Math.random());
const result = [];
while (result.length < 100) {
const n = await getRandomAsync();
if (n < 0.5) result.push(n);
}
return result;
}
const workerModuleText =
`self.addEventListener('message', async ({data: {id, fn}}) => self.postMessage({id, value: await eval(\`(\${fn})\`)()}));`;
const workerModuleSpecifier = URL.createObjectURL(
new Blob([workerModuleText], {type: 'text/javascript'}),
);
const worker = new Worker(workerModuleSpecifier, {type: 'module'});
worker.addEventListener('message', ({data: {id, value}}) => {
worker.dispatchEvent(new CustomEvent(id, {detail: value}));
});
function notOnMyThread (fn) {
return new Promise(resolve => {
const id = window.crypto.randomUUID();
worker.addEventListener(id, ({detail}) => resolve(detail), {once: true});
worker.postMessage({id, fn: fn.toString()});
});
}
async function main () {
const lesserRandoms = await notOnMyThread(get100LesserRandoms);
console.log(lesserRandoms);
}
main();
I have a nodejs cluster with a primary that handles worker cycles (in the while loop) and that listens to worker messages to progress in the cycle.
(In my code index.js does not send messages on setInterval but on other type of event, I have here simplified the code to get the essence of the problem)
Server.js
var cluster = require('cluster');
const ClusterMessages = require('cluster-messages');
const messages = new ClusterMessages();
if (cluster.isMaster){
let worker = cluster.fork()
console.log(cluster);
(async ()=>{
let cycle = 0
while(true){
console.log(cycle);
cycle ++
await Promise.all([
enough(),
])
}
function enough () {
return new Promise(resolve => {
messages.on('enough', () => {
console.log('enough');
resolve()
});
});
}})()
} else {
require('./index.js')
}
Index.js
const ClusterMessages = require('cluster-messages');
const messages = new ClusterMessages();
setInterval(() => {
messages.send('enough');
}, 1000);
The code is working fine (as such, in this example and in my code) but there seems to be a memory leak as you can understand from the output of this code:
0
enough
1
enough
enough
2
enough
enough
enough
3
enough
enough
enough
enough...
I tried several things like exchanging new Promise and messages.on(), add a return in the callback of the promise but I have no clue what is happening here. Any ideas?
The solution is to make another event that can be triggered once contrary to the 'event listener' of the cluster-messages package
Server.js
if (cluster.isMaster){
let worker = cluster.fork()
console.log(cluster);
// Importing events
const EventEmitter = require('events');
const eventEmitter = new EventEmitter();
messages.on('enough', () => {
eventEmitter.emit('event');
});
(async ()=>{let cycle = 0
while(true){
console.log(cycle);
cycle ++
await Promise.all([
enough(),
])
}
function enough () {
return new Promise(resolve => {
eventEmitter.once('event', () => {
console.log('event');
resolve()
});
});
}})()
} else {
require('./index.js')
}
Index.js
const ClusterMessages = require('cluster-messages');
const messages = new ClusterMessages();
setInterval(() => {
messages.send('enough');
}, 1000);
Every call of enough() installs another listener for the enough event on messages. They never get removed, leaking memory (and leading to an increasing number of logs per event). Instead, use the once method to install the listener:
function enough () {
return new Promise(resolve => {
messages.once('enough', () => {
// ^^^^
console.log('enough');
resolve();
});
});
}
Or even simpler, using once:
const { once } = require('events');
function enough() {
return once(messages, 'enough');
}
In your particular example, I would recommend not to use promises to handle the events. You might even miss events that are fired while you are removing and re-attaching a listener. Just write
let cycle = 0
messages.on('enough', () => {
console.log(cycle);
cycle++;
});
If for some reason you need a loop that you can break from or await other things in, I would recommend an asynchronous iterator, built with on:
const { once } = require('events');
(async () => {
let cycle = 0
for await (const _ of on(messages, 'enough')) {
console.log(cycle);
cycle++;
}
})();
I have a system where I send a large array into the worker, which then uses a library to compress it. Compression takes a long time a makes the worker busy. I'm wondering if further messages sent to the worker will be received while the worker is still busy. Would the messages be lost? Will they queued and eventually processed by the worker as well?
Yes, once the worker has finished what it's working on currently, it'll be able to process the next message in the queue. Here's an example:
// https://stackoverflow.com/questions/10343913/how-to-create-a-web-worker-from-a-string
// I put in a Stack Snippet for live demonstration
// in a real project, put this in a separate file
const workerFn = () => {
self.onmessage = () => {
self.postMessage('Worker message callback just fired');
// something expensive
for (let i = 0; i < 2e9; i++) {
}
self.postMessage('Expensive operation ending');
};
};
const workerFnStr = `(${workerFn})();`;
const blob = new Blob([workerFnStr], { type: 'text/javascript' });
const worker = new Worker(window.URL.createObjectURL(blob));
worker.onmessage = ({ data }) => console.log(data);
worker.postMessage(null);
worker.postMessage(null);
This is similar to the idea that when an element in the DOM is clicked on, that click will be processed once the current event loop task finishes, even if said task is expensive.
setTimeout(() => {
window.onclick = () => console.log('click callback running');
// something expensive
for (let i = 0; i < 3e9; i++) {
}
console.log('Main synchronous task finishing');
}, 500);
Click here *quickly* after running the snippet
I have a Node.js script that subscribes to a notification service and runs a bunch of things when push notification is received. However the service sometimes sends multiple notifications for the same event, so to avoid duplicate work I made a basic semaphore to block other tasks.
The problem is that Node still continues with execution despite the fact I see the file created on disk. I've tried a few different solutions but I think the problem comes from my lack of experience with the JS execution model, there's something I don't know about how it works that prevents my solution from working. How do I fix this?
const fse = require('fs-extra');
// notification handler callback
function handleRequest(data)
{
try{
var semaphore = fse.readJsonSync(__dirname + '/' + objectId);
console.log('task already running, stopping');
return;
}catch(err){
// semaphore doesn't exist, ok to proceed
console.log('starting new task');
fse.writeJson(__dirname + '/' + objectId, {objectId: objectId})
.then(stepOne).catch(rejectPromise)
.then(resp => stepTwo(resp, data)).catch(rejectPromise)
.then(resp => stepThree(resp, extra)).catch(rejectPromise)
.then(resp => stepFour(resp, argument)).catch(rejectPromise)
.then(sleep(20000))
.then(resp => releaseLock(objectId))
.catch(resp => rejectionHandler(resp);
}
}
function releaseLock(objectId)
{
return fse.remove(__dirname + '/' + objectId);
}
Other things I've tried
Create file in a separate function that returns promise, same outcome
Use Sync method to write file, but then I'm unable to chain promises
Wait synchronously after file creation, no effect
There is no need to create an external file to maintain locks, you can do something like this, this will also give you the performance boost ( less I/O opts).
const fse = require('fs-extra');
// notification handler callback
class NamedLocks {
constructor() {
this._pid = {};
}
acquire(pid) {
if (this._pid[pid]) {
// process is locked
// handle it
return Promise.reject();
}
this._pid[pid] = true;
return Promise.resolve();
}
release(pid) {
delete this._pid[pid];
}
}
const userLocks = new NamedLocks();
function handleRequest(data) {
userLocks.acquire(objectId)
.then(() => {
// semaphore doesn't exist, ok to proceed
console.log('starting new task');
fse.writeJson(__dirname + '/' + objectId, { objectId: objectId })
.then(stepOne).catch(rejectPromise)
.then(resp => stepTwo(resp, data)).catch(rejectPromise)
.then(resp => stepThree(resp, extra)).catch(rejectPromise)
.then(resp => stepFour(resp, argument)).catch(rejectPromise)
.then(sleep(20000))
.then(resp => userLocks.release(objectId))
.catch(resp => rejectionHandler(resp))
}).catch(() => {
// handle lock exist condition here
});
};
In this, you basically ask for a lock and if the lock exists, handle that in catch handler else do your thing and release the lock
I'm writing a bunch of mocha tests and I'd like to test that particular events are emitted. Currently, I'm doing this:
it('should emit an some_event', function(done){
myObj.on('some_event',function(){
assert(true);
done();
});
});
However, if the event never emits, it crashes the test suite rather than failing that one test.
What's the best way to test this?
If you can guarantee that the event should fire within a certain amount of time, then simply set a timeout.
it('should emit an some_event', function(done){
this.timeout(1000); //timeout with an error if done() isn't called within one second
myObj.on('some_event',function(){
// perform any other assertions you want here
done();
});
// execute some code which should trigger 'some_event' on myObj
});
If you can't guarantee when the event will fire, then it might not be a good candidate for unit testing.
Edit Sept 30:
I see my answer is accepted as the right answer, but Bret Copeland's technique (see answer below) is simply better because it's faster for when the test is successful, which will be the case most times you run a test as part of a test suite.
Bret Copeland's technique is correct. You can also do it a bit differently:
it('should emit an some_event', function(done){
var eventFired = false
setTimeout(function () {
assert(eventFired, 'Event did not fire in 1000 ms.');
done();
}, 1000); //timeout with an error in one second
myObj.on('some_event',function(){
eventFired = true
});
// do something that should trigger the event
});
This can be made a little shorter with help of Sinon.js.
it('should emit an some_event', function(done){
var eventSpy = sinon.spy()
setTimeout(function () {
assert(eventSpy.called, 'Event did not fire in 1000ms.');
assert(eventSpy.calledOnce, 'Event fired more than once');
done();
}, 1000); //timeout with an error in one second
myObj.on('some_event',eventSpy);
// do something that should trigger the event
});
Here we're checking that not only has the event fired, but also if if event has fired only once during the time-out period.
Sinon also supports calledWith and calledOn, to check what arguments and function context was used.
Note that if you expect the event to be triggered synchronously with the operation that triggered the event (no async calls in between) then you can do with a timeout of zero. A timeout of 1000 ms is only necessary when you do async calls in between which take a long time to complete. Most likely not the case.
Actually, when the event is guaranteed to fire synchronously with the operation that caused it, you could simplify the code to
it('should emit an some_event', function() {
eventSpy = sinon.spy()
myObj.on('some_event',eventSpy);
// do something that should trigger the event
assert(eventSpy.called, 'Event did not fire.');
assert(eventSpy.calledOnce, 'Event fired more than once');
});
Otherwise, Bret Copeland's technique is always faster in the "success" case (hopefully the common case), since it's able to immediately call done if the event is triggered.
This method ensures the minimum time to wait but the maximum opportunity as set by the suite timeout and is quite clean.
it('should emit an some_event', function(done){
myObj.on('some_event', done);
});
Can also use it for CPS style functions...
it('should call back when done', function(done){
myAsyncFunction(options, done);
});
The idea can also be extended to check more details - such as arguments and this - by putting a wrapper arround done. For example, thanks to this answer I can do...
it('asynchronously emits finish after logging is complete', function(done){
const EE = require('events');
const testEmitter = new EE();
var cb = sinon.spy(completed);
process.nextTick(() => testEmitter.emit('finish'));
testEmitter.on('finish', cb.bind(null));
process.nextTick(() => testEmitter.emit('finish'));
function completed() {
if(cb.callCount < 2)
return;
expect(cb).to.have.been.calledTwice;
expect(cb).to.have.been.calledOn(null);
expect(cb).to.have.been.calledWithExactly();
done()
}
});
Just stick:
this.timeout(<time ms>);
at the top of your it statement:
it('should emit an some_event', function(done){
this.timeout(1000);
myObj.on('some_event',function(){
assert(true);
done();
});`enter code here`
});
Late to the party here, but I was facing exactly this problem and came up with another solution. Bret's accepted answer is a good one, but I found that it wreaked havoc when running my full mocha test suite, throwing the error done() called multiple times, which I ultimately gave up trying to troubleshoot. Meryl's answer set me on the path to my own solution, which also uses sinon, but does not require the use of a timeout. By simply stubbing the emit() method, you can test that it is called and verify its arguments. This assumes that your object inherits from Node's EventEmitter class. The name of the emit method may be different in your case.
var sinon = require('sinon');
// ...
describe("#someMethod", function(){
it("should emit `some_event`", function(done){
var myObj = new MyObj({/* some params */})
// This assumes your object inherits from Node's EventEmitter
// The name of your `emit` method may be different, eg `trigger`
var eventStub = sinon.stub(myObj, 'emit')
myObj.someMethod();
eventStub.calledWith("some_event").should.eql(true);
eventStub.restore();
done();
})
})
Better solution instead of sinon.timers is use of es6 - Promises:
//Simple EventEmitter
let emitEvent = ( eventType, callback ) => callback( eventType )
//Test case
it('Try to test ASYNC event emitter', () => {
let mySpy = sinon.spy() //callback
return expect( new Promise( resolve => {
//event happends in 300 ms
setTimeout( () => { emitEvent('Event is fired!', (...args) => resolve( mySpy(...args) )) }, 300 ) //call wrapped callback
} )
.then( () => mySpy.args )).to.eventually.be.deep.equal([['Event is fired!']]) //ok
})
As you can see, the key is to wrap calback with resolve: (... args) => resolve (mySpy (... args)).
Thus, PROMIS new Promise().then() is resolved ONLY after will be called callback.
But once callback was called, you can already test, what you expected of him.
The advantages:
we dont need to guess timeout to wait until event is fired (in case of many describes() and its()), not depending on perfomance of computer
and tests will be faster passing
I do it by wrapping the event in a Promise:
// this function is declared outside all my tests, as a helper
const waitForEvent = (asynFunc) => {
return new Promise((resolve, reject) => {
asyncFunc.on('completed', (result) => {
resolve(result);
}
asyncFunc.on('error', (err) => {
reject(err);
}
});
});
it('should do something', async function() {
this.timeout(10000); // in case of long running process
try {
const val = someAsyncFunc();
await waitForEvent(someAsyncFunc);
assert.ok(val)
} catch (e) {
throw e;
}
}
I suggest using once() for an even simpler solution, especially if you like async/await style:
const once = require('events').once
// OR import { once } from 'events'
it('should emit an some_event', async function() {
this.timeout(1000); //timeout with an error if await waits more than 1 sec
p = once(myObj, 'some_event')
// execute some code which should trigger 'some_event' on myObj
await p
});
If you need to check values:
[obj] = await p
assert.equal(obj.a, 'a')
Finally, if you're using typescript, the following helper might be handy:
// Wait for event and return first data item
async function onceTyped<T>(event: string): Promise<T> {
return <T>(await once(myObj, event))[0]
}
Use like this:
const p = onceTyped<SomeEvent>(myObj, 'some_event')
// execute some code which should trigger 'some_event' on myObj
const someEvent = await p // someEvent has type SomeEvent
assert.equal(someEvent.a, 'a')