Only 'text' is output to the console, but 'text2' and 'text3' are not output, because exit from the stream is faster. This is the most simplified code of the real project structure. I can't figure out why this is happening and what to do about it.
This is the stream handler:
async function func1() {
await console.log('text2')
}
async function func() {
await console.log('text')
await func1();
await console.log('text3')
}
async function handler() {
await func()
await process.exit(123)
}
handler();
The only thing I can add is that the code above is contained in a handler.js file and is run like this:
const {Worker} = require('worker_threads')
const myWorker = new Worker('./handler.js')
myWorker.on('exit', (data) => {
console.log('Worker exit: ' + data)
})
И вывод следующий:
console
Despite the lack of a true, reproducible example, for your use case; this question is interesting and the answer wasn't obvious to find. I am not as read up on the worker_threads API as id like to be but ill offer my two cents.
After doing some research/testing (and not really having much to go on in terms of your specific use case), i believe it is because you are calling process.exit inside the worker thread. Since worker threads add their console statements to the main threads call stack, it takes some time before they all run. Calling process.exit here must be removing any remaining operations from that worker on the main threads call stack before they have a chance to run.
When you remove process.exit, all log statements run and the worker exits naturally.
If you need to close the worker thread at a particular point in time (which is what i think the real question here is), you might be better off sending a message back to the main thread using the parentPort.postMessage() method and then having the main thread terminate the worker:
// worker.js
const {parentPort} = require('worker_threads');
async function func1() {
await console.log('text2')
}
async function func() {
await console.log('text')
await func1();
await console.log('text3')
}
async function handler() {
await func()
parentPort.postMessage({kill: true});
}
handler();
Then listen for that message event and terminate the worker from the main thread:
// index.js
const {Worker} = require('worker_threads')
const myWorker = new Worker('./handler.js')
myWorker.on('exit', (data) => {
console.log('Worker exit: ' + data)
})
myWorker.on('message', async msg => {
if (msg.kill) {
console.log('killing worker with code');
await myWorker.terminate();
}
})
There are some gatchas here as the terminate method terminates the thread "as soon as possible" and you are relying on events and will need to prevent the worker from continuing on while the main thread executes the terminate method. However, without more information i cant be of much help. From our comments it might also be worth looking into child processes for this. Hope this helps
Related
I would like to run this code with babel:
redisClientAsync.delAsync('key');
return await someOtherAsyncFunction();
inside an async function without await the first line. is this OK?
how else can I run something that I don't care?
Can I just fire the non-promisified function del('key',null) without a callback?
Yes, you can do that, and it will run the two asynchronous functions in parallel. You've just created a promise and thrown it away.
However, this means that when the promise is rejected you won't notice. You'll just get an unhandledRejection eventually which will crash your process if not handled.
Is this OK? How can I run something that I don't care?
Probably it's not OK. If you truly wouldn't care, you hadn't run it in the first place. So you should be clear and explicit what you care about (and what not):
do you want to wait? (for side effects)
do you need the result?
do you want to catch exceptions?
If you only want to wait and don't care for the result value, you can easily throw away the result:
void (await someAsyncFunction()); // or omit the void keyword,
// doesn't make a difference in an expression statement
If you don't care about exceptions, you can ignore them using
… someAsyncFunction().catch(function ignore() {}) …
You can throw that away, await it, do anything with it.
If you want the result, you have to await it. If you care about exceptions, but don't really want to wait, you may want to execute it in parallel with the following functions:
var [_, res] = await Promise.all([
someAsyncFunction(), // result is ignored, exceptions aren't
someOtherAsyncFunction()
]);
return res;
inside an async function without await the first line. is this OK?
Yes, there are cases where you'd want to do this which are perfectly reasonable. Especially where you don't care about the result - one example is an analytics tracking operation that should not interfere with business critical code.
how else can I run something that I don't care?
In many ways, however simply calling the promise function works. Your del without a callback would probably work in this case but some functions don't guard against not passing callbacks, so you can pass an empty function instead (.del('key', () => {})).
You do want to however make sure that you know about it failing, even if you don't want to disrupt the operation of code - so please consider adding a process.on("unhandledRejection', event handler to explicitly ignore these particular exceptions or suppress them via:
redisClient.delAsync('key').catch(()=>{});
Or preferably, something like:
redisClient.delAsync('key').catch(logErr);
From all the research I've made so far, I think it's fine to do it, as long as you guarantee that the function you are not awaiting for guarantees a way to handle its own errors in case that happens. For example, a try-catch wrapping the whole function body, like you see in the following snippet for the asyncFunction.
It doesn't matter if the function throws synchronously or asynchronously. It guarantees the your mainFunction will complete no matter what. That's the key point here.
If you don't guarantee that, you have to risks:
If it throws synchronously, your main function will not complete.
If it throws asynchronously, you'll get an unhandled excepction
// THIS IS SOME API CALL YOU DON'T WANT TO WAIT FOR
const mockAPI = () => {
console.log("From mockAPI");
return new Promise((resolve,reject) => {
setTimeout(() => reject("LATE THROW: API ERROR"), 500);
});
};
// THIS IS THE SOME ASYNC FUNCTION YOU CALL BUT NOT AWAIT FOR
const asyncFunction = async (syncThrow) => {
try {
console.log("Async function START");
if (syncThrow) throw new Error("EARLY THROW");
await mockAPI();
console.log("Async function DONE");
}
catch(err) {
console.log("From async function catch");
console.log(err.message || err);
return;
}
};
// THIS IS YOUR MAIN FUNCTION
const mainFunction = async (syncThrow) => {
try {
console.clear();
console.log("Main function START");
asyncFunction(syncThrow);
console.log("Main function DONE <<< THAT'S THE IMPORTANT PART");
}
catch(err) {
console.log("THIS WILL NEVER HAPPEN");
console.log(err);
}
};
<div>
<button onClick="mainFunction(true)">Sync throw</button>
<button onClick="mainFunction(false)">Async throw</button>
</div>
Not in Node.js.
Node does not wait for ever-pending Promises. If other tasks are already completed and there is nothing left in the event loop, the Node process will be terminated even though there exists pending promise.
For the following script, if someOtherAsyncFunction() get resolved in 5 seconds, but redisClientAsync.delAsync('key') takes 10 seconds to execute, the Node process will be terminated after 5 seconds in theory, before the first line is resolved.
async function doSomething() {
redisClientAsync.delAsync('key');
return await someOtherAsyncFunction();
}
await doSomething();
I was looking around for my question but to no avail. I wanted to know if there was a way to force a worker to terminate a worker thread even if its task hadn't been completed. I have some code in there that calls itself if it somehow fails, which is intentional even though it might be bad code practice. I tried using libraries such as microjob and threads, but those didn't have what I wanted. Here is what I tried to do with threads but didn't work:
// Module part of threads (taskModule)
expose(async function startTask(taskData, profileData, taskID) {
await _Task.FetchData(taskData);
await _Task.GenerateSession(profileData);
await _Task.RequestPermission(taskID);
await _Task.SubmitOrder();
return "finished";
})
// Runs the module (main.js)
const {spawn, Thread, Worker} = require("threads");
let task = null;
async function test() {
task = await spawn(new Worker('./taskModule'))
startTask({},{},"0")
}
setTimeout(async function() {
await Thread.terminate(task)
}, 3000)
test().then(() => {
console.log('hello')
}).catch(console.error)
I was trying to test to see if the code would stop after 3 seconds to test, but it seems like it just continues. Is there a better way to do this type of task?
Thanks for any help. 😁
I have solved my issue by using clusters in Node.JS, which uses the child_process to run the workers.
Here is a link to the documentation if anyone needs it: https://nodejs.org/api/cluster.html
I would like to run this code with babel:
redisClientAsync.delAsync('key');
return await someOtherAsyncFunction();
inside an async function without await the first line. is this OK?
how else can I run something that I don't care?
Can I just fire the non-promisified function del('key',null) without a callback?
Yes, you can do that, and it will run the two asynchronous functions in parallel. You've just created a promise and thrown it away.
However, this means that when the promise is rejected you won't notice. You'll just get an unhandledRejection eventually which will crash your process if not handled.
Is this OK? How can I run something that I don't care?
Probably it's not OK. If you truly wouldn't care, you hadn't run it in the first place. So you should be clear and explicit what you care about (and what not):
do you want to wait? (for side effects)
do you need the result?
do you want to catch exceptions?
If you only want to wait and don't care for the result value, you can easily throw away the result:
void (await someAsyncFunction()); // or omit the void keyword,
// doesn't make a difference in an expression statement
If you don't care about exceptions, you can ignore them using
… someAsyncFunction().catch(function ignore() {}) …
You can throw that away, await it, do anything with it.
If you want the result, you have to await it. If you care about exceptions, but don't really want to wait, you may want to execute it in parallel with the following functions:
var [_, res] = await Promise.all([
someAsyncFunction(), // result is ignored, exceptions aren't
someOtherAsyncFunction()
]);
return res;
inside an async function without await the first line. is this OK?
Yes, there are cases where you'd want to do this which are perfectly reasonable. Especially where you don't care about the result - one example is an analytics tracking operation that should not interfere with business critical code.
how else can I run something that I don't care?
In many ways, however simply calling the promise function works. Your del without a callback would probably work in this case but some functions don't guard against not passing callbacks, so you can pass an empty function instead (.del('key', () => {})).
You do want to however make sure that you know about it failing, even if you don't want to disrupt the operation of code - so please consider adding a process.on("unhandledRejection', event handler to explicitly ignore these particular exceptions or suppress them via:
redisClient.delAsync('key').catch(()=>{});
Or preferably, something like:
redisClient.delAsync('key').catch(logErr);
From all the research I've made so far, I think it's fine to do it, as long as you guarantee that the function you are not awaiting for guarantees a way to handle its own errors in case that happens. For example, a try-catch wrapping the whole function body, like you see in the following snippet for the asyncFunction.
It doesn't matter if the function throws synchronously or asynchronously. It guarantees the your mainFunction will complete no matter what. That's the key point here.
If you don't guarantee that, you have to risks:
If it throws synchronously, your main function will not complete.
If it throws asynchronously, you'll get an unhandled excepction
// THIS IS SOME API CALL YOU DON'T WANT TO WAIT FOR
const mockAPI = () => {
console.log("From mockAPI");
return new Promise((resolve,reject) => {
setTimeout(() => reject("LATE THROW: API ERROR"), 500);
});
};
// THIS IS THE SOME ASYNC FUNCTION YOU CALL BUT NOT AWAIT FOR
const asyncFunction = async (syncThrow) => {
try {
console.log("Async function START");
if (syncThrow) throw new Error("EARLY THROW");
await mockAPI();
console.log("Async function DONE");
}
catch(err) {
console.log("From async function catch");
console.log(err.message || err);
return;
}
};
// THIS IS YOUR MAIN FUNCTION
const mainFunction = async (syncThrow) => {
try {
console.clear();
console.log("Main function START");
asyncFunction(syncThrow);
console.log("Main function DONE <<< THAT'S THE IMPORTANT PART");
}
catch(err) {
console.log("THIS WILL NEVER HAPPEN");
console.log(err);
}
};
<div>
<button onClick="mainFunction(true)">Sync throw</button>
<button onClick="mainFunction(false)">Async throw</button>
</div>
Not in Node.js.
Node does not wait for ever-pending Promises. If other tasks are already completed and there is nothing left in the event loop, the Node process will be terminated even though there exists pending promise.
For the following script, if someOtherAsyncFunction() get resolved in 5 seconds, but redisClientAsync.delAsync('key') takes 10 seconds to execute, the Node process will be terminated after 5 seconds in theory, before the first line is resolved.
async function doSomething() {
redisClientAsync.delAsync('key');
return await someOtherAsyncFunction();
}
await doSomething();
I'm experimenting with Puppeteer Cluster and I just don't understand how to use queuing properly. Can it only be used for calls where you don't wait for a response? I'm using Artillery to fire a bunch of requests simultaneously, but they all fail while only some fail when I have the command execute directly.
I've taken the code straight from the examples and replaced execute with queue which I expected to work, except the code doesn't wait for the result. Is there a way to achieve this anyway?
So this works:
const screen = await cluster.execute(req.query.url);
But this breaks:
const screen = await cluster.queue(req.query.url);
Here's the full example with queue:
const express = require('express');
const app = express();
const { Cluster } = require('puppeteer-cluster');
(async () => {
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 2,
});
await cluster.task(async ({ page, data: url }) => {
// make a screenshot
await page.goto('http://' + url);
const screen = await page.screenshot();
return screen;
});
// setup server
app.get('/', async function (req, res) {
if (!req.query.url) {
return res.end('Please specify url like this: ?url=example.com');
}
try {
const screen = await cluster.queue(req.query.url);
// respond with image
res.writeHead(200, {
'Content-Type': 'image/jpg',
'Content-Length': screen.length //variable is undefined here
});
res.end(screen);
} catch (err) {
// catch error
res.end('Error: ' + err.message);
}
});
app.listen(3000, function () {
console.log('Screenshot server listening on port 3000.');
});
})();
What am I doing wrong here? I'd really like to use queuing because without it every incoming request appears to slow down all the other ones.
Author of puppeteer-cluster here.
Quote from the docs:
cluster.queue(..): [...] Be aware that this function only returns a Promise for backward compatibility reasons. This function does not run asynchronously and will immediately return.
cluster.execute(...): [...] Works like Cluster.queue, just that this function returns a Promise which will be resolved after the task is executed. In case an error happens during the execution, this function will reject the Promise with the thrown error. There will be no "taskerror" event fired.
When to use which function:
Use cluster.queue if you want to queue a large number of jobs (e.g. list of URLs). The task function needs to take care of storing the results by printing them to console or storing them into a database.
Use cluster.execute if your task function returns a result. This will still queue the job, so this is like calling queue in addition to waiting for the job to finish. In this scenario, there is most often a "idling cluster" present which is used when a request hits the server (like in your example code).
So, you definitely want to use cluster.execute as you want to wait for the results of the task function. The reason, you do not see any errors is (as quoted above) that the errors of the cluster.queue function are emitted via a taskerror event. The cluster.execute errors are directly thrown (Promise is rejected). Most likely, in both cases your jobs fail, but it is only visible for the cluster.execute
I would like to run this code with babel:
redisClientAsync.delAsync('key');
return await someOtherAsyncFunction();
inside an async function without await the first line. is this OK?
how else can I run something that I don't care?
Can I just fire the non-promisified function del('key',null) without a callback?
Yes, you can do that, and it will run the two asynchronous functions in parallel. You've just created a promise and thrown it away.
However, this means that when the promise is rejected you won't notice. You'll just get an unhandledRejection eventually which will crash your process if not handled.
Is this OK? How can I run something that I don't care?
Probably it's not OK. If you truly wouldn't care, you hadn't run it in the first place. So you should be clear and explicit what you care about (and what not):
do you want to wait? (for side effects)
do you need the result?
do you want to catch exceptions?
If you only want to wait and don't care for the result value, you can easily throw away the result:
void (await someAsyncFunction()); // or omit the void keyword,
// doesn't make a difference in an expression statement
If you don't care about exceptions, you can ignore them using
… someAsyncFunction().catch(function ignore() {}) …
You can throw that away, await it, do anything with it.
If you want the result, you have to await it. If you care about exceptions, but don't really want to wait, you may want to execute it in parallel with the following functions:
var [_, res] = await Promise.all([
someAsyncFunction(), // result is ignored, exceptions aren't
someOtherAsyncFunction()
]);
return res;
inside an async function without await the first line. is this OK?
Yes, there are cases where you'd want to do this which are perfectly reasonable. Especially where you don't care about the result - one example is an analytics tracking operation that should not interfere with business critical code.
how else can I run something that I don't care?
In many ways, however simply calling the promise function works. Your del without a callback would probably work in this case but some functions don't guard against not passing callbacks, so you can pass an empty function instead (.del('key', () => {})).
You do want to however make sure that you know about it failing, even if you don't want to disrupt the operation of code - so please consider adding a process.on("unhandledRejection', event handler to explicitly ignore these particular exceptions or suppress them via:
redisClient.delAsync('key').catch(()=>{});
Or preferably, something like:
redisClient.delAsync('key').catch(logErr);
From all the research I've made so far, I think it's fine to do it, as long as you guarantee that the function you are not awaiting for guarantees a way to handle its own errors in case that happens. For example, a try-catch wrapping the whole function body, like you see in the following snippet for the asyncFunction.
It doesn't matter if the function throws synchronously or asynchronously. It guarantees the your mainFunction will complete no matter what. That's the key point here.
If you don't guarantee that, you have to risks:
If it throws synchronously, your main function will not complete.
If it throws asynchronously, you'll get an unhandled excepction
// THIS IS SOME API CALL YOU DON'T WANT TO WAIT FOR
const mockAPI = () => {
console.log("From mockAPI");
return new Promise((resolve,reject) => {
setTimeout(() => reject("LATE THROW: API ERROR"), 500);
});
};
// THIS IS THE SOME ASYNC FUNCTION YOU CALL BUT NOT AWAIT FOR
const asyncFunction = async (syncThrow) => {
try {
console.log("Async function START");
if (syncThrow) throw new Error("EARLY THROW");
await mockAPI();
console.log("Async function DONE");
}
catch(err) {
console.log("From async function catch");
console.log(err.message || err);
return;
}
};
// THIS IS YOUR MAIN FUNCTION
const mainFunction = async (syncThrow) => {
try {
console.clear();
console.log("Main function START");
asyncFunction(syncThrow);
console.log("Main function DONE <<< THAT'S THE IMPORTANT PART");
}
catch(err) {
console.log("THIS WILL NEVER HAPPEN");
console.log(err);
}
};
<div>
<button onClick="mainFunction(true)">Sync throw</button>
<button onClick="mainFunction(false)">Async throw</button>
</div>
Not in Node.js.
Node does not wait for ever-pending Promises. If other tasks are already completed and there is nothing left in the event loop, the Node process will be terminated even though there exists pending promise.
For the following script, if someOtherAsyncFunction() get resolved in 5 seconds, but redisClientAsync.delAsync('key') takes 10 seconds to execute, the Node process will be terminated after 5 seconds in theory, before the first line is resolved.
async function doSomething() {
redisClientAsync.delAsync('key');
return await someOtherAsyncFunction();
}
await doSomething();