I have a NextJS application that runs a recursive setTimeout when the server is started. I need to create an API endpoint that can start and stop this loop (to have more control over it in production). This loop is used to process items in a database that are added from another API endpoint.
import { clearTimeout } from "timers";
var loopFlag = true;
export function loopFlagSwitch(flag: boolean) {
loopFlag = flag;
}
export async function loop() {
try {
// Retrieve all unprocessed transactions
const unprocessedTransactions = await prisma.transaction.findMany({
take: 100,
where: { status: "UNPROCESSED" },
});
// Loop through transactions and do stuff
for (const transaction of unprocessedTransactions) {
//stuff
}
} catch (e) {
// handle error
}
if (loopFlag === true) {
setTimeout(loop, 1000); //if flag changes, this will stop running
}
}
if (require.main === module) {
loop(); // This is called when server starts, but not when file is imported
}
The reason I use setTimeout and not setInterval is because many errors can occur when processing items retrieved from DB. These errors, however, are solved by waiting a few milliseconds. So, the benefit of the pattern below is that if an error happens, the loop immediately restarts and the error will not appear because a ms has passed (it's due to concurrency problems -- let's ignore this for now).
To attempt to start and stop this loop, I have an endpoint that simply calls the loopFlagSwitch function.
import { NextApiRequest, NextApiResponse } from "next";
import { loopFlagSwitch } from "services/loop";
async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
loopFlagSwitch(req.body.flag);
} catch (error) {
logger.info({ error: error });
}
}
export default handler;
Problem is, even when this endpoint is called, the setTimeout loop keeps going. Why isn't it picking the change in flag?
clearTimeout()
The global clearTimeout() method cancels a timeout previously established by calling setTimeout().
To clear a timeout, use the id returned from setTimeout():
Usage
const myTimeout = setTimeout(function, milliseconds);
//Then you can to stop the execution by calling clearTimeout():
clearTimeout(myTimeout);
loopFlag as a condition
...
if (loopFlag === true) {
myTimeout();
} else {
clearTimeout(myTimeout)
}
...
Add abortTimer function
Full code
export function loopFlagSwitch(flag) {
flag === true ? loop : abortTimer()
}
// set timeout
var myTimeout = setTimeout(loop, 1000);
function abortTimer() { // to be called when you want to stop the timer
clearTimeout(myTimeout);
}
export async function loop() {
try {
// Retrieve all unprocessed transactions
let d = "Retrieve all unprocessed transactions"
process.stdout.write(d + '\n');
// Loop through transactions and do stuff
for (let i = 0; i<10; i++) {
//stuff
let c = "second loop"
process.stdout.write(c + '\n');
}
} catch (e) {
// handle error
console.log("error ", e)
} finally {
myTimeout = setTimeout(loop, 1000); // repeat myself
}
}
if (require.main === module) {
loop(); // This is called when server starts, but not when file is imported
}
The flag will not work because node doesn't maintain the state of a file, the import only cares about the things it obtains from a file, it doesn't mind about the state of the variables declared in it.
Even though the clearTimeout() function may be sufficient, i think there is an even better option you can use to stop this loop.
Use a JS Class!
Instead of using just a function without state. You could instantiate a class that runs on the server with an internal boolean that can be called "shouldKeepLooping" for example:
class NextJsLooper { // or whatever better name you can use
private shouldKeepLooping: boolean
constructor () {
this.shouldKeepLooping = true
}
public shouldKeepLooping(value) { this.shouldKeepLooping = value }
public async loop () {
if (shouldKeepLooping) {
//... rest of the loop code
setTimeout(this.loop, 1000);
}
}
}
This way if you set the value to false it will automatically stop. Since this is a reference to the object.
Keep in mind that you would need to keep this instance alive as probably something global, and would need it to be accesible by nextJS.
You can use Symbol.for and the Node global to mantain the instance saved somewhere in the server!
My suggestion is to introduce the use of signals.
Service like Pusher will trigger the event that will be listened by the transaction processor.
your transaction processing api / code above or any other
Signal actions like "start" and "stop" will be triggered anytime even in production by either frontend or through the pusher portal that will be used to change the loop flag to either true or false.
you can retrieve the thread in charge of the operation and interrupt after a given expected time ,and log something to inform you about the time out in case that your operation took more than the necessary time .
Related
I am making a chrome extension (mv3). Based on user activity, the content.js passes a message to the background.js which then calls an async function to add data in Google Docs using Docs API.
I want each request to execute only after the previous one has finished running. I am using chrome.runtime.sendMessage to send a message from content.js and don't see a way of calling background.js serially from there. So I need a way of executing them one by one in background.js only. The order of these requests is also important (but if the order of the requests gets changed by one/two places, I think that would still be okay from a user perspective).
I tried something and it is working but I am not sure if I am missing some edge cases, because I was unable to find the approach in any other answers -
Semaphore-like queue in javascript?
Run n number of async function before calling another method in nodejs
JavaScript: execute async function one by one
The approach I used is: I use a stack like structure to store requests, use setInterval to check for any pending requests and execute them serially.
content.js:
chrome.runtime.sendMessage({message});
background.js:
let addToDocInterval = "";
let addToDocCalls = [];
async function addToDoc(msg) {
// Await calls to doc API
}
async function addToDocHelper() {
if(addToDocCalls.length === 0)
return;
clearInterval(addToDocInterval)
while(addToDocCalls.length > 0) {
let msg = addToDocCalls.shift();
await addToDoc(msg);
}
addToDocInterval = setInterval(addToDocHelper, 1000);
}
chrome.runtime.onMessage.addListener((msg) => {
// Some other logic
addToDocCalls.push(msg);
})
addToDocInterval = setInterval(addToDocHelper, 1000);
Is this approach correct? Or is there any better way to do this?
I'd suggest changing several things.
Don't use timers polling the array. Just initiate processing the array anytime you add a new item to the array.
Keep a flag on whether if you're already processing the array so you don't start duplicate processing.
Use a class to encapsulate this functionality into an object.
Encapsulate the addToDocCalls array and adding to it so your class is managing it and outside code just calls a function to add to it which also triggers the processing. Basically, you're making it so callers don't have to know how the insides work. They just call helper.addMsg(msg) and the class instance does all the work.
Here's an implementation:
async function addToDoc(msg) {
// Await calls to doc API
}
class docHelper {
constructor() {
this.addToDocCalls = [];
this.loopRunning = false;
}
addMsg(msg) {
// add item to the queue and initiate processing of the queue
this.addToDocCalls.push(msg);
this.process();
}
async process() {
// don't run this loop twice if we're already running it
if (this.loopRunning) return;
try {
this.loopRunning = true;
// process all items in the addToDocCalls we have
while(this.addToDocCalls.length > 0) {
let msg = addToDocCalls.shift();
await addToDoc(msg);
}
} finally {
this.loopRunning = false;
}
}
}
const helper = new docHelper();
chrome.runtime.onMessage.addListener((msg) => {
// Some other logic
helper.addMsg(msg);
});
So, process() will run until the array is empty. Any interim calls to addMsg while process() is running will add more items to array and will call process() again, but the loopRunning flag will keep it from starting duplicate processing loops. If addMsg() is called while process is not running, it will start the process loop.
P.S. You also need to figure out what sort of error handling you want if addToDoc(msg) rejects. This code protects the this.loopRunning flag if it rejects, but doesn't actually handle a reject error. In code like this that is processing a queue, often times all you can really do is log the error and move on, but you need to decide what is the proper course of action on a rejection.
You don't need to use setTimeout. You do not even need a while loop.
let addToDocInterval = "";
let addToDocCalls = [];
let running = false;
async function addToDoc(msg) {
// Await calls to doc API
}
async function addToDocHelper() {
if(running || addToDocCalls.length === 0)
return;
running = true;
let msg = addToDocCalls.shift();
await addToDoc(msg);
running = false;
addToDocHelper();
}
chrome.runtime.onMessage.addListener((msg) => {
// Some other logic
addToDocCalls.push(msg);
addToDocHelper();
});
The code should be self explanatory. There is no magic.
Here is a generic way to run async tasks sequentially (and add more tasks to the queue at any time).
const tasks = [];
let taskInProgress = false;
async function qTask(newTask) {
if (newTask) tasks.push(newTask);
if (tasks.length === 0) return;
if (taskInProgress) return;
const nextTask = tasks.shift();
taskInProgress = true;
try {
await nextTask();
} finally {
taskInProgress = false;
//use setTimeout so call stack can't overflow
setTimeout(qTask, 0);
}
}
//the code below is just used to demonstrate the code above works
async function test() {
console.log(`queuing first task`);
qTask(async () => {
await delay(500); //pretend this task takes 0.5 seconds
console.log('first task started');
throw 'demonstrate error does not ruin task queue';
console.log('first task finished');
});
for (let i = 0; i < 5; i++) {
console.log(`queuing task ${i}`)
qTask(async () => {
await delay(200); //pretend this task takes 0.2 seconds
console.log(`task ${i} ran`);
});
}
await delay(1000); //wait 1 second
console.log(`queuing extra task`);
qTask(async () => {
console.log('extra task ran');
});
await delay(3000); //wait 3 seconds
console.log(`queuing last task`);
qTask(async () => {
console.log('last task ran');
});
}
test();
function delay(ms) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
I'm designing a system where a user will interact with my RESTful API through JS, and will replace the ts html element with a different SVG if the returned answer is true. If the API call returns false, the JS should wait for 5 seconds, before retrying the API call. again, if the API call returns true, the SVG will be changed, and the function will stop calling the API. How can I have the function "sleep" for 5 seconds before running again, and ending the function if the returned API call is true?
Code:
async function getTransaction(){
var lookup = $("meta[name='lookup']").attr("content");
axios.get('http://127.0.0.1:5000/client_api/transaction_status/' + lookup)
.then(function (response) {
var tStat = response.data.transactionStatus;
console
if(tStat){
document.getElementById("ts").innerHTML = 'drawing the new svg and replacing the old one'
console.log('transaction_found')
}
else if(!tStat){
console.log("transaction not found")
}
else{
console.log("error")
}
});
}console.log("hello"); while(true){setTimeout(getTransaction,5000);}
Don't use sleep sice it pauses your programm to pause. Instead use setTimeout like so. This creates an event listener which calls the function again after 5s if tStat is false.
In case of false the function will set an timeout after which the function is called again. After the timeout is completed it clears itself. This step repeats until the api request eventually resolves to true
async function getTransaction() {
var lookup = $("meta[name='lookup']").attr("content");
axios.get('http://127.0.0.1:5000/client_api/transaction_status/' + lookup)
.then(function(response) {
var tStat = response.data.transactionStatus;
console
if (tStat) {
document.getElementById("ts").innerHTML = 'drawing the new svg and replacing the old one'
console.log('transaction_found')
} else {
console.log("transaction not found")
setTimeout(getTransaction, 5000);
}
});
}
console.log("hello");
I have two files, main.js and job.js. When a button is clicked in main.js, I want a new, seperate process of the function in job.js to run.
What this process does is launch a new puppeteer browser instance. When the stop button is clicked, this process should be killed by pid. (For this we use process.kill(child.pid)?)
So would I want to use a worker or child process, and if any of those two, how would I implement it so it runs this function?
Important note: every time the start button is clicked, I would like a new process running the function to be started, so the specific process with that pid can be killed.
I suggest you to use a wrapper module for the child_process module. Example usage with execa module.
Main.js
const { execa } = require('execa')
// function for spawning a process with cancel handle!.
async function spawnSubprocess(command, args, cb) {
let subprocess = execa(command, args);
// create a cancel function for later usage!.
function cancel() {
if(subprocess) {
subprocess.kill('SIGTERM', {
// wait for it to terminate before killing it.
forceKillAfterTimeout: 1500
});
// set to null so it won't be killed mutliple times.
subprocess = null
}
}
// add the event listener to subprocess when it's done!
// Can be used for logging or for correctly awaiting a process
// termination before proceeding.
subprocess.then(() => {
subprocess = null
cb()
})
.catch(err => {
subprocess = null
cb(err)
})
// return the cancel handler !.
return cancel
}
// reference to the last cancel. It has to be accessible in
// onClickHandler ofc!.
var processCancel = null
// call this function on click.
// keep the processCancel in scope!
function onClickHandler() {
// first, check if a process is already running
if(typeof processCancel === 'function') {
console.log('Process already running. Calling cancel!')
// the process is not directly terminated. You amy have
// 2 seconds where multiple instances are running but that's not a big deal i guess.
processCancel()
}
// spawn the new process !
// adjust the path to job.js ofc!.
processCancel = spawnSubprocess('node', ['job.js'], (err) => {
// on done callback!. Log some information!.
if(err) {
console.error('Process error ', err)
} else {
console.log('process stopped / DONE')
}
processCancel = null
})
}
This should give you an idea on how to implement it. I suggest to use child_process or any wrapper module. ^^
Background
As to my knowledge, Firebase-triggered functions can run more than once from just one triggering event (I think I can see that from the logs in Firebase).
Since this is potentially a data-corrupting behavior, I would like to implement some flag, which allows the function to stop if it finds that the flag is true.
Upon creation of that object in the database I would like this function (Typescript) to run only once.
For example:
adm.database().ref('/Users/{UserID}')
I plan to store a key value pair for each user node in the database:
{"alreadyTriggered":true}
Or store the 'trigger' in a separate node, it does not matter.
And then when Firebase decides to trigger the respective function, I will do a check and stop the execution of the function if it is not the first time it is being run:
data = snapshot.val();
if (data.alreadyTriggered) {
return;
}
// The function continues here
Implementation
(some parentheses and brackets can be missing, it's a psecudocode-Typescript written in the browser window)
index.ts:
import * as u from './users';
import * as ff from 'firebase-functions';
import * as adm from 'firebase-admin';
import * as types from './types';
// Triggered function
export const userCreated = ff.database.ref('/Users/{userId}').onCreate((sn, ctx) => {
let user: types.User = sn.val();
console.log(`New user created, User Id: ${ctx.params.userId}`);
return u.onCreated(user);
});
users.ts:
// Function implementation
export function onCreated(user: types.User): Promise<void> {
return adm.database().ref("/Triggers").once('value', snapshot => {
let data: any = snapshot.val();
const alreadyTriggered: boolean = data.alreadyTriggered;
}).then(alreadyTriggered => {
if (alreadyTriggered) {
// We would like to stop any execution here
return
}
else {
console.log(`Continuing processing`)
}
}).then(() => {
// This code should be reached only if alreadyTriggered is set to false
// do actual work with the user object
//...someNewData
//then save it
adm.database().ref('/Users').child(user.userId).update(someNewData)
})
}
Problem
The problem is that is that I don't know how reliably to stop executing the function. I have tried to use return (doesn't work), break (only for loops) and I came to using throw new Error('Stopping function execution'), but it does not seem to stop reliably - sometimes I see in the logs that the function does continue the execution although alreadyTriggered is definitely true.
Question
How can I stop execution of a Firebase Typescript function from within its code?
Your return statement isn't returning from the top-level function. It's just returning from the lambda function you passed to then(). You will have to propagate information down the chain of promises to tell the next callback whether or not to do its work.
return adm.database().ref("/Triggers").once('value', snapshot => {
let data: any = snapshot.val();
const alreadyTriggered: boolean = data.alreadyTriggered;
}).then(alreadyTriggered => {
if (alreadyTriggered) {
// We would like to stop any execution here
return false
}
else {
console.log(`Continuing processing`)
return true
}
}).then((continue) => {
// This code should be reached only if alreadyTriggered is set to false
// do actual work with the user object
//...someNewData
//then save it
if (continue) {
return adm.database().ref('/Users').child(user.userId).update(someNewData)
}
})
You also missed a return of the promise returned by update above, which would cause your function to time out.
Note that this is all lot easier with async/await syntax.
Having a websocket connection, I try to get information if the connection is upright. That I try to do with RxJS and an interval. The problem is that after one timeout the stream ends and I want the interval to continue afterwards so that I can see if it already reconnected.
function listenToConnectionLost () {
return rx.Observable
.interval(5000) // every 5 seconds
.flatMap(tryPing)
.distinctUntilChanged()
// so I want to have either "connected" or "timeout" here
// onNext I want to handle the different outputs
;
}
function tryPing () {
var pingPromise = getPingPromise();
var pingObservable = rx.Observable
.fromPromise(pingPromise)
.timeout(5000)
.catch(Rx.Observable.just('timeout')) // catches error, but
// completes the stream
;
return pingObservable;
}
function getPingPromise () {
// returns a promise, which resolves when connection is upright
}
Here I also have a live example with a "faked" interval: http://jsbin.com/gazuvu/4/edit?js,console
Thanks!
Apperently the above posted code is working, because completed stream will be fed into the flatMap in listenToConnectionLost. I tested it by inserting the following code into getPingPromise.
function getPingPromise () {
if(Math.random() < 0.5) {
return $q.when('connected'); // resolves immediately
}
else {
return $q.defer().promise; // never resolves, triggers timeout
}
}