Preface:
In order to solve my problem, you have to have knowledge in the following areas: thread-safety, Promise, async-await.
For people who are not familiar with TypeScript, it's just normal JavaScript (ES6) with type annotations.
I have a function named excludeItems that accepts an item list (each item is a string), and calls an API (that excludes the item) for each item. It's important not to call the API twice for the same item, not even in different executions of the function, so I save in a local DB the items that are already excluded.
async function excludeItems(items: string[]) {
var excludedItems = await db.getExcludedItems();
for (var i in items) {
var item = items[i];
var isAlreadyExcluded = excludedItems.find(excludedItem => excludedItem == item);
if (isAlreadyExcluded) {
continue;
}
await someApi.excludeItem(item);
await db.addExcludedItem(item);
}
}
This function is called asynchronously by its client several times instantaneously, meaning the client calls the function say 5 times before the first execution is completed.
A concrete scenario:
excludeItems([]);
excludeItems(['A','B','C']);
excludeItems([]);
excludeItems(['A','C']);
excludeItems([]);
In this case, although Node.js is single-threaded, the Critical Section problem is existing here, and I get the wrong results. This is my "excludedItems" collection in my local DB after the execution of that scenario:
[{item: 'A'},
{item: 'B'},
{item: 'C'},
{item: 'A'},
{item: 'C'}]
As you can see, the last 'A' and 'C' are redundant (meaning that the API was also called twice for these items).
It occurs due to the await statements in the code. Every time an await statement is reached, a new Promise is created under the hood, therefore although Node.js is single-threaded, the next async function that was waiting to be executed is getting executed, and that way this critical section is executed parallelly.
To solve that problem, I've implemented a locking mechanism:
var excludedItemsLocker = false;
async function safeExcludeItems(items: string[]) {
while (excludedItemsLocker) {
await sleep(100);
}
try {
excludedItemsLocker = true;
var excludedItems: string[] = await db.getExcludedItems();
for (var i in items) {
var item = items[i];
var isAlreadyExcluded = excludedItems.find(excludedItem => excludedItem == item);
if (isAlreadyExcluded) {
continue;
}
await someApi.excludeItem(item);
await db.addExcludedItem(item);
}
}
finally {
excludedItemsLocker = false;
}
}
async function sleep(duration: number): Promise<Object> {
return new Promise(function (resolve) {
setTimeout(resolve, duration);
});
}
However, this implementation does not work for some reason. I still get more than one (alleged) "thread" in the critical section, meaning it's still getting executed parallelly and my local DB is filled with the same wrong results. BTW the sleep method works as expected, its purpose is just to give CPU time to the next function call that's waiting to be executed.
Does anybody see what's broken in my implementation?
BTW I know that I can achieve the same goal without implementing a Lock, for example by calling to db.getExcludedItems inside the loop, but I want to know why my Lock implementation is broken.
If the parameters are:
['A','B','C']
and db.getExcludedItems() returns:
[{item: 'A'},
{item: 'B'},
{item: 'C'}]
Then you are trying to find a string in an array of objects, which will always return undefined:
var isAlreadyExcluded = excludedItems.find(excludedItem => excludedItem == item);
Just a thought, because I can't see any problem with the locking itself, it should work as expected.
I encountered a similar problem a while back and I ended up implementing a ticket system where each "thread" would request a ticket and wait in a queue (I know it's not a thread, but it's easier to say than 'next set of functions in the event loop'). It's an NPM package found at promise-ticket, but the crux of the solution was to have a generator function returning promises that would resolve when an EventEmitter would emit it's ticket number
let emitter = new EventEmitter();
let nextTicket = 0;
let currentTicket = 0;
let skips = [];
const promFn = (resolve) => {
let num = currentTicket++;
emitter.once(num, () => resolve(num));
};
const generator = (function* () {
while(true) yield new Promise(promFn);
})();
// Someone takes a ticket from the machine
this.queue = (resolveValue) => {
let ticketNumber = currentTicket;
let p = generator.next().value;
if(resolveValue !== undefined) p = p.then(() => resolveValue);
if(skips.includes(ticketNumber)) emitter.emit(ticketNumber);
return p;
};
Using promise-ticket is pretty easy:
const TicketMachine = require("promise-ticket");
const tm = new TicketMachine();
// Queue some tickets
tm.queue("resolve1"); // id=0
tm.queue("resolve2"); // id=1
tm.queue("resolve4"); // id=2
tm.queue("resolve3"); // id=3
// Call those tickets
tm.next(); // resolve1
tm.next(); // resolve2
tm.next(3); // resolve3
tm.next(2); // resolve4
I guess it's not a perfect solution, but IMO it's better than wrapping entire code blocks in callbacks.
I guess for your problem (if you still have it 5 years later), you should query your localstorage and abort if those items have been sent (by calling ticketMachine.next()).
Related
this is more of a opinionated question. I do have a working solution, but I'm not 100% comfortable with it, as it has it's flaws.
Maybe someone can help me to improve this.
Goal:
I have an external api that only allows 4 calls to be made concurrently against it (for each user). Our app can impersonate multiple users at once.
So the issue comes, if more than 4 calls are made against the api simultaneously. (sometimes more than 20 calls are made)
Using a batched approach with Promise.all and chunking would be very inefficient, as the calls have a different runtime each.
Ideally, the queue would work FIFO and as soon as one call finishes, the next call is started. At the current standing, I created 4 own FIFO queues with somewhat of a Promise chain and I fill these evenly (if all are running).
The problem that I have is, that I do not know how long a request is running.
So choosing one of the queues can lead to an overall longer runtime as necessary.
The calls are automatically rejected after 30s from the external api, so no dead lock here.
Another problem that I have is, that I have to return the data provided from the api to a dependency. The queue is called/filled from withan a callback function...
This is wrapped in a Promise itself, so we can wait as long as we want.
But storing the values in a cache for later retrieval is no option either.
So long story short, here is the code
class QueuingService {
/** ~FIFO queue */
private static queue: Record<string, { promise: Promise<Function>, uuid: string }[]> = {};
private static MAX_CONCURRENCY = 4
/** cacheKey is used as cachekey for grouping in the queue */
public static async addToQueueAndExecute(func: Function, cacheKey: string) {
let resolver: Function | null = null;
let promise = new Promise<Function>(resolve => {
resolver = resolve;
});
//in the real world, this is a real key created by nanoId
let uuid = `${Math.random()}`;
Array.isArray(this.queue[cacheKey])
? this.queue[cacheKey].push({ promise: promise, uuid: uuid })
: this.queue[cacheKey] = [{ promise: promise, uuid: uuid }];
//queue in all calls, until MAX_CONCURRENCY is reached. After that, slice the first entry and await the promise
if (this.queue[cacheKey].length > this.MAX_CONCURRENCY) {
let queuePromise = this.queue[cacheKey].shift();
if (queuePromise){
await queuePromise.promise;
}
}
//console.log("elements in queue:", this.queue[cacheKey].length, cacheKey)
//technically this wrapping is not necessary, but it makes to code more readable imho
let result = async () => {
let res = await func();
if (resolver) {
//resolve and clean up
resolver();
this.queue[cacheKey] = this.queue[cacheKey].filter(elem => elem.uuid !== uuid);
}
//console.log(res, cacheKey, "finshed after", new Date().getTime() - enteredAt.getTime(),"ms", "entered at:", enteredAt)
return res;
}
return await result();
}
}
async function sleep(ms:number){
return await new Promise(resolve=>window.setTimeout(resolve,ms))
}
async function caller(){
/* //just for testing with fewer entries
for(let i=0;i<3;i++){
let groupKey = Math.random() <0.5? "foo":"foo"
QueuingService.addToQueueAndExecute(async ()=>{
await sleep(Math.floor(4000-Math.random()*2000));
console.log(i, new Date().getSeconds(), new Date().getMilliseconds(),groupKey)
return Math.random()
},groupKey)
}
*/
for(let i=0;i<20;i++){
let groupKey = Math.random() <0.5? "foo":"foo";
let startedAt = new Date();
QueuingService.addToQueueAndExecute(async ()=>{
await sleep(Math.floor(4000-Math.random()*2000));
console.log(i, new Date().getTime()-startedAt.getTime(),groupKey)
return Math.random()
},groupKey)
}
}
caller()
Also, here is a playground with the code to play around with:
https://www.typescriptlang.org/play?#code/MYGwhgzhAECKCuBTeBLAdgcwMqIE4DcVhFoBvAWACgBIAegCp7oA-AMQElWB5aARySTR6tKtQAOuFPjAAXEhBmyifAYgBc0AEqJgAe1wATADwLJmADRloE3QFsUEddAAKuOw8RHW8NMBkpdNAA+S3hUAw1TdAxoAF8AbQBdIOgAXjJYgG5RCSlZeUV-YGgAWQBBAA0AfQBhLgA5GoBVTU0AUUaATTToABYqUQYmYDBgAAtEAGlEAE9oB2h4RwNoSGgR8cQAa1noADN9aAw3eDFo+bRoGQmVZBJhHPgAIxBlBSViyBnfVYMDABVdAg7mU0AY2gAPHTwOQACj2PmAGm8vn8gUsGwm0xmkRkZgwAEoyKJqCBEDJoLhEBBdCB8HhkYi0ZcAD7QNDwEAgHocrnZGik8nWNz2Rw8xAAdxcIo8XiZAWCsKpNLpJFSKQoAuoytp9NwPR1qv51GosQJ-OglqtltotHQVxuVLA3Il+hABks1wWCzAlMQzugOzmwCdchWTzmaDAaF07AMJLJFLCKBW6QABgASUglWRjAB0uGjBjssIJsTT-IGArKuELMzzDhrddhXogef4d3imKms0SBJJ1AA-A6HO3VF3Rlje3mxEsxrDSML3I4NDZRYhQuENMmVmaBxpW2PO93sYkevFF2uPKuZY5Nynt+E4olKwLbR3BPbndyRlyIKE0H8blymqOpGhadounmGAnU2Aw82gMo9jkfVrlkSwIFeYgHRIPYUFwBRoEQQDcDmItVglMAUApa4SCvRwSRQPZoBbMZRw-RAJ02U88zJTBrmgFJDxA2oGmaVoOhqToiU1E1BQpDjXGXNURzbDiuKnGZEjzCA2OQ0tjRNJiWMU29EAJWS5LASjqNuJAlPXGczIta1XMtWISQ8yg3JtWg9DQFVEF43QMFhAAiRAyVsYiZBge0OLUMLPTYtTxxPac+Iwa4MUnHsZn7SgSVtORxjQIhvzmVtoAlQsxDOTBoPZXQKTQHRqQgMBSMsJ4YXmClbDAHYYBkXR1l0AwSFsfQSCdAwwBeEgUFsMZdATIVlU5Cl0i+H5SzSDUB0TP0YG2myKQRXwDIHYylWpXU8Bkgc6FoQ16VWMF1jJaNFjEJ7XrwK6tWoQ91PSrSehBtLcp4vCQBQ2FIsQWx9qIqK8x3aAAEJUnSHdzQHLyfNc21-MC4LQuVHLuNmSwwrwgKJhWMBkLwJL2UlaAABF8lLPMMHJf4lsQPaAFoiMAvBEAMMoZD5gWhdLcwwtsCA2YiiWqSZmREssGLJelmQCoHKkZHgXBLmVQyvJJE2zcuayqIpDa4cB00qGtygduKC6-AVaBMMQRAxFhFW1A5Wwnge2TbfNijHfZqUHI8W6VXpdUJXQYsJR0+Xot0GEU-u8wVYJAqPa9-Z5UCdZvwBiyqGtBhoFtAArJZzsOOQFHODOBL2SU8HFvEUGpBurQOXBYSOlBUgABkyFAjAAZgXgBqVf6+8nyjuOfOxGxHoc2uAsixLIkjFnvMAFZhzp3RdDCxKDgfse3OBVBMBwAgiCCsA-kBd+iBQTgihMAGEwsK6lnVJqIm1oHa2QDkHWER98x7BAPfSevRZ7YJFigk+YIz70AAEzYNnqXFysCxoBVpEFdBoUUCWFalKbmcICRyxkDgfyBgICKwTlzHmbD+YyBKCgLkHguE8IJOYXepxsQFUoZaGOlw8GFgIbYUsr9XKxGkScfesx5FWkJlaB4W9LSaInlPIUM956LxIWvDeMDt5ChkXouY6QVGn3UefS+N9oB3wfk-e+YUKGuSOu8XAYYZbimYQIkJ1p37RC-oQYgeY-4AiBKoYBkJoRwkgQSaBmiibwIpIg4OeC0EYNhFgnBHi1GlmIaQ8hhSfKkxoeTWEDC+EsOFoI3OPSRbhMibLIRgtoqKxcXI5pbklGlFzPg4sXiplxB0XvSZpi4iaPdlQX8ZJJ4EiAA
I have a sample code and questions along with that
'use strict';
const AWS = require('aws-sdk');
const promiseFunction = (item) => {
const database = new AWS.DynamoDB();
const params = {
TableName: 'some-table',
Item: item
}
try {
//database.putItem(params).promise(); //3
//Updated as per the comments..
return database.putItem(params).promise(); //3
} catch (err) {
console.log(`Error`);
}
}
const test = async () => {
const arr = [];
try {
for (let data=1;data<=1000;data++) {
const obj = { data: {N: data.toString()}}
arr.push(promiseFunction(obj)); //1
}
const result = await Promise.all(arr); //2
} catch (e) {
console.log(`Error`);
}
}
test();
Some follow-up questions:
At line-2, the result will contain the resolve/reject result once all the promise based function get executed on line //1. Right?
How, the promiseFunction at line-1 is executing and inserting item in dynamodb as I am just pushing it into an array rather than calling the dynamo putItem API. Technically, all 'putItem' should start execute in parallel at line-2. Please help here?
Is the above piece of code is valid in terms of Promise.all and executing insert operations in parallel. Please advice.
At line-2, the result will contain the resolve/reject result once all the promise based function get executed on line //1. Right?
The result will be assigned once all the promises in the arr have been fulfilled. Promise.all doesn't know (or care) how those promises were created.
Technically, all 'putItem' should start execute in parallel at line-2.
No. All the putItem() calls are executed in your loop. Promise.all doesn't execute anything, it only waits for promises created earlier. The request are done concurrently because the code starts them all at once, without waiting in between each loop iteration, not because it uses Promise.all.
Is the above piece of code is valid in terms of Promise.all and executing insert operations in parallel.
No. Your promiseFunction doesn't actually return a promise, since you forgot a return statement on the line marked as //3. Once you add that, the code is fine though.
I am looking at https://www.promisejs.org/patterns/ and it mentions it can be used if you need a value in the form of a promise like:
var value = 10;
var promiseForValue = Promise.resolve(value);
What would be the use of a value in promise form though since it would run synchronously anyway?
If I had:
var value = 10;
var promiseForValue = Promise.resolve(value);
promiseForValue.then(resp => {
myFunction(resp)
})
wouldn't just using value without it being a Promise achieve the same thing:
var value = 10;
myFunction(10);
Say if you write a function that sometimes fetches something from a server, but other times immediately returns, you will probably want that function to always return a promise:
function myThingy() {
if (someCondition) {
return fetch('https://foo');
} else {
return Promise.resolve(true);
}
}
It's also useful if you receive some value that may or may not be a promise. You can wrap it in other promise, and now you are sure it's a promise:
const myValue = someStrangeFunction();
// Guarantee that myValue is a promise
Promise.resolve(myValue).then( ... );
In your examples, yes, there's no point in calling Promise.resolve(value). The use case is when you do want to wrap your already existing value in a Promise, for example to maintain the same API from a function. Let's say I have a function that conditionally does something that would return a promise — the caller of that function shouldn't be the one figuring out what the function returned, the function itself should just make that uniform. For example:
const conditionallyDoAsyncWork = (something) => {
if (something == somethingElse) {
return Promise.resolve(false)
}
return fetch(`/foo/${something}`)
.then((res) => res.json())
}
Then users of this function don't need to check if what they got back was a Promise or not:
const doSomethingWithData = () => {
conditionallyDoAsyncWork(someValue)
.then((result) => result && processData(result))
}
As a side node, using async/await syntax both hides that and makes it a bit easier to read, because any value you return from an async function is automatically wrapped in a Promise:
const conditionallyDoAsyncWork = async (something) => {
if (something == somethingElse) {
return false
}
const res = await fetch(`/foo/${something}`)
return res.json()
}
const doSomethingWithData = async () => {
const result = await conditionallyDoAsyncWork(someValue)
if (result) processData(result)
}
Another use case: dead simple async queue using Promise.resolve() as starting point.
let current = Promise.resolve();
function enqueue(fn) {
current = current.then(fn);
}
enqueue(async () => { console.log("async task") });
Edit, in response to OP's question.
Explanation
Let me break it down for you step by step.
enqueue(task) add the task function as a callback to promise.then, and replace the original current promise reference with the newly returned thenPromise.
current = Promise.resolve()
thenPromise = current.then(task)
current = thenPromise
As per promise spec, if task function in turn returns yet another promise, let's call it task() -> taskPromise, well then the thenPromise will only resolve when taskPromise resolves. thenPromise is practically equivalent to taskPromise, it's just a wrapper. Let's rewrite above code into:
current = Promise.resolve()
taskPromise = current.then(task)
current = taskPromise
So if you go like:
enqueue(task_1)
enqueue(task_2)
enqueue(task_3)
it expands into
current = Promise.resolve()
task_1_promise = current.then(task_1)
task_2_promise = task_1_promise.then(task_2)
task_3_promise = task_2_promise.then(task_3)
current = task_3_promise
effectively forms a linked-list-like struct of promises that'll execute task callbacks in sequential order.
Usage
Let's study a concrete scenario. Imaging you need to handle websocket messages in sequential order.
Let's say you need to do some heavy computation upon receiving messages, so you decide to send it off to a worker thread pool. Then you write the processed result to another message queue (MQ).
But here's the requirement, that MQ is expecting the writing order of messages to match with the order they come in from the websocket stream. What do you do?
Suppose you cannot pause the websocket stream, you can only handle them locally ASAP.
Take One:
websocket.on('message', (msg) => {
sendToWorkerThreadPool(msg).then(result => {
writeToMessageQueue(result)
})
})
This may violate the requirement, cus sendToWorkerThreadPool may not return the result in the original order since it's a pool, some threads may return faster if the workload is light.
Take Two:
websocket.on('message', (msg) => {
const task = () => sendToWorkerThreadPool(msg).then(result => {
writeToMessageQueue(result)
})
enqueue(task)
})
This time we enqueue (defer) the whole process, thus we can ensure the task execution order stays sequential. But there's a drawback, we lost the benefit of using a thread pool, cus each sendToWorkerThreadPool will only fire after last one complete. This model is equivalent to using a single worker thread.
Take Three:
websocket.on('message', (msg) => {
const promise = sendToWorkerThreadPool(msg)
const task = () => promise.then(result => {
writeToMessageQueue(result)
})
enqueue(task)
})
Improvement over take two is, we call sendToWorkerThreadPool ASAP, without deferring, but we still enqueue/defer the writeToMessageQueue part. This way we can make full use of thread pool for computation, but still ensure the sequential writing order to MQ.
I rest my case.
I am looking for ideas/help to improve my code. It's already working, but I am not confident with it and not really proud of it. This is short version of my function -
module.exports.serverlist = async () => {
let promises = [];
const serverlist = [];
serverlist.push({ mon_sid: 'AAA', mon_hostname: 'aaaa.com', mon_port: 80 })
serverlist.push({ mon_sid: 'BBB', mon_hostname: 'bbbb.com', mon_port: 80 })
serverlist.forEach(async (Server) => {
if (Server.mon_sid.includes('_DB')) {
// Function home.checkOracleDatabase return promise, same as above functions
promises.push(home.checkOracleDatabase(Server.mon_hostname, Server.mon_port));
} else if (Server.mon_sid.includes('_HDB')) {
promises.push(home.checkHANADatabase(Server.mon_hostname, Server.mon_port));
} else {
promises.push(home.checkPort(Server.mon_port, Server.mon_hostname, 1000));
}
})
for (let i = 0; i < serverlist.length; i++) {
serverlist[i].status = await promises[i];
}
console.table(serverlist);
What does the code do?:
- It asynchronously performing needed availability check of the service.
What is the expectation?
- That the code will run asynchronously, at the end of function it will wait for all promises to be resolved, for failed results it will perform the check over again, and to have more control over the promises. At this moment the promises are not really connected somehow with the array of systems, it just base on the order but it can be problematic in future.
If someone can assist/give some advises, I would be more than happy.
Also I am not sure how many parallel asynchronous operations NodeJS can perform (or the OS). Currently there are 30 systems on the list, but in future it can be 200-300, I am not sure if it can be handled at once.
Update
I used promise.all - it works fine. However the problem is as I mentioned, I don't have any control over the $promises array. The results as saved in the sequence as they were triggered, and this is how they are being assigned back to serverlist[i].status array from $promises. But I would like to have more control over it, I want to have some index, or something in the $promises array so I can make sure that the results are assigned to PROPER systems (not luckily counting that by the sequence it will be assigned as it should).
Also I would like to extend this function with option to reAttempt failed checks, and for that definitely I need some index in $promises array.
Update 2
After all of your suggestion this is how the code looks for now -
function performChecks(serverlist) {
const Checks = serverlist.map(Server => {
if (Server.mon_sid.includes('_DB')) {
let DB_SID = Server.mon_sid.replace('_DB', '');
return home.checkOracleDatabase(DB_SID, Server.mon_hostname, Server.mon_port)
} else if (Server.mon_sid.includes('_HDB')) {
let DB_SID = Server.mon_sid.replace('_HDB', '');
return home.checkHANADatabase(DB_SID, Server.mon_hostname, Server.mon_port);
} else {
return home.checkPort(Server.mon_port, Server.mon_hostname, 1000)
}
})
return Promise.allSettled(Checks)
}
// Ignore the way that the function is created, it's just for debug purpose
(async function () {
let checkResults = [];
let reAttempt = [];
let reAttemptResults = [];
const serverlist = [];
serverlist.push({ mon_id: 1, mon_sid: 'AAA', mon_hostname: 'hostname_1', mon_port: 3203 })
serverlist.push({ mon_id: 2, mon_sid: 'BBB', mon_hostname: 'hostname_2', mon_port: 3201 })
serverlist.push({ mon_id: 3, mon_sid: 'CCC', mon_hostname: 'hostname_3', mon_port: 3203 })
// Perform first check for all servers
checkResults = await performChecks(serverlist);
// Combine results from check into serverlist array under status key
for(let i = 0; i < serverlist.length; i++) {
serverlist[i]['status'] = checkResults[i].value;
}
// Check for failed results and save them under reAttempt variable
reAttempt = serverlist.filter(Server => Server.status == false);
// Perform checks again for failed results to make sure that it wasn't temporary netowrk/script issue/lag
// Additionally performChecks function will accept one more argument in future which will activate additional trace for reAttempt
reAttemptResults = await performChecks(reAttempt);
// Combine results from reAttempt checks into reAttempt array
for(let i = 0; i < reAttempt.length; i++) {
reAttempt[i]['status'] = reAttemptResults[i].value;
}
// Combine reAttempt array with serverlist array so serverlist can have latest updated data
serverlist.map(x => Object.assign(x, reAttempt.find(y => y.mon_id == x.mon_id)));
// View the results
console.table(serverlist);
})();
Firstly instead of doing a for each and push promises you can map them and do a Promise all. You need no push. Your function can return directly your promise all call. The caller can await it or use then...
Something like this (I didn't test it)
// serverlist declaration
function getList(serverlist) {
const operations = serverlist.map(Server => {
if (Server.mon_sid.includes('_DB')) {
return home.checkOracleDatabase(Server.mon_hostname, Server.mon_port);
} else if (Server.mon_sid.includes('_HDB')) {
return home.checkHANADatabase(Server.mon_hostname, Server.mon_port);
} else {
return home.checkPort(Server.mon_port, Server.mon_hostname, 1000);
}
});
return Promise.all(operations)
}
const serverlist = [...]
const test = await getList(serverlist)
// test is an array of fulfilled/unfulfilled results of Promise.all
So I would create a function that takes a list of operations(serverList) and returns the promise all like the example above without awaiting for it.
The caller would await it or using another promise all of a series of other calls to your function.
Potentially you can go deeper like Inception :)
// on the caller you can go even deeper
await Promise.all([getList([...]) , getList([...]) , getList([...]) ])
Considering what you added you can return something more customized like:
if (Server.mon_sid.includes('_DB')) {
return home.checkOracleDatabase(Server.mon_hostname, Server.mon_port).then(result => ({result, name: 'Oracle', index:..., date:..., hostname: Server.mon_hostname, port: Server.mon_port}))
In the case above your promise would return a json with the output of your operation as result, plus few additional fields you might want to reuse to organize your data.
For more consistency and for resolving this question i/we(other people which can help you) need all code this all variables definition (because for now i can't find where is the promises variable is defined). Thanks
I need to call a function multiple times from different contexts, but i need that each call fires not before that one second has passed after the previous call started.
i'll make an example:
var i = 0;
while(i<50) {
do_something(i)
i++
}
function do_something(a) {
console.log(a)
}
I want that this log:
'1', then after a second '2', then after a second '3', then after a second '4'...
I can't use simple setInterval or setTimeout because this function 'do_something(param)' can be called in the same moment from different sources cause i am working with async function in nodejs.
I want that the order of calls is kept, but that they fires with minimum delay of one second.
I think i should add these calls to a queue, and then each second a call is dequeued and the function fires, but i really don't know how to do it in nodejs. Thank you in advance
i had to do something like this:
var tasks = [] //global var
var processor = setInterval(function() {
process_task()}, 1000)
function add_task() {
tasks.push('my task') //add task to the end of queue
}
process_task() {
var task_to_use = tasks[0];
tasks.shift() //remove first task in the queue (tasks[0])
//do what i need to with the task 'task_to_use'
}
in this way i can add tasks to the queue from wherever i want (tasks is a variable of the global context) just calling tasks.push('mytask') and the tasks will be processed one each second following the order they were put in the queue.
However, i didn't really need to do it. I needed because i am using Twilio's apis, and in their doc i read each phone number can send up to an sms for second and no more, but then the support told me they queue requests and send one message each second, so that sending more than a request for second is really not a problem and no sms sending will fail. Hope this will help, byee
Coming late to a party
I know I am late, but I had this exact same problem with this exact same technologies.
Your post was very helpful, but it lacked good practices and used Global variables.
My solution
If you are reading this today, I want you to know that after a week of bashing my head I ended up creating a question that lead to two different answers, both capable of helping you:
How to delay execution of functions, JavaScript
The queue approach, pioneered by #Arg0n and revamped by me is the closest one to your example, but with none of you drawbacks:
let asyncFunc = function(url) {
return new Promise((resolve, reject) => {
setTimeout(function() {
resolve({
url: url,
data: "banana"
});
}, 5000);
});
};
let delayFactory = function(args) {
let {
delayMs
} = args;
let queuedCalls = [];
let executing = false;
let queueCall = function(url) {
return new Promise((resolve, reject) => {
queuedCalls.push({
url,
resolve,
reject
});
if (executing === false) {
executing = true;
nextCall();
}
});
};
let execute = function(call) {
console.log(`sending request ${call.url}`);
asyncFunc(call.url)
.then(call.resolve)
.catch(call.reject);
setTimeout(nextCall, delayMs);
};
let nextCall = function() {
if (queuedCalls.length > 0)
execute(queuedCalls.shift());
else
executing = false;
};
return Object.freeze({
queueCall
});
};
let myFactory = delayFactory({
delayMs: 1000
});
myFactory.queueCall("http://test1")
.then(console.log)
.catch(console.log);
myFactory.queueCall("http://test2")
.then(console.log)
.catch(console.log);
myFactory.queueCall("http://test3")
.then(console.log)
.catch(console.log);
Give it a try and have fun!