Edit: Why this is not a duplicate: because Cypress, just read instead of tagging everything as duplicate.
Edit 2: Also, see answer for better understanding of the differences between usual async for loops problems and this question.
I am writing cypress tests and I want to create a cypress command that populates my database with a list of users. I want the creation loop to wait for each user to be created before it moves on to the next one (because I want that done in a specific order).
For now, my loop looks like this:
Cypress.Commands.add("populateDb", (users) => {
var createdItems = []
for (const user of users) {
cy.createUser(user, 'passe').then(response => {
createdUsers.unshift(response.body.user)
})
}
return createdItems
})
Of course, this loop does not wait for each user to be created before moving onto the next one (I want 'sequential treatment', NOT 'parallel and then wait for all promise to resolve')
I have read the answers about async for-loop here:
JavaScript ES6 promise for loop
Using async/await with a forEach loop
How do I return the response from an asynchronous call?
But I can't seem to find what I want, mainly because cypress wont allow me to declare my function as async as follow :
Cypress.Commands.add("populateDb", async (users) => {
//Some code
})
And If I don't declare it async I am not able to use await.
Isn't there some king of get() method that just synchronously wait for a Promise to resolve?
Using a combination of wrap and each cypress command I was able to achieve a loop that waits for each iteration and can return the full results without needing a global variable.
The only reason I use the wrap command is because the cypress command each requires it to be chained off a previous cypress command. The each command will evaluate each iteration and finally call the then where you can return the complete results array. I am using this to upload multiple files and return the list of keys, but you can modify this for your own need.
Cypress.Commands.add("uploadMultipleFiles", (files) => {
var uploadedS3Keys = []
cy.wrap(files).each((file, index, list) => {
cy.uploadFileToOrdersApi(file)
.then(s3Key => uploadedS3Keys.push(s3Key))
}).then(() => uploadedS3Keys)
})
Turns out there is a reason why cypress is so restrictive about what you can do about waiting for async method to resolve: it always automatically runs all the async commands in sequential order, as they are called, not in parallel, so this will execute in the right order even if createUser is Async :
Cypress.Commands.add("populateDb", (users) => {
for (const user in users) {
cy.createUser(user).then(response => {
console.log(response)
})
}
})
If you want to get the value of what is returned (in my case, I need the user ID to delete them later), you can just store them in a var at file root level and add a cypress command that returns that var.
var createdItems = []
Cypress.Commands.add("getCreatedItems", () => {
return createdItems
})
Cypress.Commands.add("populateDb", (users) => {
for (const user in users) {
cy.createUser(user).then(response => {
createdItems.unshift(response) // I unshift() because I need to delete them in FILO order later
})
}
})
Then in the cypress test file you can just call them in the order you need them to execute :
cy.populateDb(users)
cy.getCreatedItems().then(response => {
//response is my createdItems Array, and cy.getCreatedItems() will run only when cy.populateDb() is resolved
})
You could do it also like:
Cypress.Commands.add("populateDb", (users) => {
for (const user in users) {
cy.createUser(user).then(response => {
createdItems.unshift(response) // I unshift() because I need to delete them in FILO order later
})
}
return createdItems;
})
cy.populateDb(users).then(response => {
// ... runs after populate ...
})
But the other answer is also right. Each cy.xxxxx command is actually added to command queue and ran one after each other. If during queue execution new cy.xxxxx2 commands are called, they will be added to front of the queue.
Here is simple example to explain the execution order of cypress command queue and synchronous code:
const a = 1; // a == 1
cy.cmd1().then(() => {
cy.cmd2().then(() => {
a += 1; // a == 5
});
a += 1; // a == 3
cy.cmd3()
a += 1; // a == 4
});
cy.cmd4().then(() => {
a += 1; // a == 6
});
a += 1; // a == 2
// at this point cypress command queue starts running the queue that is cmd1, cmd4
// 1. cmd1 runs and adds cmd2 and cmd3 to front of command queue also adds +2 to a
// 2. cmd2 runs and adds +1 to a
// 3. cmd3 runs
// 4. cmd4 runs and adds +1 to a
// ... all done ...
So from this example you can see that in your case you loop will be executed serially, because each cy.createUser is added to cypress command queue and executed then sequentially.
Related
I am making a chrome extension (mv3). Based on user activity, the content.js passes a message to the background.js which then calls an async function to add data in Google Docs using Docs API.
I want each request to execute only after the previous one has finished running. I am using chrome.runtime.sendMessage to send a message from content.js and don't see a way of calling background.js serially from there. So I need a way of executing them one by one in background.js only. The order of these requests is also important (but if the order of the requests gets changed by one/two places, I think that would still be okay from a user perspective).
I tried something and it is working but I am not sure if I am missing some edge cases, because I was unable to find the approach in any other answers -
Semaphore-like queue in javascript?
Run n number of async function before calling another method in nodejs
JavaScript: execute async function one by one
The approach I used is: I use a stack like structure to store requests, use setInterval to check for any pending requests and execute them serially.
content.js:
chrome.runtime.sendMessage({message});
background.js:
let addToDocInterval = "";
let addToDocCalls = [];
async function addToDoc(msg) {
// Await calls to doc API
}
async function addToDocHelper() {
if(addToDocCalls.length === 0)
return;
clearInterval(addToDocInterval)
while(addToDocCalls.length > 0) {
let msg = addToDocCalls.shift();
await addToDoc(msg);
}
addToDocInterval = setInterval(addToDocHelper, 1000);
}
chrome.runtime.onMessage.addListener((msg) => {
// Some other logic
addToDocCalls.push(msg);
})
addToDocInterval = setInterval(addToDocHelper, 1000);
Is this approach correct? Or is there any better way to do this?
I'd suggest changing several things.
Don't use timers polling the array. Just initiate processing the array anytime you add a new item to the array.
Keep a flag on whether if you're already processing the array so you don't start duplicate processing.
Use a class to encapsulate this functionality into an object.
Encapsulate the addToDocCalls array and adding to it so your class is managing it and outside code just calls a function to add to it which also triggers the processing. Basically, you're making it so callers don't have to know how the insides work. They just call helper.addMsg(msg) and the class instance does all the work.
Here's an implementation:
async function addToDoc(msg) {
// Await calls to doc API
}
class docHelper {
constructor() {
this.addToDocCalls = [];
this.loopRunning = false;
}
addMsg(msg) {
// add item to the queue and initiate processing of the queue
this.addToDocCalls.push(msg);
this.process();
}
async process() {
// don't run this loop twice if we're already running it
if (this.loopRunning) return;
try {
this.loopRunning = true;
// process all items in the addToDocCalls we have
while(this.addToDocCalls.length > 0) {
let msg = addToDocCalls.shift();
await addToDoc(msg);
}
} finally {
this.loopRunning = false;
}
}
}
const helper = new docHelper();
chrome.runtime.onMessage.addListener((msg) => {
// Some other logic
helper.addMsg(msg);
});
So, process() will run until the array is empty. Any interim calls to addMsg while process() is running will add more items to array and will call process() again, but the loopRunning flag will keep it from starting duplicate processing loops. If addMsg() is called while process is not running, it will start the process loop.
P.S. You also need to figure out what sort of error handling you want if addToDoc(msg) rejects. This code protects the this.loopRunning flag if it rejects, but doesn't actually handle a reject error. In code like this that is processing a queue, often times all you can really do is log the error and move on, but you need to decide what is the proper course of action on a rejection.
You don't need to use setTimeout. You do not even need a while loop.
let addToDocInterval = "";
let addToDocCalls = [];
let running = false;
async function addToDoc(msg) {
// Await calls to doc API
}
async function addToDocHelper() {
if(running || addToDocCalls.length === 0)
return;
running = true;
let msg = addToDocCalls.shift();
await addToDoc(msg);
running = false;
addToDocHelper();
}
chrome.runtime.onMessage.addListener((msg) => {
// Some other logic
addToDocCalls.push(msg);
addToDocHelper();
});
The code should be self explanatory. There is no magic.
Here is a generic way to run async tasks sequentially (and add more tasks to the queue at any time).
const tasks = [];
let taskInProgress = false;
async function qTask(newTask) {
if (newTask) tasks.push(newTask);
if (tasks.length === 0) return;
if (taskInProgress) return;
const nextTask = tasks.shift();
taskInProgress = true;
try {
await nextTask();
} finally {
taskInProgress = false;
//use setTimeout so call stack can't overflow
setTimeout(qTask, 0);
}
}
//the code below is just used to demonstrate the code above works
async function test() {
console.log(`queuing first task`);
qTask(async () => {
await delay(500); //pretend this task takes 0.5 seconds
console.log('first task started');
throw 'demonstrate error does not ruin task queue';
console.log('first task finished');
});
for (let i = 0; i < 5; i++) {
console.log(`queuing task ${i}`)
qTask(async () => {
await delay(200); //pretend this task takes 0.2 seconds
console.log(`task ${i} ran`);
});
}
await delay(1000); //wait 1 second
console.log(`queuing extra task`);
qTask(async () => {
console.log('extra task ran');
});
await delay(3000); //wait 3 seconds
console.log(`queuing last task`);
qTask(async () => {
console.log('last task ran');
});
}
test();
function delay(ms) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
As the questions says I'm trying to wait for a map to finish executing before I run another command. In this case console log. I have
let concernsInDB = [];
let ctr = 0;
this.state.templateOptions.map(el => {
el.details.map(elDetail => {
this.getConcernsDB(elDetail.values)
.then(elDetailResults => {
let detailsToPush = {};
if(elDetailResults.length) {
detailsToPush[elDetailResults[0].value_id] = elDetailResults;
concernsInDB.push(detailsToPush);
}
})
})
})
console.log(concernsInDB);
The above code executes and I get a console.log of Array []. The console.log is resolving before all other methods are finished.
this.state.templateOptions contains an array of several objects (about 50, so I'm not putting all here):
Object {
"active": 1,
"id": 1378,
"name": "Wires are running through trees",
"narrative": "Service drop wires are running through tree(s).It is recommended to have a contacting utility company or a qualified electrician and/or tree s",vice company to correct conditions as necessary.
"value_id": 13935,
},
As I loop through them I run this method: this.getConcernsDB(elDetail.values)
elDetail.values is the "value_id" from the above example. I then wait for it to resolve .then push the results into concernsInDB. This has to be done on several records. How would I wait for it all to finish before console.logging concernsInDB so it'll have all the completed data?
If I wrap it in a promise the resolve would run on each loop so I don't see how I could accomplish it. Any solutions?
You want to use Promise.all (twice) to wait for all the Promise based asynchrony to finish
You also want to return something in .map otherwise you'd use .forEach (.map is the correct method in this case though)
let concernsInDB = [];
let ctr = 0;
Promise.all(this.state.templateOptions.map(el =>
Promise.all(el.details.map(elDetail =>
this.getConcernsDB(elDetail.values)
.then(elDetailResults => {
let detailsToPush = {};
if (elDetailResults.length) {
detailsToPush[elDetailResults[0].value_id] = elDetailResults;
concernsInDB.push(detailsToPush);
}
})
))
)).then(() => {
console.log(concernsInDB);
});
sup guys, i'm working on this firebase project and i need to iterate trought a subcollection of all sales of all stores in the root collection and sum their values... the problem that i'm getting is that i'm getting the sum printed before the iteration. I'm new to TS and Firebase... this is what i got so far:
export const newBilling = functions.firestore.document('billings/{billId}').onCreate(event =>
{
const valueArray = []
const feeArray = []
const storesCollection = afs.collection('stores').where('active', '==', true).get().then(stores => {
stores.forEach(store => {
const salesCollection = afs.collection('stores').doc(store.id).collection('sales').get().then(sales => {
sales.forEach(sale => {
return valueArray.push(sale.data().value) + feeArray.push(sale.data().fee)
// other aproach
// valueArray.push(sale.data().value)
// feeArray.push(sale.data().fee)
})
})
})
}).catch(error => {console.log(error)})
let cashbackSum, feeSum : number
cashbackArray.forEach(value => {
cashbackSum += value
})
feeArray.forEach(value => {
feeSum += value
})
console.log(cashbackSum, feeSum)
return 0
})
TKS =)
You're not using promises correctly. You've got a lot of get() method call, each of which are asynchronous and return a promise, but you're never using them to case the entire function to wait for all the work to complete. Calling then() doesn't actually make your code wait - it just runs the next bit of code and returns another promise. Your final console.log is executing first because none of the work you kicked off ahead of it is complete yet.
Your code actually needs to be substantially different in order to work correctly, and you need to return a promise from the entire function that resolves only after all the work is complete.
You can learn better how to use promises in Cloud Functions by going through the video tutorials.
I am working on a project where I am building a simple front end in Angular (typescript) / Node to make call to a back end server for executing different tasks. These tasks take time to execute and thus need to be queued on the back end server. I solved this issue by following the following tutorial: https://github.com/realpython/flask-by-example and everything seems to work just fine.
Now I am finishing things up on the front end, where most of the code has been already written in Typescript using Angular and Rxjs. I am trying to replicate the following code in Typescript:
https://github.com/dimoreira/word-frequency/blob/master/static/main.js
This code consists of two functions, where first function "getModelSummary"(getResults in the example) calls a post method via:
public getModelSummary(modelSummaryParameters: ModelSummaryParameters): Observable<ModelSummary> {
return this.http.post(`${SERVER_URL}start`, modelSummaryParameters)
.map(res => res.json())
;
}
to put the job in queue and assign a jobID to that function on the back end server. The second function "listenModelSummary", ideally should get executed right after the first function with the jobId as it's input and loops in a short interval checking if the job has been completed or not:
public listenModelSummary(jobID: string) {
return this.http.get(`${SERVER_URL}results/` + jobID).map(
(res) => res.json()
);
}
Once the job is done, it needs to return the results, which would update the front end.
I am new to Typescript, Observables and rxjs and wanted to ask for the right way of doing this. I do not want to use javascript, but want to stick to Typescript as much as possible in my front end code stack. How can I use the first function to call the second function with it's output "jobID" and have the second function run via interval until the output comes back?
Observables are great, and are the type of object returned by Angular's HttpClient class, but sometimes, in my opinion, dealing with them is a lot more complicated than using promises.
Yes, there is a slight performance hit for the extra operation to convert the Observable to a Promise, but you get a simpler programming model.
If you need to wait for the first function to complete, and then hand the returned value to another function, you can do:
async getModelSummary(modelSummaryParameters: ModelSummaryParameters): Promise<ModelSummary> {
return this.http.post(`${SERVER_URL}start`, modelSummaryParameters).toPromise();
}
async doStuff(): Promise<void> {
const modelSummary = await this.getModelSummary(params);
// not sure if you need to assign this to your viewmodel,
// what's returned, etc
this.listenModelSummary(modelSummary)
}
If you're dead-set on using Observables, I would suggest using the concatMap pattern, which would go something like this:
doStuff(modelSummaryParameters: ModelSummaryParameters): Observable<ModelSummary> {
return this.http
.post(`${SERVER_URL}start`, modelSummaryParameters)
.pipe(
concatMap(modelSummary => <Observable<ModelSummary>> this.listenModelSummary(modelSummary))
);
}
Here's an article on different mapping solutions for Observables: https://blog.angularindepth.com/practical-rxjs-in-the-wild-requests-with-concatmap-vs-mergemap-vs-forkjoin-11e5b2efe293 that might help you out.
You can try the following:
getModelSummary(modelSummaryParameters: ModelSummaryParameters): Promise<ModelSummary> {
return this.http.post(`${SERVER_URL}start`, modelSummaryParameters).toPromise();
}
async someMethodInYourComponent() {
const modelSummary = await this.get(modelSummary(params);
this.listenModelSummary(modelSummary)
}
// OR
someMethodInYourComponent() {
this.get(modelSummary(params).then(() => {
this.listenModelSummary(modelSummary);
});
}
After doing more reading/researching into rxjs, I was able to make my code work and I wanted to thank you guys for the feedback and to post my code below.
In my services I created two observables:
First one is to fetch a jobId returned by queue server:
// API: GET / FUNCTION /:jobID
public getModelSummaryQueueId(modelSummaryParameters: ModelSummaryParameters): Observable<JobId>{
return this.http.post(${SERVER_URL}start, modelSummaryParameters).map(
(jobId) => jobId.json()
)
}
Use the jobId from first segment to fetch data:
// API: GET / FUNCTION /:results
public listenModelSummary(jobId: JobId): Observable <ModelSummary>{
return this.http.get(${SERVER_URL}results/+ jobId).map(
(res) => res.json()
)
}
Below is the component that works with the 2 services above:
`
this.subscription = this.developmentService.getModelSummaryQueueId(this.modelSummaryParameters)
.subscribe((jobId) => {
return this.developmentService.listenModelSummary(jobId)
// use switchMap to pull value from observable and check if it completes
.switchMap((modelSummary) =>
// if value has not changed then invoke observable again else return
modelSummary.toString() === 'Nay!'
? Observable.throw(console.log('...Processing Request...'))
// ? Observable.throw(this.modelSummary = modelSummary)
: Observable.of(modelSummary)
)
.retryWhen((attempts) => {
return Observable
// specify number of attempts
.range(1,20)
.zip(attempts, function(i) {
return(i);
})
.flatMap((res:any) => {
// res is a counter of how many attempts
console.log("number of attempts: ", res);
res = 'heartbeat - ' + res
this.getProgressBar(res);
// this.res = res;
// delay request
return Observable.of(res).delay(100)
})
})
// .subscribe(this.displayData);
// .subscribe(modelSummary => console.log(modelSummary));
.subscribe((modelSummary) => {
console.log("FINAL RESULT: ", modelSummary)
this.modelSummary = modelSummary;
this.getProgressBar('Done');
});
});
`
I need to control concurrency in a Node.js script I'm making. Currently I'm trying to use npm promise-task-queue but I'm open to other suggestions.
I'm not sure how to implement promise-task-queue into my code. This is my original program:
readURLsfromFile().then( (urls) => {
urls.reduce( (accumulator, current, i) => {
return accumulator.then( () => {
return main(urls[i], i, urls.length)
})
}, Promise.resolve())
})
As you can see I'm reading urls from a file, then using .reduce() to run main() in serial on each one of these urls. Serial was too slow though so I need to do it with controlled concurrency.
Here's the code I started to write using promise-task-queue (It's very wrong, I have no idea what I'm doing):
var taskQueue = require("promise-task-queue");
var queue = taskQueue();
var failedRequests = 0;
queue.on("failed:apiRequest", function(task) {
failedRequests += 1;
});
queue.define("apiRequest", function(task) {
return Promise.try( () => {
return main(urls[i], i, urls.length));
}).then( () => {
return console.log("DONE!");
});
}, {
concurrency: 2
});
Promise.try( () => {
/* The following queues up the actual task. Note how it returns a Promise! */
return queue.push("apiRequest", {url: urls[i], iteration: i, amountToDo: urls.length)});
})
As you can see I've put my main() function with its argument after the Promise.try, and I've put my arguments after the return queue.push. Not sure if that's correct or not.
But regardless now I'm stuck, how do I load all the iterations into the queue?
You could use the qew module from npm: https://www.npmjs.com/package/qew.
Install using npm install qew.
To initialise you do
const Qew = require('qew');
const maxConcurrent = 3;
const qew = new Qew(maxConcurrent);
Using the above code qew will now be a queue that allows you to push asynchronous functions onto that will execute with a maximum concurrency of 3.
To push a new async function onto the qew you can do
qew.pushProm(asyncFunc);
So in your case if I understood you correctly you could do something like
readURLsfromFile()
.then(urls => {
return Promise.all(urls.map(url => { // wait for all promises to resolve
return qew.pushProm(() => main(url)); // push function onto queue
}));
})
.then(results => {
// do stuff with results
})
In this snippet you are reading urls from a file, and then loading a bunch of functions into the qew one by one and waiting for them all to resolve before doing something with them.
Full disclaimer: I am the author of this package.