I have a sample code and questions along with that
'use strict';
const AWS = require('aws-sdk');
const promiseFunction = (item) => {
const database = new AWS.DynamoDB();
const params = {
TableName: 'some-table',
Item: item
}
try {
//database.putItem(params).promise(); //3
//Updated as per the comments..
return database.putItem(params).promise(); //3
} catch (err) {
console.log(`Error`);
}
}
const test = async () => {
const arr = [];
try {
for (let data=1;data<=1000;data++) {
const obj = { data: {N: data.toString()}}
arr.push(promiseFunction(obj)); //1
}
const result = await Promise.all(arr); //2
} catch (e) {
console.log(`Error`);
}
}
test();
Some follow-up questions:
At line-2, the result will contain the resolve/reject result once all the promise based function get executed on line //1. Right?
How, the promiseFunction at line-1 is executing and inserting item in dynamodb as I am just pushing it into an array rather than calling the dynamo putItem API. Technically, all 'putItem' should start execute in parallel at line-2. Please help here?
Is the above piece of code is valid in terms of Promise.all and executing insert operations in parallel. Please advice.
At line-2, the result will contain the resolve/reject result once all the promise based function get executed on line //1. Right?
The result will be assigned once all the promises in the arr have been fulfilled. Promise.all doesn't know (or care) how those promises were created.
Technically, all 'putItem' should start execute in parallel at line-2.
No. All the putItem() calls are executed in your loop. Promise.all doesn't execute anything, it only waits for promises created earlier. The request are done concurrently because the code starts them all at once, without waiting in between each loop iteration, not because it uses Promise.all.
Is the above piece of code is valid in terms of Promise.all and executing insert operations in parallel.
No. Your promiseFunction doesn't actually return a promise, since you forgot a return statement on the line marked as //3. Once you add that, the code is fine though.
Related
I want to use Eslint plugin in my project with Webpack but it does not let me use await inside the loop.
According to Eslint docs it recommends to remove await from the loop and just add a Promise after.
Wrong example:
async function foo(things) {
const results = [];
for (const thing of things) {
// Bad: each loop iteration is delayed until the entire asynchronous operation completes
results.push(await bar(thing));
}
return baz(results);
}
Correct example:
async function foo(things) {
const results = [];
for (const thing of things) {
// Good: all asynchronous operations are immediately started.
results.push(bar(thing));
}
// Now that all the asynchronous operations are running, here we wait until they all complete.
return baz(await Promise.all(results));
}
But in my code I just merge data into one array which comes from HTTP request:
async update() {
let array = [];
for (url of this.myUrls) {
const response = await this.getData(url);
array = await array.concat(response);
}
}
Is it possible to remove await from this loop and add Promise just for array concatenation? I don't have an idea how to do it...
If you like one-liners.
const array = await Promise.all(this.myUrls.map((url)=> this.getData(url)));
In this case, the map method returns a bunch of promises, based on the URL, and your getData method. The Promise.all waits until all of your promises will be resolved. These promises run parallel.
You can use promise like this:
function update() {
let array = [],req=[];
for (url of this.myUrls) {
req.push(this.getData(url));
}
return Promise.all(req).then((data)=>{
console.log(data);
return data;
})
}
If I'm understanding you correctly, your getData function returns an array?
Using the one-liner supplied by Anarno, we can wait until all promises are resolved and then concatenate all the arrays.
const allResults = await Promise.all(this.myUrls.map((url) => this.getData(url)));
let finalArray = [];
allResults.forEach((item) => finalArray = finalArray.concat(item));
I am looking at https://www.promisejs.org/patterns/ and it mentions it can be used if you need a value in the form of a promise like:
var value = 10;
var promiseForValue = Promise.resolve(value);
What would be the use of a value in promise form though since it would run synchronously anyway?
If I had:
var value = 10;
var promiseForValue = Promise.resolve(value);
promiseForValue.then(resp => {
myFunction(resp)
})
wouldn't just using value without it being a Promise achieve the same thing:
var value = 10;
myFunction(10);
Say if you write a function that sometimes fetches something from a server, but other times immediately returns, you will probably want that function to always return a promise:
function myThingy() {
if (someCondition) {
return fetch('https://foo');
} else {
return Promise.resolve(true);
}
}
It's also useful if you receive some value that may or may not be a promise. You can wrap it in other promise, and now you are sure it's a promise:
const myValue = someStrangeFunction();
// Guarantee that myValue is a promise
Promise.resolve(myValue).then( ... );
In your examples, yes, there's no point in calling Promise.resolve(value). The use case is when you do want to wrap your already existing value in a Promise, for example to maintain the same API from a function. Let's say I have a function that conditionally does something that would return a promise — the caller of that function shouldn't be the one figuring out what the function returned, the function itself should just make that uniform. For example:
const conditionallyDoAsyncWork = (something) => {
if (something == somethingElse) {
return Promise.resolve(false)
}
return fetch(`/foo/${something}`)
.then((res) => res.json())
}
Then users of this function don't need to check if what they got back was a Promise or not:
const doSomethingWithData = () => {
conditionallyDoAsyncWork(someValue)
.then((result) => result && processData(result))
}
As a side node, using async/await syntax both hides that and makes it a bit easier to read, because any value you return from an async function is automatically wrapped in a Promise:
const conditionallyDoAsyncWork = async (something) => {
if (something == somethingElse) {
return false
}
const res = await fetch(`/foo/${something}`)
return res.json()
}
const doSomethingWithData = async () => {
const result = await conditionallyDoAsyncWork(someValue)
if (result) processData(result)
}
Another use case: dead simple async queue using Promise.resolve() as starting point.
let current = Promise.resolve();
function enqueue(fn) {
current = current.then(fn);
}
enqueue(async () => { console.log("async task") });
Edit, in response to OP's question.
Explanation
Let me break it down for you step by step.
enqueue(task) add the task function as a callback to promise.then, and replace the original current promise reference with the newly returned thenPromise.
current = Promise.resolve()
thenPromise = current.then(task)
current = thenPromise
As per promise spec, if task function in turn returns yet another promise, let's call it task() -> taskPromise, well then the thenPromise will only resolve when taskPromise resolves. thenPromise is practically equivalent to taskPromise, it's just a wrapper. Let's rewrite above code into:
current = Promise.resolve()
taskPromise = current.then(task)
current = taskPromise
So if you go like:
enqueue(task_1)
enqueue(task_2)
enqueue(task_3)
it expands into
current = Promise.resolve()
task_1_promise = current.then(task_1)
task_2_promise = task_1_promise.then(task_2)
task_3_promise = task_2_promise.then(task_3)
current = task_3_promise
effectively forms a linked-list-like struct of promises that'll execute task callbacks in sequential order.
Usage
Let's study a concrete scenario. Imaging you need to handle websocket messages in sequential order.
Let's say you need to do some heavy computation upon receiving messages, so you decide to send it off to a worker thread pool. Then you write the processed result to another message queue (MQ).
But here's the requirement, that MQ is expecting the writing order of messages to match with the order they come in from the websocket stream. What do you do?
Suppose you cannot pause the websocket stream, you can only handle them locally ASAP.
Take One:
websocket.on('message', (msg) => {
sendToWorkerThreadPool(msg).then(result => {
writeToMessageQueue(result)
})
})
This may violate the requirement, cus sendToWorkerThreadPool may not return the result in the original order since it's a pool, some threads may return faster if the workload is light.
Take Two:
websocket.on('message', (msg) => {
const task = () => sendToWorkerThreadPool(msg).then(result => {
writeToMessageQueue(result)
})
enqueue(task)
})
This time we enqueue (defer) the whole process, thus we can ensure the task execution order stays sequential. But there's a drawback, we lost the benefit of using a thread pool, cus each sendToWorkerThreadPool will only fire after last one complete. This model is equivalent to using a single worker thread.
Take Three:
websocket.on('message', (msg) => {
const promise = sendToWorkerThreadPool(msg)
const task = () => promise.then(result => {
writeToMessageQueue(result)
})
enqueue(task)
})
Improvement over take two is, we call sendToWorkerThreadPool ASAP, without deferring, but we still enqueue/defer the writeToMessageQueue part. This way we can make full use of thread pool for computation, but still ensure the sequential writing order to MQ.
I rest my case.
I'm new in JavaScript and Node.js. I have the following code:
const populateSetup = async () => {
token = await getToken();
const promises = await constant.accounts.map(async (x) => {
const accountId = await createAccountRequest(x.account);
const peoid = await createPeopleRequests(x.people);
const pid = await createProjectsRequests(x.project);
return [accountId, pid, peoid];
});
const [accountId, pid, peoid] = await Promise.all(promises);
};
In the above, token is first fetched and is required to create account and then accountId returned is required to create people and projects. Let's say I have the following input:
exports.accounts = [
{ account: this.testAccountFirst, project: this.projectOne, people: this.testUserOne },
{ account: this.testAccountSecond, project: this.projectTwo, people: this.testUserTwo },
];
After running the populateSetup() in node environment my result is (not the console output but the output of the populateSetup():
testAccountFirst has 1 people -> testUserOne
testAccountSecond has 2 projects and 1 user -> projectOne, projectTwo, testUserTwo
expected result is:
testAccountFirst should have 1 project and 1 people -> projectOne, testUserOne
testAccountSecond should have 1 project and 1 people -> projectTwo, testUserTwo
The problem here is that the accountId of first account is not sent to the projectsRequest. I don't know how to resolve this. I have gone through this Stackoverflow question but still couldn't figure out.
I'm having a hard time understanding exactly what question you're asking, but .map() is not async savvy. That means that even though you declare the callback as async, .map() doesn't do anything with the promise that returns and thus it doesn't wait to start the 2nd iteration before the 1st iteration is done. Therefore, you end up running all the async operations from all the iterations of the loop in parallel and they can finish in any random order.
If you really want to run them sequentially, one after another, then switch your .map() to a for loop because a for loop will wait for an await in the first iteration of the loop before starting the 2nd iteration of the loop and so on...
return [accountId, pid, peoid]; is returning resolved promises, also, you wait one promise resolving after another. For example, if one resolving is 5 sec, then you need to wait 5+5+5=15 secs
But more of that, it is bad practice to use .map() with promises inside, because it is sync operator.
In your case, I would use something like that:
const populateSetup = async () => {
const token = await getToken();
const [accountId, peoid, pid] =[
createAccountRequest(x.account),
createPeopleRequests(x.people),
createProjectsRequests(x.project)
]
return Promise.all([accountId, peoid,pid])
};
Here, You return promise, that can be used like this:
const [accountId, peoid,pid] = await populateSetup()
Promise.all() do promise execution simultaniosly and wait for all to be resolved, so it is 5 secs instead of 15
It should have given me not null output but I am getting null output. It is working fine if I console inside the block of code but outside it gives me null value
Here is the code:
app.post("/downloadDb",async (req,res)=>{
var docData = [];
var idList = [];
console.log("downloadBd");
const mahafuzCol = firestore.collection("Mahafuz")
await mahafuzCol.listDocuments()
.then( listDoc=>{
//List of id fetch
listDoc.forEach(data=>{
idList.push(data.id)
});
}).catch(e=>console.log(e));
//document is fetched
await idList.forEach(id=>{
mahafuzCol.doc(id).get().then(
doc=>{
docData.push(doc.data());
//Here I get desire output w=if I log with console
}
);
});
//Here I get null output
await console.log(docData);
});
Ok, looking at your piece of code, I would like to point out a few things.
You are using the latest and greatest ES7 async and await feature which is great. Why are you stuck with the old way of defining variables? Try to use let and const instead of var. Don't mix the ECMAScript versions like this. This is considered a bad practice.
Loops in Node.js are synchronous as of now (though asynchronous loops are in the pipeline of Node and we would see them soon). You cannot put asynchronous code inside a loop and expect them to work as expected. There is a whole concept of event loop in Node.js and how Node handles asynchronous tasks. So, if you have time, you should definitely go through the event loop concepts.
Here is the better way to write the code:
app.post('/downloadDb', async (req, res) => {
// always wrap asynchronous code in async/await in try/catch blocks
console.log('downloadBd');
const mahafuzCol = firestore.collection('Mahafuz');
try {
// assuming that listDocuments returns a promise
// await on it until it gets resolved
// all the listed documents will be assigned to docs
// once the promise is resolved
const docs = await mahafuzCol.listDocuments();
const idList = docs.map(data => data.id);
// again assuming that get() returns a promise
// pushing all the promises to an array so that
// we can use Promise.all to resolve all the promises
// at once
const promisesList = idList.map(id => mahafuzCol.doc(id).get());
// fetching document is here
// once all the promises are resolved, data contains
// the result of all the promises as an array
const data = await Promise.all(promisesList);
const docData = data.map(doc => doc.data());
console.log(docData);
// return some response
return res.status(200).send();
} catch (error) {
console.log('error: ', error);
// return some response
return res.status(500).send();
}
});
PS:
If you still somehow want to use asynchronous loops, have a look at this library
https://caolan.github.io/async/docs.html#each
Since forEach is async in nature, as soon as you write a promise in a forEach() loop. The next command gets executed, which in your case is:
// Here I get null output
await console.log(docData);
You'd need the typical for loop for this task:
Try the snippet below:
let idListLen = idList.length;
for (let i =0; i< idListLen; i++) {
mahafuzCol.doc(id).get().then(
doc=>{
docData.push(doc.data());
//Here I get desire output w=if I log with console
}
);
}
console.log(docData) //prints the list of data
I need to perform some async tasks in Nodejs. In this case, I need to iterate throw al levels of a JSON. For that reason, I need to "iterate" syncronusly that object but in order.
I'm doing tests with this code which is a simple example adapted from this site
var fnlist = [ doFirstThing, doSecondThing, doThirdThing, lastThing];
// Promise returning functions to execute
function doFirstThing(){ return Promise.resolve(1); }
function doSecondThing(res){ return Promise.resolve(res + 1); }
function doThirdThing(res){ return Promise.resolve(res + 2); }
function lastThing(res){ console.log("result:", res); }
// Execute a list of Promise return functions in series
function pseries(req,json,list) {
var p = Promise.resolve();
return doFirstThing()
.then((value) => {
console.log('value');
console.log(value);
return doSecondThing(value).then((value2) => {
console.log('value2');
console.log(value2);
});
});
}
router.get('/', function(req, res, next) {
var thisArray = json[0].array;
for(var i = 0;i < thisArray.length; i++){
pseries(req,json,fnlist);
}
});
Console output is:
1
value
1
value
1
value2
2
value2
2
value2
2
And is not still valid because I would need to have this kind of flow:
value
1
value2
2
value
1
value2
2
value
1
value2
2
I know I need to use promises factories in order to don't execute them as soon as they are created, but seems to not be working now. I know I can't use .all because I need to use some data from one promise in the next one.
Any ideas? Thanks!
You have started multiple independent promise chains in your for loop (each call to pseries() is a separate promise chain). As such, you cannot control the sequencing of the separate promise chains. If you want to control one chain vs. another, then you will have to link them (e.g. chain them together) so the ordering is explicit rather than left to chance.
The output you see is not surprising because the first thing your for loop does is register a bunch of .then() handlers. Because the promises are already resolved for those, the .then() handlers are all queued to run as soon as your for loop is done (.then() handlers are ALWAYS queued to run asynchronously). The for loop finishes and then the first crop of .then() handlers all run. The process of running them schedules three more .then() handlers. Those are then queued and they run when the first crop of .then() handlers is all done. While I explained the likely logic for why you get the order you see, this is not guaranteed. These are async operations and the only thing you know is that they complete some uncertain time in the future. If you want explicit order, you have to force that through explicit synchronization of your promises.
You can sequence an iteration through an array in a known order like this using a fairly common design pattern with array.reduce():
router.get('/', function(req, res, next) {
var thisArray = json[0].array;
thisArray.reduce(function(p, item) {
return p.then(function() {
return pseries(req,json,fnlist);
});
}, Promise.resolve()).then(function(result) {
// all done here
}, function(err) {
// error here
});
});
Try to chain all your promise using a foreach:
var sequence = Promise.resolve();
// Loop through our chapter urls
story.chapterUrls.forEach(function(chapterUrl) {
// Add these actions to the end of the sequence
sequence = sequence.then(function() {
return getJSON(chapterUrl);
}).then(function(chapter) {
addHtmlToPage(chapter.html);
});
});
for more complex combination, check this page:
http://www.html5rocks.com/en/tutorials/es6/promises/#toc-parallelism-sequencing