We Keep Coding

Clarification regarding promises and async/await needed

I have been using Promises and async/await, they are pretty much the same thing right? Usually what I would do is wrap my promise and return it etc.
function someFetchThatTakesTime(){
// Promisify the request.
return new Promise((resolve, reject) => {
if(allGood){
resolve();
}else{
reject();
});
}
Then I can do:
someFetchThatTakesTime()
.then(console.log('all good.')
.catch(console.log('some error occured.');
or I can do the:
async function wrapMyFetch() {
try {
// Make the async call.
data = await someFetchThatTakesTime();
return data;
} catch(err) {
// Propagate the exception up stream.
throw err;
}
}
(async () => {
let response = await wrapMyFetch();
// Do stuff with the response.
})();
I guess, pretty clear so far.
However, I have recently encountered situations where my application does not care about waiting for the results to be fetched and the data containers to be updated etc. Let's say there is one larger loop that runs infinitely and any Promised requests simply will fill-in the gaps as the application runs.
In that case, we don't really need the async/await pattern right? We just want to move forward with our loop and stones will fall in place behind us, whenever they are ready to fall in place (or not, in case of an error).
I would like to clarify this: async/await will just force things to run linearly, right? But if we do not want linearity, and we are ok with spawned tasks - Promises - to finish their thing in the background in some time when we are resuming with our run-cycle, we don't need that async/await pattern? Is that correct?

I have been using Promises and async/await, they are pretty much the same thing right?
Well, async/await are helpful syntax built on top of promises. Async/await relies on promises in order to work. So, I wouldn't quite call them the same thing, but you can code identically functioning code with await surrounded by try/catch and .then().catch(). You can use either. Both methods rely on promises.
However, I have recently encountered situations where my application does not care about waiting for the results to be fetched and the data containers to be updated etc. Let's say there is one larger loop that runs infinitely and any Promised requests simply will fill-in the gaps as the application runs.
In that case, we don't really need the async/await pattern right? We just want to move forward with our loop and stones will fall in place behind us, whenever they are ready to fall in place (or not, in case of an error).
There is no rule that the caller has to pay attention to a returned promise or has to wait for it before going on with their task. So, if you want to "just let the stones fall as they may in the background" as you say and that's appropriate for your application, you can code that way.
This is often referred to as "fire and forget" coding where you start some asynchronous operation that is going to do something on its own and you don't need to pay attention to when it finishes or if it had an error because that is of no consequence to the calling code.
But, there is a rule that you can't let a promise rejection go unhandled. So, if you're returning a promise and the caller isn't going to do anything with it, then you have to make sure that any rejections are handled by the operation itself (with .catch()) so that the only thing that ever gets returned back is a resolved promise. Even if all you do is notice the error and do nothing with it, you have to catch the error.

async/await will just force things to run linearly, right?
Not globally, no. await makes the code in the async function it's used in wait for the promise you pass it to settle, but that only affects the function you use it in, not anything calling that function.
async functions are syntactic sugar for writing a function that returns a promise. await is syntactic sugar for consuming promises. Together, they dramatically simplify using promises (in particular by making rejections errors that automatically propagate through the call tree).
More specifically, an async function runs its code synchronously until the first await or return (or until code runs off the end of the function), at which point it returns a promise. Its logic then waits for the promise to settle, and continues; eventually it settles its promise based on what happened to the last promise it awaited or returned (or fulfills it with undefined if code execution runs off the end).
If you don't want to wait for a promise to settle, you don't have to, not even in an async function. Just don't use await on it. For instance, assume we have this wrapper for fetch that gets JSON for us (and fixes the API footgun):
async function fetchJSON(...args) {
const response = await fetch(...args);
if (!response.ok) {
throw new Error("HTTP error " + response.status);
}
return response.json();
}
Here's an async function using it that does an initial query to get a list of things to fetch, and then fetches all of those things in parallel::
async function fetchList(listUrl) {
const list = await fetchJSON(listUrl);
return Promise.all(list.map(item => fetchJSON(itemUrl)));
}
Notice how fetchList uses await to wait for the list of things to fetch, but then doesn't wait for the items; it just returns the promise from Promise.all for the list of items it starts fetching.
Also note how await within fetchJSON makes fetchJSON's logic wait for the fetch promise to settle, but doesn't make the caller calling fetchJSON wait unless that caller uses await. fetchList only awaits the first call (to get the list), it doesn't wait for the item from the list.

map() with async vs promise.all()

If I have an array of elements and I want to do parallel operations on them.
I would use promise.all().
I knew promise.all() accepts array of promises. Correct me if I am wrong, I don't think so.
Here, it clearly says.
The Promise.all() method returns a single Promise that fulfills when all of the promises passed as an iterable have been fulfilled or when the iterable contains no promises or when the iterable contains promises that have been fulfilled and non-promises that have been returned. It rejects with the reason of the first promise that rejects, or with the error caught by the first argument if that argument has caught an error inside it using try/catch/throw blocks.
So, yes we can pass simple functions to promise.all(), and it resolves if they return and rejects if they throw an error.
Now look at the below code.
const promises = todayAssignedJobs.map(async todayAssigned => {
const [leaderboard, created] = await Leaderboard.findOrCreate({...});
if (!created) {
const rating = (todayAssigned.rating + leaderboard.rating * leaderboard.jobs_completed) / (leaderboard.jobs_completed + 1);
const commission = todayAssigned.commission + leaderboard.commission;
const jobsCompleted = leaderboard.jobs_completed + 1;
await Leaderboard.update({
rating,
commission,
jobs_completed: jobsCompleted,
updated_by: 'system',
}, {
where: {
id: leaderboard.id,
},
});
}
await AssignedJob.update({
is_leaderboard_generated: true,
}, {
where: {
id: todayAssigned.id,
},
});
});
await Promise.all(promises);
Here, I have a doubt.
We are iterating on each element of the array and doing operation on them asyncrhonously. They don't return anything explicitly.
So, I think map is doing parallel operations here too.
Why shuold then use promise.all() here?

.map() is not promise-aware. So, when you pass it an async callback like you are, it does not pay any attention to that returned promise. So, it just runs the loop one after another, not waiting for any of the returning promises. Thus, all the asynchronous operations started in the .map() loop will be in-flight at the same time.
If that's what you want and you want to collect all the returned promises so you can later see when they are all done with Promise.all(), then this pattern works well:
Promise.all(someArray.map(callbackThatReturnsAPromiseHere))
And, this is a common design pattern for it. In fact, the Bluebird promise library has a special function that combined these two called Promise.map(). It also offered another nice feature that let you control how many concurrent async operations could run at once (because it's map() operation is promise-aware).
It sounds like you're trying to figure out if you should only use .map() and not use Promise.all(). If you do that, you will run your asynchronous operations in parallel, but you will not have any idea when they are all done or any ability to collect all the results. You would use Promise.all() on the array of returned promises to know when they are all done and/or to collect their resolved results.
FYI, .map() is JUST a plain loop. It doesn't have any special asynchronous features or any special run-in-parallel features. You can do the same thing with a for loop if you want. It does NOT pause for your async callback to wait for it to be done so a side effect of running it is that you launch a bunch of parallel asynchronous operations.

The purpose of Promise.all here is to be able to await all the promises before continuing.
If you added a console.log immediately before the await Promise.all(promises); it would run synchronously before any promises have resolved, whereas a console.log immediately after that line will only appear after all the promises have resolved.

You only need Promise.all if you want to do something when all operations are complete. For example:
const promises = todayAssignedJobs.map(async todayAssigned => {
// lots of async stuff
});
await Promise.all(promises);
// now, all Promises have resolved
// alert the user that the leaderboard is completely updated
If you don't need anything to occur once you've determined that all Promises have completed, then there's no point to the Promise.all - you may as well just create the Promises in a loop and leave them as-is. In that case, since you wouldn't be using the resulting array of Promises, it would be more appropriate to use forEach (which is the array method to use for side-effects).
There's one issue, though - you aren't handling errors, but errors should be handled, otherwise they'll give warnings or exit the Node process:
const processJob = async (todayAssigned) => {
const [leaderboard, created] = await Leaderboard.findOrCreate({...});
if (!created) {
// ...
// ...
todayAssignedJobs.forEach(async todayAssigned => {
try {
await processJob(todayAssigned);
} catch(e) {
// handle errors
}
});

Why should we wrap await inside an async function?

Why should we have an async function in order to use await? Why can't we just use await without async? JS is async by default too, this just adds to the confusion.
Update:
I've seen some lads put my question on hold so I'll try to elaborate.
I'm just curious as to why this won't work:
some code
let users = await getUsers();
some code
Why should it be inside an async for it to work, ie
$(async function() {
some code
let users = await getUsers();
some code
});

JS is async by default too...
No, JavaScript is not async by default. The only async features of JavaScript are fairly newly-added:
Promise resolution
async/await
JavaScript is commonly used in environments where you interact with asynchronous things (like event handlers in the DOM, or I/O completions in Node.js), but JavaScript is not asynchronous (other than above).
In the words of Allen Wirfs-Brock, who was the editor of the ECMAScript specification for many years, JavaScript...
(has) an observably synchronous execution model. Other than via Atomics/SABs there are no observable shared-state race conditions.
Back to your question:
Why should we have an async function in order to use await?
Before too long, with modules you won't have to, once the top level await proposal finishes working through the process. It just got to Stage 3.
But the answer is that await is syntactic sugar for consuming a promise, and one of the rules of promises is that you either handle errors or return the chain to the caller (so it can handle errors or return the chain to its caller). await doesn't handle errors, so it has to return the chain to the caller. The way it does that is that an async function always returns a promise, and that promise is chained to the promise await awaits.
That is, this:
async function foo() {
const thingy = await somethingAsyncReturningAPromise();
return thingy.foo;
}
is conceptually (but not literally) this:
function foo() {
return somethingAsyncReturningAPromise()
.then(thingy => thingy.foo);
}
If something goes wrong in somethingAsyncReturningAPromise, the promise returned by foo rejects — the error is propagated to the caller.
As far as I can tell from the top-level await proposal, it simply allows unhandled rejections at the top level of the module to be unhandled rejections. So just like this code causes an unhandled error:
null.doSomething();
this code in an async module would cause an unhandled rejection:
await somethingThatReturnsAPromiseAndRejects();

Why should we have an async function in order to use await? Why can't we just use await without async?
Because async/await is "just" syntactic sugar for Promises. If the function is async, then it returns a Promise. It is not possible to have the "await" behaviour without returning a promise. The fact that the function is async has to be marked explicitly.
JS is async by default too, this just adds to the confusion.
This statement is too "simplified". While it is true that JS is async in nature, because of the event loop, this doesn't mean that every function has an async behavior. This does not add to the confusion. You're probably confused due to misunderstanding how JS really works. You should read about Promises, which are behind the scenes when you see async/await.

JavaScript has task based concurrency. It basically means that code blocks (tasks) runs synchronously without being interrupted, until it reaches a certain breakpoint (the end of a task). That has some advantages:
1) You do have concurrency, e.g. a network call does not block your script from doing other things in the meantime (so the task that consumes the network request will only run if the network request was done, other tasks can be done in the meantime).
2) On the other hand, you do not have concurrent mutations, which eliminates a lot of problems (let a = 1; a += 1; could evaluate to 3, you would need locks / semaphores to prevent those, c.f. Java & others).
Now async / await allow you to define such tasks:
An async function can be divided into tasks, await serves as a breakpoint:
let a = 1;
async function stuff() {
a = a + 1; // this is totally secure, as no other code might run in the meantime
a = a + await other(); // this is unsafe, as we await, which means that other tasks might be executed in the meantime.
}
If you want to "await in non async functions" that basically means that you won't know wether a certain function call runs synchronously (meaning: without other code running in the meantime) or not:
function dangerous() { await stuff(); }
let a = 1;
a = a + dangerous(); // does that work or not? Can a be manipulated in the meantime?
So with your proposal you would basically remove the borders around tasks, every code might run every inbetween. So at the end that causes chaos, and chaos is not good if you want to be productive.

using forEach and async/await, behaves different for node and Jest

I have a function that writes data to a mongodb, like so:
const writeToDB = async (db, data) => {
const dataKeys = Object.keys(data)
dataKeys.forEach(async key => db.collection(key).insertMany(data[key]))
}
This works fine if I run it in a node script. But when I tried to use it in Jest's beforeAll I got this async error from Jest:
Jest did not exit one second after the test run has completed. This
usually means that there are asynchronous operations that weren't
stopped in your tests.
after some troubleshooting I found out that forEach was causing the trouble. Using a for loop solved this problem:
const writeToDB = async (db, data) => {
const dataKeys = Object.keys(data)
for (const key of dataKeys) {
await db.collection(key).insertMany(data[key])
}
}
Searching for this problem I came across this article:
https://codeburst.io/javascript-async-await-with-foreach-b6ba62bbf404
The explanation there made sense, but it left me with some questions:
According to the article, it should not have worked even in the node
script. How come it did?
Why is executing it in node different from running it in Jest?
edit
After reading all the comments, I realise my first question was a bit nonsense. Normally I assign the result of an async function to a variable, and if I don't put await, there will an undefined error down the line. But that's not the case here, so script exits normally and the db writes happen concurrently in the background.

The existing answer already explains in detail why forEach shouldn't be used with promises the way it's used. forEach callback doesn't take returned promises into account and breaks promise chain. async..await needs to be used with for..of to evaluate promises in series or with Promise.all and map to evaluate in parallel.
Jest supports promises and expects that a promise that is returned from asynchronous function (it, etc) signifies that asynchronous process that occurred in this function has ended.
Once Jest finishes a test, it checks if there are open handles that prevent Node from exiting. Since promises weren't returned and chained by Jest, processes that they represent prevent Jest from finishing test process.
This problem is represented by said error message:
Jest did not exit one second after the test run has completed.
This usually means that there are asynchronous operations that weren't
stopped in your tests. Consider running Jest with --detectOpenHandles
to troubleshoot this issue.

Async functions work even in contexts that don't await or call .then() on them, that is, I can definitely do this:
async function foo() {
// async calls here
}
foo(); // no await or .then().
This means you cannot wait for the operation to finish, you can't use the value, and worst of all, you cannot catch or recover from any async errors that might be thrown (or rejected, if we're being accurate)
The main difference is that the .forEach() doesn't care or wait for the operations to finish before calling the next one (since async functions return immediately), whereas your for..of call uses await to wait for each operation to complete before moving on to the next.
Your first .forEach example would be roughly equivalent to the bottom one if you removed the await from the call inside of the loop.
The result is that your first example returns a Promise that resolves immediately, rather than after all of your DB calls were finished, so the test expects the operations to finish, but they are not. In the second example, you properly await for all the calls to finish before the async function finishes, so the return Promise will only resolve after all of the calls resolve themselves, first.
On that note, the two examples are not equivalent because the first would call the insertMany one after the other without waiting for them to finish, causing the DB calls to perform in parallel.
If you want to preserve this behavior, but still return a correct Promise which waits for everything to finish, you should be using [].map() instead of [].forEach:
const writeToDB = async (db, data) => {
const dataKeys = Object.keys(data)
const allPromises = dataKeys.map(async key =>
await db.collection(key).insertMany(data[key])) // run all calls in parallel
return await Promise.all(allPromises); // wait for them all to finish
}

Differences between Futures in Python3 and Promises in ES6

Since Python 3.5, the keywords await and async are introduced to the language. Now, I'm more of a Python 2.7 person and I have been avoiding Python 3 for quite some time so asyncio is pretty new to me. From my understanding it seems like await/async works very similar to how they work in ES6 (or JavaScript, ES2015, however you want to call it.)
Here are two scripts I made to compare them.
import asyncio
async def countdown(n):
while n > 0:
print(n)
n -= 1
await asyncio.sleep(1)
async def main():
"""Main, executed in an event loop"""
# Creates two countdowns
futures = asyncio.gather(
countdown(3),
countdown(2)
)
# Wait for all of them to finish
await futures
# Exit the app
loop.stop()
loop = asyncio.get_event_loop()
asyncio.ensure_future(main())
loop.run_forever()
function sleep(n){
// ES6 does not provide native sleep method with promise support
return new Promise(res => setTimeout(res, n * 1000));
}
async function countdown(n){
while(n > 0){
console.log(n);
n -= 1;
await sleep(1);
}
}
async function main(){
// Creates two promises
var promises = Promise.all([
countdown(3),
countdown(2)
]);
// Wait for all of them to finish
await promises;
// Cannot stop interpreter's event loop
}
main();
One thing to notice is that the codes are very similar and they work pretty much the same.
Here are the questions:
In both Python and ES6, await/async are based on generators. Is it a correct to think Futures are the same as Promises?
I have seen the terms Task, Future and Coroutine used in the asyncio documentation. What are the differences between them?
Should I start writing Python code that always has an event loop running?

In both Python and ES6, await/async are based on generators. Is it a correct to think Futures are the same as Promises?
Not Future, but Python's Task is roughly equivalent to Javascript's Promise. See more details below.
I have seen the terms Task, Future and Coroutine used in the asyncio documentation. What are the differences between them?
They're quite different concepts. Mainly, Task consists of Future and Coroutine. Let's describe these primitives briefly (I am going to simplify lots of things to describe only main principles):
Future
Future is simply an abstraction of value that may be not computed yet and will be available eventually. It's a simple container that only does one thing - whenever the value is set, fire all registered callbacks.
If you want to obtain that value, you register a callback via add_done_callback() method.
But unlike in Promise, the actual computation is done externally - and that external code has to call set_result() method to resolve the future.
Coroutine
Coroutine is the object very similar to Generator.
A generator is typically iterated within for loop. It yields values and, starting from PEP342 acceptance, it receives values.
A coroutine is typically iterated within the event loop in depths of asyncio library. A coroutine yields Future instances. When you are iterating over a coroutine and it yields a future, you shall wait until this future is resolved. After that you shall send the value of future into the coroutine, then you receive another future, and so on.
An await expression is practically identical to yield from expression, so by awaiting other coroutine, you stop until that coroutine has all its futures resolved, and get coroutine's return value. The Future is one-tick iterable and its iterator returns actual Future - that roughly means that await future equals yield from future equals yield future.
Task
Task is Future which has been actually started to compute and is attached to event loop. So it's special kind of Future (class Task is derived from class Future), which is associated with some event loop, and it has some coroutine, which serves as Task executor.
Task is usually created by event loop object: you give a coroutine to the loop, it creates Task object and starts to iterate over that coroutine in manner described above. Once the coroutine is finished, Task's Future is resolved by coroutine's return value.
You see, the task is quite similar to JS Promise - it encapsulates background job and its result.
Coroutine Function and Async Function
Coroutine func is a factory of coroutines, like generator function to generators. Notice the difference between Python's coroutine function and Javascript's async function - JS async function, when called, creates a Promise and its internal generator immediately starts being iterated, while Python's coroutine does nothing, until Task is created upon it.
Should I start writing Python code that always has an event loop running?
If you need any asyncio feature, then you should. As it turns out it's quite hard to mix synchronous and asynchronous code - your whole program had better be asynchronous (but you can launch synchronous code chunks in separate threads via asyncio threadpool API)

I see the main difference downstream.
const promise = new Promise((resolve, reject) => sendRequest(resolve, reject));
await promise;
In JavaScript, the two resolve and reject functions are created by the JS engine and they have to be passed around for you to keep track of them. In the end, you're still using two callback functions most of the time and the Promise won't really do more than setTimeout(() => doMoreStuff()) after doStuff calls resolve. There's no way to retrieve an old result or the status of a Promise once the callbacks were called. The Promise is mostly just the glue code between regular calls and async/await (so you can await the promise somewhere else) and a bit of error callback forwarding for chaining .thens.
future = asyncio.Future()
sendRequest(future)
await future
In Python, the Future itself becomes the interface with which a result is returned and it keeps track of the result.
Since Andril has given the closest Python equivalent to JavaScript's Promise (which is Task; you give it a callback and wait for it to complete), I'd like to go the other way.
class Future {
constructor() {
this.result = undefined;
this.exception = undefined;
this.done = false;
this.success = () => {};
this.fail = () => {};
}
result() {
if (!this.done) {
throw Error("still pending");
}
return this.result();
}
exception() {
if (!this.done) {
throw Error("still pending");
}
return this.exception();
}
setResult(result) {
if (this.done) {
throw Error("Already done");
}
this.result = result;
this.done = true;
this.success(this.result);
}
setException(exception) {
if (this.done) {
throw Error("Already done");
}
this.exception = exception;
this.done = true;
this.fail(this.exception);
}
then(success, fail) {
this.success = success;
this.fail = fail;
}
}
The JS await basically generates two callbacks that are passed to .then, where in a JS Promise the actual logic is supposed to happen. In many examples, this is where you'll find a setTimeout(resolve, 10000) to demonstrate the jump out of the event loop, but if you instead keep track of those two callbacks you can do with them whatever you want..
function doStuff(f) {
// keep for some network traffic or so
setTimeout(() => f.setResult(true), 3000);
}
const future = new Future();
doStuff(future);
console.log('still here');
console.log(await future);
The above example demonstrates that; three seconds after 'still here' you get 'true'.
As you can see, the difference is Promise receives a work function and deals with resolve and reject internally, while Future doesn't internalize the work and only cares about the callbacks. Personally, I prefer the Future because it's one layer less callback hell - which was one of the reasons for Promises in the first place: callback chaining.