I've got a recursive function that loops over an object. It looks for keys with the name subcomponent (which is always an array) and performs an asynchronous function on each of subcomponent's children, the output of which is used to replace the child's data.
In the example below populateSubcomponent() is the async function.
Code:
async function doPopulate(data) {
Object.keys(data).map((key) => {
if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
const promises = data[key].map(subcomponent => populateSubcomponent(subcomponent));
return Promise.all(promises).then((output) => {
data[key] = output;
console.log('1');
});
}
doPopulate(data[key]); // Check recursively
console.log('2');
}
console.log('3');
return data;
});
}
doPopulate(data);
My expectations are that each of the console.log numbers should fire sequentially, but instead I get 2, 3, then 1. As a result, the recursive functions runs before the async function has completed, therefore never replacing the child as intended; I get the correct result at 1 but it's not passed to 2 or 3.
How do I best incorporate the recursive doPopulate() call with the if statement?
I've looked at the following SO posts:
Using async/await inside for loop
Best way to call an async function within map?
Async/Await inside Array#map()
Use async await with Array.map
but I can't relate any of the answers to my own problem, mostly due to the fact that I've got an if statement within my recursive function and I'm not sure how to deal with that in context of the async stuff.
Edit
Thanks to everyone's comments I came up with the following:
async function doPopulate(data) {
const promisesOuter = Object.keys(data).map(async (key) => {
if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
const promisesInner = data[key].map(subcomponent => populateSubcomponent(subcomponent));
data[key] = await Promise.all(promisesInner);
}
await doPopulate(data[key]); // Check recursively
}
return data;
});
return Promise.all(promisesOuter);
}
return doPopulate(data);
As this all happens within a NodeJS stream (using through2) I also needed to make the stream function async too:
const through = require('through2');
return through.obj(async (file, enc, done) => {
const data = JSON.parse(file.contents.toString());
await doPopulate(data);
file.contents = Buffer.from(JSON.stringify(data));
return done(null, file);
});
You will have to await every promise sequentially as long as the next promise relies on the previous one. If they can be done concurrently, you just have to aggregate the promises and await them together with Promise.all. By the way you only need the async keyword if you need to use await. Otherwise there's no reason to make the function async.
If you wanted to wait for each promise sequentially you would rewrite it like this:
function doPopulate(data) {
return Object.keys(data).map(async (key) => {
if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
const promises = data[key].map((subcomponent) =>
populateSubcomponent(subcomponent)
);
data[key] = await Promise.all(promises);
console.log('1');
}
await doPopulate(data[key]); // Check recursively
console.log('2');
}
console.log('3');
return data;
});
}
Turns out I needed to return two sets of promises as #Bergi suggested in the comments:
async function doPopulate(data) {
const promisesOuter = Object.keys(data).map(async (key) => {
if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
const promisesInner = data[key].map(subcomponent => populateSubcomponent(subcomponent));
data[key] = await Promise.all(promisesInner);
}
await doPopulate(data[key]); // Check recursively
}
return data;
});
return Promise.all(promisesOuter);
}
return doPopulate(data);
As this all happens within a NodeJS stream (using through2) I also needed to make the stream function async too:
return through.obj(async (file, enc, done) => {
const data = JSON.parse(file.contents.toString());
await doPopulate(data);
file.contents = Buffer.from(JSON.stringify(data));
return done(null, file);
});
Related
I have multiple functions that pass data from one to another. I moved one of the functions to the backend and receive data as API with Axios. Now I cannot manage to assign data from Axios to some local variable.
Simple code would be like:
function function1()
{
axios({get, url})
.then(response => {
globalVariable = response.data;
function2(globalVariable);
}
function function2(globalVariable)
{
const local = globalVariable;
return local;
}
And then inside of function3, I want to do:
function function3()
{
const from_local = function2()
from_local
}
When I try this I receive undefined result. Please help.
This is what promises are for. No need for globals or jumping through hoops to get the data out. Just remember to await any function that's async, (like axios) and annotate any function that contains an "await" as being async.
// note "async" because it contains await
async function backend() {
// note await because axios is async
const response = await axios({get, url});
return response.data;
}
// same thing up the calling chain
async function middleend() {
const local = await backend();
return local;
}
async function frontend() {
const local = await middleend();
console.log('ta da! here\'s the data', local);
}
It looks like you are looking for some sort of piping asynchronous operation. By piping I mean result of one function execution will be feed to another.
So basically function1 which is mimicking a axios operation here.
// making an api call
function function1() {
return fetch('https://jsonplaceholder.typicode.com/todos/1').then((d) =>
d.json()
);
}
// some random function
function function2(data) {
console.log(' Calling function 2 ');
return data?.title;
}
// some random function
function function3(data) {
console.log(' Calling function 3 ');
return `Hello ${data}`;
}
/** a function to resolve functions sequentially. The result of first
function will be input to another function.
Here ...fns is creating an array like object
so array operations can be performed here **/
const runAsynFunctions = (...fns) => {
return (args) => {
return fns.reduce((acc, curr) => {
return acc.then(curr);
}, Promise.resolve(args));
};
};
// calling runAsynFunctions with and passing list of
// functions which need to resolved sequentially
const doOperation = runAsynFunctions(function2, function3);
// resolving the api call first and the passing the result to
// other functions
function1().then(async(response) => {
const res = await doOperation(response);
console.log(res);
});
I have an async function that returns an object after running fetch and .json(). I want to have more than one callback that take the object as input. Since I need to chain .then() to call the callback, I need to run the fetching function each time.
Is there a way to fetch once, and then have the callbacks do their thing with the output without having to refetch?
async function getData(){
const response = await fetch('api-url');
return await response.json();
}
function foo(data){
// do stuff
}
function bar(data){
// do stuff
}
const doFoo = getData().then(foo) // I don't want to run getData()
const doBar = getData().then(bar) // each time
I guess, I can save the output to a cache, but is there a way to do it with promises?
Any time you want to generate a value once and then use it multiple times: Store it in a variable.
const data = getData();
data.then(foo);
data.then(bar);
//to check the state of the promise
undefined === Promise.prototype.state && (Promise.prototype.state = function () {
const t = {};
return Promise.race([this, t]).then(v => (v === t) ? 'pending' : 'fulfilled', () => 'rejected');
});
function promiseWrapper(promiseFunction) {
let getPromise = Promise.reject();
let getFinalData = async function(force) {
let args = [].slice.call(arguments, 1);
//if force available, api will be called
if(force) {
return promiseFunction.apply(this, args);
}
let state = await this.getPromise.state();
//if state is rejected, trigger and fetch the data
'rejected' === state && (this.getPromise = promiseFunction.apply(this, args));
return this.getPromise;
}
//get method first method is force, rest will be treated as arguments
return {getPromise, get: getFinalData};
}
//instead of creating function pass the function in promiseWrapper
//promiseWrapper helps if somehow it is rejected, we wan recall the function without refreshing the page
var getData = promiseWrapper(async function() {
const response = await fetch('https://reqres.in/api/products/3');
return await response.json();
});
function foo(data) {
console.log('foo data\n', data)
}
function bar(data) {
console.log('bar data\n', data)
}
getData.get().then(foo) //It will trigger the api, because it is first time
getData.get(true).then(bar) //data by calling api
//getData.get(false).then(console.log) and getData.get().then(console.log), will act same
This is how I'm using a sequential forEach loop to handle some file uploads. Now I want to get a total result for each upload:
For each file I resolve with an object (success or error). At the end I want to get a returned array to display the result (File 1 successful, file 2 error and so on).
But with this code I only get undefined for the last output. What am I missing?
Maybe it would even be better to return two string arrays: one successful and one failed array with filenames.
Array.prototype.forEachSequential = async function (
func: (item: any) => Promise<void>
): Promise<void> {
for (let item of this) await func(item)
}
async uploadFiles(files: [File]): Promise<any> {
const result = await files.forEachSequential(
async (file): Promise<any> => {
const res = await new Promise(async (resolve, reject) => {
// do some stuff
resolve({ success: file.filename })
// or maybe
resolve({ error: file.filename })
})
console.log(res) // returns object as expected
return res
}
)
// Get all results after running forEachSequential
console.log('result', result) // result: undefined
})
Your forEachSequential function has no return statement, so it never returns anything, so calling it will always result in undefined (in this specific case, a promise that's fulfilled with undefined, since it's an async function). It completely ignores the return value of the callback.
You probably want a mapSequential (or mapSerial) instead, that builds an array of the results:
Object.defineProperty(Array.prototype, "mapSequential", {
async value(func) {
const result = [];
for (const item of this) {
result.push(await func(item));
}
return result;
},
writable: true,
configurable: true,
});
Note: I recommend you don't add to Array.prototype. Consider doing this instead:
async function mapSequential(iterable, func) {
const result = [];
for (const item of iterable) {
result.push(await func(item));
}
return result;
}
But if you do add to Array.prototype, be sure to do it with defineProperty as I did above, not just with assignment, and make sure the property you add isn't enumerable (either use enumerable: false or leave it off, false is the default).
You can spin this a lot of ways. You might consider a variant similar to Promise.allSettled that catches errors and returns an array indicating success/failure (with or without stopping early on first error). For instance:
async function mapSequential(iterable, func, options) {
const result = [];
for (const item of iterable) {
try {
const value = await func(item);
result.push({ status: "fulfilled", value });
} catch (reason) {
result.push({ success: "rejected", reason });
if (options?.shortCircuit) { // Option that says "stop at first error"
break;
}
}
}
return result;
}
Again, though, you can spin it lots of ways. Maybe add an index and the iterable to the callback like map does, perhaps have an optional thisArg (although arrow functions make that largely unnecessary these days), etc.
Your forEachSequential doesn't really return anything. That's why you get undefined. When awaiting, you should attach a result to some variable, store it somewhere and then at the end of this function you should return it.
Now, you would like to return a tuple [string[], string[]]. To do so you have to create to arrays at the beginning of forEachSequential body, then call func in the loop and add a result to first array (let's say - success array) or second (failure array).
const successArray = [];
const failureArray = [];
for (let item of this) {
const result = await func(item);
if ('success' in result) {
successArray.push(result.success);
} else {
failureArray.push(result.error);
}
}
return [successArray, failureArray];
I am using a function fetchChain called with an array of objects, each object includes the url and other parameters to call another function fetchData to chain fetch-Calls. The async/await loop returns its results to the array I fed it with, like so (simplified):
fetchChain(array){
const loop = async (items) => {
for (let i = 0; i < items.length; i++) {
await fetchData(items[i])
}
}
return loop(array)
.then(()=>{ return array })
}
I can wait for all promises/results to return like so:
fetchChain(prepareAjax)
.then((res)=> console.log(res) )
.then(()=> console.log('done') )
So why does the loop returns its results to the array it got fed with? There is no specific return or then which returns the results to the origin and I can't wrap my head around about what happens here.
As requested, the fetchDacta-function:
fetchData(obj){
// some parameters get added, like so
obj.timeout = obj.timeout || 10000, [...];
const fetchRetry = async (url, options, n) => {
try {
return await fetch(url, options)
.then((response)=> return response.json());
} catch(err) {
if (n === 1) throw err;
return await sleep(obj.delay)
.then(()=> fetchRetry(url, options, n - 1));
}
};
return fetchRetry(url, {}, obj.retries);
}
I'm not sure if I understand the question correctly, but I think you're asking why the array argument in function fetchChain contains information assigned in fetchData.
For this you have to look at the difference of passing by reference vs passing by value.
In JavaScript Objects and Arrays are automatically passed by reference to a function; which means array points to the same memory as items, and when you modify items you are modifying array.
This is a simple example to illustrate passing by reference
let changeMessage = (o) => o.message = 'world';
let obj = { message: 'hello'};
changeMessage(obj);
console.log(obj.message);
// Output: world
You can avoid modifying the array by cloning it first
Consider this code
const response = await fetch('<my url>');
const responseJson = await response.json();
responseJson = _.sortBy(responseJson, "number");
responseJson[0] = await addEnabledProperty(responseJson[0]);
What addEnabledProperty does is to extend the object adding an enabled property, but this is not important. The function itself works well
async function addEnabledProperty (channel){
const channelId = channel.id;
const stored_status = await AsyncStorage.getItem(`ChannelIsEnabled:${channelId}`);
let boolean_status = false;
if (stored_status == null) {
boolean_status = true;
} else {
boolean_status = (stored_status == 'true');
}
return _.extend({}, channel, { enabled: boolean_status });
}
Is there a way to use _.map (or another system), to loop trough entire responseJson array to use addEnabledProperty against each element?
I tried:
responseJson = _.map(responseJson, function(channel) {
return addEnabledProperty(channell);
});
But it's not using async so it freeze the app.
I tried:
responseJson = _.map(responseJson, function(channel) {
return await addEnabledProperty(chanell);
});
But i got a js error (about the row return await addEnabledProperty(chanell);)
await is a reserved word
Then tried
responseJson = _.map(responseJson, async function(channel) {
return await addEnabledProperty(channell);
});
But I got an array of Promises... and I don't understand why...
What else!??
EDIT: I understand your complains about I didn't specify that addEnabledProperty() returns a Promise, but, really, I didn't know it. In fact, I wrote "I got an array of Promises... and I don't understand why "
To process your response jsons in parallel you may use Promise.all:
const responseJson = await response.json();
responseJson = _.sortBy(responseJson, "number");
let result = await Promise.all(_.map(responseJson, async (json) =>
await addEnabledProperty(json))
);
Since addEnabledProperty method is async, the following also should work (per #CRice):
let result = await Promise.all(_.map(responseJson, addEnabledProperty));
I found that I didn't have to put the async / await inside of the Promise.all wrapper.
Using that knowledge, in conjunction with lodash chain (_.chain) could result in the following simplified version of the accepted answer:
const responseJson = await Promise.all( _
.chain( response.json() )
.sortBy( 'number' )
.map( json => addEnabledProperty( json ) )
.value()
)
How about using partial.js(https://github.com/marpple/partial.js)
It cover both promise and normal pattern by same code.
_p.map([1, 2, 3], async (v) => await promiseFunction());
You can use Promise.all() to run all the promises in your array.
responseJson = await Promise.all(_.map(responseJson, (channel) => {
return addEnabledProperty(channel);
}));
If you want to iterate over some object, you can use keys first to get an array of keys and then just loop over your keys while awaiting for necessary actions.
This works when you want to wait until every previous iteration step is finished before getting into the next one.
Here is a consolidated asyncMap function that you can use for your object:
async function asyncMap(obj, cb) {
const result = {};
const keysArr = keys(obj);
let keysArrLength = keysArr.length;
while (keysArrLength-- > 0) {
const key = keysArr[keysArrLength];
const item = obj[key];
// eslint-disable-next-line no-await-in-loop
result[key] = await cb(item, key);
}
return result;
}
And then, for your example case:
responseJson = await asyncMap(responseJson, addEnabledProperty);
Otherwise use Promise.all like was proposed above to run all the iteration steps in parallel