We Keep Coding

Aggregate multiple calls then separate result with Promise

Currently I have many concurrent identical calls to my backend, differing only on an ID field:
getData(1).then(...) // Each from a React component in a UI framework, so difficult to aggregate here
getData(2).then(...)
getData(3).then(...)
// creates n HTTP requests... inefficient
function getData(id: number): Promise<Data> {
return backend.getData(id);
}
This is wasteful as I make more calls. I'd like to keep my getData() calls, but then aggregate them into a single getDatas() call to my backend, then return all the results to the callers. I have more control over my backend than the UI framework, so I can easily add a getDatas() call on it. The question is how to "mux" the JS calls into one backend call, the "demux" the result into the caller's promises.
const cache = Map<number, Promise<Data>>()
let requestedIds = []
let timeout = null;
// creates just 1 http request (per 100ms)... efficient!
function getData(id: number): Promise<Data> {
if (cache.has(id)) {
return cache;
}
requestedIds.push(id)
if (timeout == null) {
timeout = setTimeout(() => {
backend.getDatas(requestedIds).then((datas: Data[]) => {
// TODO: somehow populate many different promises in cache??? but how?
requestedIds = []
timeout = null
}
}, 100)
}
return ???
}
In Java I would create a Map<int, CompletableFuture> and upon finishing my backend request, I would look up the CompletableFuture and call complete(data) on it. But I think in JS Promises can't be created without an explicit result being passed in.
Can I do this in JS with Promises?

A little unclear on what your end goal looks like. I imagine you could loop through your calls as needed; Perhaps something like:
for (let x in cache){
if (x.has(id))
return x;
}
//OR
for (let x=0; x<id.length;x++){
getData(id[x])
}
Might work. You may be able to add a timing method into the mix if needed.
Not sure what your backend consists of, but I do know GraphQL is a good system for making multiple calls.
It may be ultimately better to handle them all in one request, rather than multiple calls.

The cache can be a regular object mapping ids to promise resolution functions and the promise to which they belong.
// cache maps ids to { resolve, reject, promise, requested }
// resolve and reject belong to the promise, requested is a bool for bookkeeping
const cache = {};
You might need to fire only once, but here I suggest setInterval to regularly check the cache for unresolved requests:
// keep the return value, and stop polling with clearInterval()
// if you really only need one batch, change setInterval to setTimeout
function startGetBatch() {
return setInterval(getBatch, 100);
}
The business logic calls only getData() which just hands out (and caches) promises, like this:
function getData(id) {
if (cache[id]) return cache[id].promise;
cache[id] = {};
const promise = new Promise((resolve, reject) => {
Object.assign(cache[id], { resolve, reject });
});
cache[id].promise = promise;
cache[id].requested = false;
return cache[id].promise;
}
By saving the promise along with the resolver and rejecter, we're also implementing the cache, since the resolved promise will provide the thing it resolved to via its then() method.
getBatch() asks the server in a batch for the not-yet-requested getData() ids, and invokes the corresponding resolve/reject functions:
function getBatch() {
// for any
const ids = [];
Object.keys(cache).forEach(id => {
if (!cache[id].requested) {
cache[id].requested = true;
ids.push(id);
}
});
return backend.getDatas(ids).then(datas => {
Object.keys(datas).forEach(id => {
cache[id].resolve(datas[id]);
})
}).catch(error => {
Object.keys(datas).forEach(id => {
cache[id].reject(error);
delete cache[id]; // so we can retry
})
})
}
The caller side looks like this:
// start polling
const interval = startGetBatch();
// in the business logic
getData(5).then(result => console.log('the result of 5 is:', result));
getData(6).then(result => console.log('the result of 6 is:', result));
// sometime later...
getData(5).then(result => {
// if the promise for an id has resolved, then-ing it still works, resolving again to the -- now cached -- result
console.log('the result of 5 is:', result)
});
// later, whenever we're done
// (no need for this if you change setInterval to setTimeout)
clearInterval(interval);

I think I've found a solution:
interface PromiseContainer {
resolve;
reject;
}
const requests: Map<number, PromiseContainer<Data>> = new Map();
let timeout: number | null = null;
function getData(id: number) {
const promise = new Promise<Data>((resolve, reject) => requests.set(id, { resolve, reject }))
if (timeout == null) {
timeout = setTimeout(() => {
backend.getDatas([...requests.keys()]).then(datas => {
for (let [id, data] of Object.entries(datas)) {
requests.get(Number(id)).resolve(data)
requests.delete(Number(id))
}
}).catch(e => {
Object.values(requests).map(promise => promise.reject(e))
})
timeout = null
}, 100)
}
return promise;
}
The key was figuring out I could extract the (resolve, reject) from a promise, store them, then retrieve and call them later.

Javascript: reject still calling next then in the sequence

I am recursively calling a function which returns a Promise, but i am noticing that the next then is called even if i am rejecting with error. Below is the relevant part of the code.
const applyFilters = (counter) => {
return new Promise((resolve, reject) => {
let filter = filters[counter];
if(filter) {
applyStep(filter).then(promiseTimeout(filter.delay), function(err) {
console.log('reject with error');
reject(err);
}).then(function(res) {
console.log('still executing after reject');
resolve(applyFilters(++counter).catch(function(err) {
reject(err);
}));
});
} else {
resolve(true);
}
});
};
const applyStep = (step) => {
if(step.step_type == 'filter') {
return worksheet.applyFilterAsync(step.name, values, 'replace');
} else if(step.step_type == 'parameter') {
return workbook.changeParameterValueAsync(`${step.name}`, value);
} else {
return Promise.resolve(true);
}
};
I am seeing on console
reject with error
still executing after reject
Is this the expected behaviour, may be I am missing something. Any help in understating this further will be really great. Thanks.

You are passing the second callback to then, which handles the error (in this case by rejecting the outer promise), and then fulfills the promise returned by the then() call with the callback return value (undefined). The next then() in the chain will be executed once that promise is fulfilled.
I could tell you how to work around this problem by using a different then/catch structure, but really you need to avoid the Promise constructor antipattern here!
function applyFilters(counter) {
if (counter >= filter.length)
return Promise.resolve(true);
const filter = filters[counter];
return applyStep(filter)
.then(promiseTimeout(filter.delay))
.then(res => applyFilters(++counter));
}

How to get data from async functions and after getting data allow the next code to execute

I get some data from mongodb using mongoose find() and perform some validation on that data, but the problem is that this function is async and it does not wait for the data to be completed and execute the next code.
and when the next code is executed it enables to perform because it has null data. then i wrote my validation logic in the async function so that when data is available only then it move to next code but on every return it sends undefined data.
function isValidObject(obj) {
schemaData.find({}, (error, data) => { // calls data from db
var contactSchema = data; // this is data i need
if(//validation fails){
return "wrong data";// every time it returns undrfined on every
// condition
}
});
}
var validationResp = isValidObject(obj);
console.log(validationResp); // output is undefined
i also used "await" to wait for the data, but in that case it return [object promise] on every return statement

use async/await
In your case:
async function isValidObject(obj) {
let data = await schemaData.find({}); // wait until it resolve promise
//do your validaion
return data;
}
isValidObject(obj).then((validationResp)=>{console.log(validationResp)});

use the then() method it return a promise
var promise1 = new Promise(function(resolve, reject) {
resolve('Success!');
});
promise1.then(function(value) {
console.log(value);
// expected output: "Success!"
});
more details at MDN Using the then method

Are you familiar with "promises".You can use ".then(callback)". It will wait until async function is executed. And you can perform your validation in the callback function
User.findById(id)
.then(function(user) {
if (!user) {
return "here you return an error";
}
return "you return OK with payload from database";
})
.catch(next);

What is happening in your case is that when you assign var validationResp = isValidObject(obj); the function isValidObject(obj) has not returned anything and has only scheduled a callback(This is a very important concept when working with callbacks). As pointed out above you need to use Promises. The below is an example of your case as to how you can use Promises.
function isValidObject(obj){
return new Promise((resolve,reject) => {
schemaData.find({}, (error, data) => { // calls data from db
if(validation fails){
reject(error)// every time it returns undrfined on every
// condition
}else{
var contactSchema = data; // this is data i need
resolve(data)// You can resolve the promise with the data directly instead
// of assigning it to a variable or you can use (resolve(contactSchema))
}
})
})
}
After this when you want to use the data you can use do something like the below code snippet.
isValidObject(obj)
.then(result => {
// Use the result object here. This is the result from the Database
})
.catch(error => {
//Handle the Error
})

Chaining Promises and Passing Parameters between Them

I'm new to Node/Express and am trying to use Promises to executive successive API calls to Apple's CloudKit JS API.
I'm unclear on how to put the functions in sequence and pass their respective return values from one function to the next.
Here's what I have so far:
var CloudKit = require('./setup')
//----
var fetchUserRecord = function(emailConfirmationCode){
var query = { ... }
// Execute the query
CloudKit.publicDB.performQuery(query).then(function (response) {
if(response.hasErrors) {
return Promise.reject(response.errors[0])
}else if(response.records.length == 0){
return Promise.reject('Email activation code not found.')
}else{
return Promise.resolve(response.records[0])
}
})
}
//-----
var saveRecord = function(record){
// Update the record (recordChangeTag required to update)
var updatedRecord = { ... }
CloudKit.publicDB.saveRecords(updatedRecord).then(function(response) {
if(response.hasErrors) {
Promise.reject(response.errors[0])
}else{
Promise.resolve()
}
})
}
//----- Start the Promise Chain Here -----
exports.startActivation = function(emailConfirmationCode){
CloudKit.container.setUpAuth() //<-- This returns a promise
.then(fetchUserRecord) //<-- This is the 1st function above
.then(saveRecord(record)) //<-- This is the 2nd function above
Promise.resolve('Success!')
.catch(function(error){
Promise.reject(error)
})
}
I get an error near the end: .then(saveRecord(record)) and it says record isn't defined. I thought it would somehow get returned from the prior promise.
It seems like this should be simpler than I'm making it, but I'm rather confused. How do I get multiple Promises to chain together like this when each has different resolve/reject outcomes?

There are few issues in the code.
First: you have to pass function to .then() but you actually passes result of function invocation:
.then(saveRecord(record))
Besides saveRecord(record) technically may return a function so it's possible to have such a statement valid it does not seem your case. So you need just
.then(saveRecord)
Another issue is returning nothing from inside saveRecord and fetchUserRecord function as well.
And finally you don't need to return wrappers Promise.resolve from inside .then: you may return just transformed data and it will be passed forward through chaining.
var CloudKit = require('./setup')
//----
var fetchUserRecord = function(emailConfirmationCode){
var query = { ... }
// Execute the query
return CloudKit.publicDB.performQuery(query).then(function (response) {
if(response.hasErrors) {
return Promise.reject(response.errors[0]);
}else if(response.records.length == 0){
return Promise.reject('Email activation code not found.');
}else{
return response.records[0];
}
})
}
//-----
var saveRecord = function(record){
// Update the record (recordChangeTag required to update)
var updatedRecord = { ... }
return CloudKit.publicDB.saveRecords(updatedRecord).then(function(response) {
if(response.hasErrors) {
return Promise.reject(response.errors[0]);
}else{
return Promise.resolve();
}
})
}
//----- Start the Promise Chain Here -----
exports.startActivation = function(emailConfirmationCode){
return CloudKit.container.setUpAuth() //<-- This returns a promise
.then(fetchUserRecord) //<-- This is the 1st function above
.then(saveRecord) //<-- This is the 2nd function above
.catch(function(error){});
}
Don't forget returning transformed data or new promise. Otherwise undefined will be returned to next chained functions.

Since #skyboyer helped me figure out what was going on, I'll mark their answer as the correct one.
I had to tweak things a little since I needed to pass the returned values to subsequent functions in my promise chain. Here's where I ended up:
exports.startActivation = function(emailConfirmationCode){
return new Promise((resolve, reject) => {
CloudKit.container.setUpAuth()
.then(() => {
return fetchUserRecord(emailConfirmationCode)
})
.then((record) => {
resolve(saveRecord(record))
}).catch(function(error){
reject(error)
})
})
}

Adding a Promise to Promise.all() [duplicate]

This question already has answers here:
How to know when all Promises are Resolved in a dynamic "iterable" parameter?
(5 answers)
Closed 6 years ago.
I've got an api call that sometimes returns paged responses. I'd like to automatically add these to my promises so I get the callback once all the data has arrived.
This is my attempt. I'd expect the new promise to be added and Promise.all to resolve once that is done.
What actually happens is that Promise.all doesn't wait for the second request. My guess is that Promise.all attaches "listeners" when it's called.
Is there a way to "reintialize" Promise.all()?
function testCase (urls, callback) {
var promises = [];
$.each(urls, function (k, v) {
promises.push(new Promise(function(resolve, reject) {
$.get(v, function(response) {
if (response.meta && response.meta.next) {
promises.push(new Promise(function (resolve, reject) {
$.get(v + '&offset=' + response.meta.next, function (response) {
resolve(response);
});
}));
}
resolve(response);
}).fail(function(e) {reject(e)});
}));
});
Promise.all(promises).then(function (data) {
var response = {resource: []};
$.each(data, function (i, v) {
response.resource = response.resource.concat(v.resource);
});
callback(response);
}).catch(function (e) {
console.log(e);
});
}
Desired flow is something like:
Create a set of promises.
Some of the promises spawn more promises.
Once all the initial promises and spawned promises resolve, call the callback.

It looks like the overall goal is:
For each entry in urls, call $.get and wait for it to complete.
If it returns just a response without "next", keep that one response
If it returns a response with a "next," we want to request the "next" as well and then keep both of them.
Call the callback with response when all of the work is done.
I would change #2 so you just return the promise and fulfill it with response.
A key thing about promises is that then returns a new promise, which will be resolved based on what you return: if you return a non-thenable value, the promise is fulfilled with that value; if you return a thenable, the promise is resolved to the thenable you return. That means that if you have a source of promises ($.get, in this case), you almost never need to use new Promise; just use the promises you create with then. (And catch.)
(If the term "thenable" isn't familiar, or you're not clear on the distinction between "fulfill" and "resolve", I go into promise terminology in this post on my blog.)
See comments:
function testCase(urls) {
// Return a promise that will be settled when the various `$.get` calls are
// done.
return Promise.all(urls.map(function(url) {
// Return a promise for this `$.get`.
return $.get(url)
.then(function(response) {
if (response.meta && response.meta.next) {
// This `$.get` has a "next", so return a promise waiting
// for the "next" which we ultimately fulfill (via `return`)
// with an array with both the original response and the
// "next". Note that by returning a thenable, we resolve the
// promise created by `then` to the thenable we return.
return $.get(url + "&offset=" + response.meta.next)
.then(function(nextResponse) {
return [response, nextResponse];
});
} else {
// This `$.get` didn't have a "next", so resolve this promise
// directly (via `return`) with an array (to be consistent
// with the above) with just the one response in it. Since
// what we're returning isn't thenable, the promise `then`
// returns is resolved with it.
return [response];
}
});
})).then(function(responses) {
// `responses` is now an array of arrays, where some of those will be one
// entry long, and others will be two (original response and next).
// Flatten it, and return it, which will settle he overall promise with
// the flattened array.
var flat = [];
responses.forEach(function(responseArray) {
// Push all promises from `responseArray` into `flat`.
flat.push.apply(flat, responseArray);
});
return flat;
});
}
Note how we never use catch there; we defer error handling to the caller.
Usage:
testCase(["url1", "url2", "etc."])
.then(function(responses) {
// Use `responses` here
})
.catch(function(error) {
// Handle error here
});
The testCase function looks really long, but that's just because of the comments. Here it is without them:
function testCase(urls) {
return Promise.all(urls.map(function(url) {
return $.get(url)
.then(function(response) {
if (response.meta && response.meta.next) {
return $.get(url + "&offset=" + response.meta.next)
.then(function(nextResponse) {
return [response, nextResponse];
});
} else {
return [response];
}
});
})).then(function(responses) {
var flat = [];
responses.forEach(function(responseArray) {
flat.push.apply(flat, responseArray);
});
return flat;
});
}
...and it'd be even more concise if we were using ES2015's arrow functions. :-)
In a comment you've asked:
Could this handle if there was a next next? Like a page 3 of results?
We can do that by encapsulating that logic into a function we use instead of $.get, which we can use recursively:
function getToEnd(url, target, offset) {
// If we don't have a target array to fill in yet, create it
if (!target) {
target = [];
}
return $.get(url + (offset ? "&offset=" + offset : ""))
.then(function(response) {
target.push(response);
if (response.meta && response.meta.next) {
// Keep going, recursively
return getToEnd(url, target, response.meta.next);
} else {
// Done, return the target
return target;
}
});
}
Then our main testCase is simpler:
function testCase(urls) {
return Promise.all(urls.map(function(url) {
return getToEnd(url);
})).then(function(responses) {
var flat = [];
responses.forEach(function(responseArray) {
flat.push.apply(flat, responseArray);
});
return flat;
});
}

Assuming you are using jQuery v3+ you can use the promises returned by $.ajax to pass to Promise.all().
What you are missing is returning the second request as a promise instead of trying to push it to the promises array
Simplified example
var promises = urls.map(function(url) {
// return promise returned by `$.ajax`
return $.get(url).then(function(response) {
if (response.meta) {
// return a new promise
return $.get('special-data.json').then(function(innerResponse) {
// return innerResponse to resolve promise chain
return innerResponse;
});
} else {
// or resolve with first response
return response;
}
});
})
Promise.all(promises).then(function(data) {
console.dir(data)
}).catch(function(e) {
console.log(e);
});
DEMO