I am performing an async request to pull data from a server and then call a function after the request. My question is how do I ensure the request is complete and all data loaded before processRecords() runs?
Thanks in advance.
function getRecords () {
var ids = Server.getIds();
var allTheRecords = [];
ids.forEach(function(recordId) {
Server.getRecord(recordId, function (error, data) {
if(error) {
console.log(error);
} else {
allTheRecords.push(data);
};
});
});
processRecords(allTheRecords);
}
How are you performing the Asynchronous request? If it's an AJAX request, the API provides for callbacks to be supplied based on the result of the call.
https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest
You could use the native Promise api to perform the async actions for you.
Using Promise.all you can give it an array of promises that need to be resolved before calling the processRecords function.
It also now more reusable as you have a getRecord function that you could use elsewhere in your code.
You should probably think of a way to add in the ability to get multiple records from the server if you control it though. You don't really want to fire off a bunch of network requests if you can do it in just one.
// Server mock of the api you have shown
const Server = {
getRecord(id, callback) {
console.log('getRecord', id)
callback(null, {id})
},
getIds() {
return [1, 2, 3]
}
}
function getRecords (ids, processRecords) {
console.log('getRecords', ids.join())
// mapping the array of id's will convert them to an
// array of Promises by calling getRecord with the id
Promise.all(ids.map(getRecord))
// then is called once all the promises are resolved
.then(processRecords)
// this will be called if the reject function of any
// promise is called
.catch(console.error.bind(console))
}
function getRecord(recordId) {
// this function returns a Promise that wraps your
// server call
return new Promise((resolve, reject) => {
Server.getRecord(recordId, function (error, data) {
if(error) {
reject(error)
} else {
resolve(data)
}
})
})
}
getRecords(Server.getIds(), function(records) {
console.log('resolved all promises')
console.log(records)
})
Related
Introduction
Hey there,
I am trying to pass out data from the mongoose withTransaction callback. Right now, I am using the following code which implements callbacks:
const transactionSession = await mongoose.startSession()
await transactionSession.withTransaction(async (tSession) => {
try {
// MARK Transaction writes & reads removed for brevity
console.log("Successfully performed transaction!")
cb(null, "Any test data")
return Promise.resolve()
} catch (error) {
console.log("Transaction aborted due to error:", error)
cb(error)
return Promise.reject()
}
})
} catch (error) {
console.log(error)
return cb(error)
}
A more detailed snippet of the withTransaction helper in use can be found here.
A link to the official Mongoose documentation regarding the withTransaction helper can be found here.
At the moment, I am using a callback to pass out data from the withTransactioncallback:
cb(null, "Any test data")
However, the problem is that naturally the callback is executed first, before the Promise.resolve() is returned. This means, that (in my case) a success response is sent back to the client before any necessary database writes are committed:
// this is executed first - the callback will send back a response to the client
cb(null, "Any test data")
// only now, after the response already got sent to the client, the transaction is committed.
return Promise.resolve()
Why I think this is a problem:
Honestly, I am not sure. It just doesn't feel right to send back a success-response to the client, if there hasn't been any database write at that time. Does anybody know the appropriate way to deal with this specific use-case?
I thought about passing data out of the withTransaction helper using something like this:
const transactionResult = await transactionSession.withTransaction({...})
I've tried it, and the response is a CommandResult of MongoDB, which does not include any of the data I included in the resolved promise.
Summary
Is it a problem, if a success response is sent back to the client before the transaction is committed? If so, what is the appropriate way to pass out data from the withTransaction helper and thereby committing the transaction before sending back a response?
I would be thankful for any advice I get.
It looks like there is some confusion here as to how to correctly use Promises, on several levels.
Callback and Promise are being used incorrectly
If the function is supposed to accept a callback, don't return a Promise. If the function is supposed to return a Promise, use the callback given by the Promise:
const transactionSession = await mongoose.startSession()
await transactionSession.withTransaction( (tSession) => {
return new Promise( (resolve, reject) => {
//using Node-style callback
doSomethingAsync( (err, testData) => {
if(err) {
reject(err);
} else {
resolve(testData); //this is the equivalent of cb(null, "Any test data")
}
});
})
Let's look at this in more detail:
return new Promise( (resolve, reject) => { This creates a new Promise, and the Promise is giving you two callbacks to use. resolve is a callback to indicate success. You pass it the object you'd like to return. Note that I've removed the async keyword (more on this later).
For example:
const a = new Promise( (resolve, reject) => resolve(5) );
a.then( (result) => result == 5 ); //true
(err, testData) => { This function is used to map the Node-style cb(err, result) to the Promise's callbacks.
Try/catch are being used incorrectly.
Try/catch can only be used for synchronous statements. Let's compare a synchronous call, a Node-style (i.e. cb(err, result)) asynchronous callback, a Promise, and using await:
Synchronous:
try {
let a = doSomethingSync();
} catch(err) {
handle(err);
}
Async:
doSomethingAsync( (err, result) => {
if (err) {
handle(err);
} else {
let a = result;
}
});
Promise:
doSomethingPromisified()
.then( (result) => {
let a = result;
})
.catch( (err) => {
handle(err);
});
Await. Await can be used with any function that returns a Promise, and lets you handle the code as if it were synchronous:
try {
let a = await doSomethingPromisified();
} catch(err) {
handle(err);
}
Additional Info
Promise.resolve()
Promise.resolve() creates a new Promise and resolves that Promise with an undefined value. This is shorthand for:
new Promise( (resolve, reject) => resolve(undefined) );
The callback equivalent of this would be:
cb(err, undefined);
async
async goes with await. If you are using await in a function, that function must be declared to be async.
Just as await unwraps a Promise (resolve into a value, and reject into an exception), async wraps code into a Promise. A return value statement gets translated into Promise.resolve(value), and a thrown exception throw e gets translated into Promise.reject(e).
Consider the following code
async () => {
return doSomethingSync();
}
The code above is equivalent to this:
() => {
const p = new Promise(resolve, reject);
try {
const value = doSomethingSync();
p.resolve(value);
} catch(e) {
p.reject(e);
}
return p;
}
If you call either of the above functions without await, you will get back a Promise. If you await either of them, you will be returned a value, or an exception will be thrown.
I have a function that pulls an array from firebase via its rest api, which I need to input into another function to create a calendar.
function array_from_firebase(){
//code that pulls from firebase
return array
}
function calendar_create_from_array(array){
//code that generates calendar
}
The following does not work:
calendar_create_from_array(array_from_firebase())
However, this does work
array = array_from_firebase()
setTimeout(calendar_create_from_array,9000,array)
I believe this means that array_from_firebase takes a bit of time longer than calendar_create_from_array and calendar_create_from_array triggers too quickly.
How could I use chaining and promises to solve this?
From the first function you can return a promise. In this example usage of setTimeout is just to demonstrate that the next function execute after the first function
function array_from_firebase() {
return new Promise(
function(resolve) {
setTimeout(function() {
return resolve([1, 2, 3])
}, 4000)
},
function(reject) {})
}
function calendar_create_from_array(array) {
console.log(array);
}
array_from_firebase().then(function(resp) {
calendar_create_from_array(resp)
})
You can also use async & await.In this case the function which will wait for the result from another function need to be declared as async and declare await keyword before the function which will return the result.
function array_from_firebase() {
return new Promise(
function(resolve) {
setTimeout(function() {
return resolve([1, 2, 3])
}, 5000)
},
function(reject) {})
}
async function calendar_create_from_array() {
console.log('Calling function and waiting for result for 5secs....')
let getResult = await array_from_firebase();
console.log('Got result after 5secs', getResult)
}
calendar_create_from_array()
When you work with asynchronous code, you usually deal with Promises, which you have handle with .then and .catch blocks, to force you functions to wait for response. I've simulated the async call inside array_from_firebase function, it returns the response after 3 seconds, and the response is a Promise. Then you can handle this promise via .then block (as shown in calendar_create_from_array function), it will be executed, when you get the response with no errors, and .catch block will be executed when you get the error in response. Something like this:
function array_from_firebase() {
// Simulate async response
return new Promise(function(resolve, reject) {
setTimeout(function() {
resolve([1,2,3]);
}, 3000);
})
}
function calendar_create_from_array() {
array_from_firebase()
.then(function(response) {
console.log(response);
})
}
calendar_create_from_array();
I'm creating an app that will use the https://github.com/vpulim/node-soap to communicate with a soap server.
I would like to create a client component which I will forward the necessary methods to the soap-client created with this module.
I'm having trouble to return an object that will use this client, since the client is created asynchronously.
var soap = require('soap');
var url = 'http://someurl?wsdl';
soap.createClient(url, function(err, client) {
if (err) {
console.log(err);
return;
}
console.log(client.describe());
// I need to publish this client so that other functions in this file will be able to use it
});
module.exports = {
doSomething: function() {
//how can I access the client here?
}
};
How would I go about doing this?
One solution to this problem is to use promises:
var soap = require('soap');
var url = 'http://someurl?wsdl';
var clientPromise = new Promise(function(resolve, reject) {
soap.createClient(url, function(err, client) {
if (err) {
// reject the promise when an error occurs
reject(err);
return;
}
// resolve the promise with the client when it's ready
resolve(client);
});
});
module.exports = {
doSomething: function() {
// promise will wait asynchronously until client is ready
// then call the .then() callback with the resolved value (client)
return clientPromise.then(function(client) {
// do something with client here
}).catch(function(err) {
// handle errors here
console.error(err);
});
}
};
A few advantages to this:
Promises are native JavaScript objects (as of Node 4.0.0, with packages such as bluebird providing support for prior versions)
Promises can be "reused": if clientPromise has already resolved once, it will immediately resolve when doSomething is later called.
Some disadvantages:
doSomething and all other exported functions are inherently asynchronous.
Is not directly compatible with Node-type callbacks.
Not sure if my response helps you, but this is how I do. I create createClient every time and then within the client, I call the actual SOAP method (here GetAccumulators). May be not a great way, but this works for me. Here is my code sample
soap.createClient(url, function (err, client) {
if (err) {
logger.error(err, 'Error creating SOAP client for %s', tranId);
reject('Could not create SOAP client');
}
client.addSoapHeader(headers);
// envelope stuff
client.wsdl.definitions.xmlns.soapenv = 'http://schemas.xmlsoap.org/soap/envelope/';
client.wsdl.definitions.xmlns.acc = 'http://exampleurl/';
client.wsdl.xmlnsInEnvelope = client.wsdl._xmlnsMap();
client.GetAccumulators(args, function (err, result) {
if (err) {
if (isNotFoundError(err)) {
logger.debug('Member not found for tranId %s', tranId);
reject({status:404, description:'No match found'});
}
reject({status:500, description:'GetAccumulators error'});
}
return resolve({data: result, tranId: tranId});
});
In my code below, 5 always prints before 4. I thought because the callback to postUsers was in a return statement from matchAgainstAD it would wait for the for loop and ad lookup to complete before returning. How can I do this in the most simple way?
var matchAgainstAD = function(stUsers) {
stUsers.forEach(function (element, i) {
var sAMAccountName = stUsers[i].guiLoginName;
// Find user by a sAMAccountName
var ad = new ActiveDirectory(config);
ad.findUser(sAMAccountName, function(err, user) {
if (err) {
console.log('ERROR: ' +JSON.stringify(err));
return;
}
if (!user) {
staleUsers.push(stUsers[i])
console.log(4)
}
// console.log(staleUsers);
});
})
return postUsers(staleUsers)
}
var postUsers = function(staleUsers) {
console.log(5);
request.post({
headers: {'content-type' : 'application/x-www-form-urlencoded'},
url: 'http://localhost:8000/api/record/newRecord',
qs: staleUsers
}, function(err, res, body) {
// console.log(body);
})
}
matchAgainstAD();
This is very classic asynchronous problem in node.js. Your findUser() function has an asynchronous response which means the callback is called sometime later. Meanwhile, the rest of your loop continues to run so that all the requests are in-flight at the same time and then the responses start coming in sometime later. Thus, you can't call postUsers() after matchAgainstAd() returns because the inner async operations are not yet completed and thus staleUsers is not yet populated.
There are multiple approaches to solving this problem. In general, it is worth learning how to use promises for operations like this because they offer all sorts of very useful flow of control options when using asynchronous operations and node.js does pretty much all I/O operations as async. But, to best illustrate what's going on in this type of issue, I'll first show you a manually coded solution. In this manually coded solution, you keep track of how many operations are still remaining to complete and when all have completed, then and only then, do you call postUsers() with the accumulated data.
Manually Coded Solution
var matchAgainstAD = function (stUsers) {
var remaining = stUsers.length;
stUsers.forEach(function (element, i) {
var sAMAccountName = stUsers[i].guiLoginName;
function checkDone() {
if (remaining === 0) {
postUsers(staleUsers);
}
}
// Find user by a sAMAccountName
var ad = new ActiveDirectory(config);
ad.findUser(sAMAccountName, function (err, user) {
--remaining;
if (err) {
console.log('ERROR: ' + JSON.stringify(err));
checkDone();
return;
}
if (!user) {
staleUsers.push(stUsers[i])
}
checkDone();
});
});
}
var postUsers = function(staleUsers) {
request.post({
headers: {'content-type' : 'application/x-www-form-urlencoded'},
url: 'http://localhost:8000/api/record/newRecord',
qs: staleUsers
}, function(err, res, body) {
// console.log(body);
})
}
The core logic here is that you initialize a counter to the number of operations that will be carried out. Then, in your loop where each operation is happening, you decrement the remaining counter anytime one of the operations completes (calls it's completion callback). Then, after processing the result (in both the success and error code paths), you check if the remaining count has reached 0 indicating that all requests are now done. If so, then the staleUsers array is now fully populated and you can call postUsers(staleUsers) to process the accumulated result.
Solution Coded Using Bluebird Promises
The idea here is that we use the control flow logic and enhanced error handling of promises to manage the async control flow. This is done by "promisifying" each asynchronous interface we are using here. "promisifying" is the process of creating a small function wrapper around any async function that follows the node.js calling convention where the last argument to the function is a callback that takes at least two arguments, the first an error and the second a value. This can be automatically turned into a wrapper function that returns a promise, allow using to use promise logic flow with any normal async operation.
Here's how this would work using the bluebird promise library.
var Promise = require('bluebird');
var request = Promise.promisifyAll(request('require'));
var matchAgainstAD = function (stUsers) {
var staleUsers = [];
var ad = new ActiveDirectory(config);
// get promisified version of findUser
var findUser = Promise.promisify(ad.findUser, ad);
return Promise.map(stUsers, function(userToSearchFor) {
var sAMAccountName = userToSearchFor.guiLoginName;
return findUser(sAMAccountName).then(function(user) {
// if no user found, then consider it a staleUser
if (!user) {
staleusers.push(userToSearchFor);
}
}, function(err) {
// purposely skip any requests we get an error on
// having an error handler that does nothing will
// stop promise propagation of the error (it will be considered "handled")
});
}).then(function() {
if (staleUsers.length) {
return postUsers(staleUsers);
}
return 0;
});
}
var postUsers = function (staleUsers) {
return request.postAsync({
headers: {
'content-type': 'application/x-www-form-urlencoded'
},
url: 'http://localhost:8000/api/record/newRecord',
qs: staleUsers
}).spread(function (res, body) {
// console.log(body);
return staleUsers.length;
})
}
matchAgainstAD(users).then(function(qtyStale) {
// success here
}, function(err) {
// error here
})
Standard ES6 Promises Version
And, here's a version that uses only the standard ES6 promises built-into node.js. The main difference here is that you have to code your own promisified versions of the async functions you want to use because you can't use the built-in promisify capabilities in Bluebird.
var request = request('require');
// make a promisified version of request.post
function requestPostPromise(options) {
return new Promise(function(resolve, reject) {
request.post(options, function(err, res, body) {
if (err) {
reject(err);
} else {
resolve([res, body]);
}
});
});
}
// make a function that gets a promisified version of ad.findUser
function getfindUserPromise(ad) {
return function(name) {
return new Promise(function(resolve, reject) {
ad.findUser(name, function(err, user) {
if (err) {
reject(err);
} else {
resolve(user);
}
});
});
}
}
var matchAgainstAD = function (stUsers) {
var staleUsers = [];
var promises = [];
var ad = new ActiveDirectory(config);
// get promisified version of findUser
var findUser = getFindUserPromise(ad);
stUsers.each(function(userToSearchFor) {
promises.push(findUser(userToSearchFor.guiLoginName).then(function(user) {
// if no user found, then consider it a staleUser
if (!user) {
staleusers.push(userToSearchFor);
}
}, function(err) {
// purposely skip any requests we get an error on
// have an error handler that does nothing will
// stop promise propagation of the error (it will be considered "handled")
}));
});
return Promise.all(promises).then(function() {
if (staleUsers.length) {
return postUsers(staleUsers);
}
return 0;
});
}
var postUsers = function (staleUsers) {
return requestPostPromise({
headers: {
'content-type': 'application/x-www-form-urlencoded'
},
url: 'http://localhost:8000/api/record/newRecord',
qs: staleUsers
}).then(function (err, results) {
var res = results[0];
var body = results[1];
// console.log(body);
return staleUsers.length;
})
}
matchAgainstAD(users).then(function(qtyStale) {
// success here
}, function(err) {
// error here
})
ad.findUser takes a callback that contains the console.log(4). That function is async, and will hit your callback when the IO operation has completed.
On the other hand, postUsers is called completely synchronously, so it will hit console.log(5) before ad.findUser enters your callback.
A simple way to fix this is to call postUsers from inside of your ad.findUser callback.
I would suggest looking into the promise pattern for JavaScript to manage dependencies between async operations. There are several popular libraries (Q and RSVSP.js being a couple of them).
I am trying to work through JS Promises in node.js and don't get the solution for passing promises between different function.
The task
For a main logic, I need to get a json object of items from a REST API. The API handling itself is located in a api.js file.
The request to the API inthere is made through the request-promise module. I have a private makeRequest function and public helper functions, like API.getItems().
The main logic in index.js needs to wait for the API function until it can be executed.
Questions
The promise passing kind of works, but I am not sure if this is more than a coincidence. Is it correct to return a Promise which returns the responses in makeRequest?
Do I really need all the promises to make the main logic work only after waiting for the items to be setup? Is there a simpler way?
I still need to figure out, how to best handle errors from a) the makeRequest and b) the getItems functions. What's the best practice with Promises therefor? Passing Error objects?
Here is the Code that I came up with right now:
// index.js
var API = require('./lib/api');
var items;
function mainLogic() {
if (items instanceof Error) {
console.log("No items present. Stopping main logic.");
return;
}
// ... do something with items
}
API.getItems().then(function (response) {
if (response) {
console.log(response);
items = response;
mainLogic();
}
}, function (err) {
console.log(err);
});
api.js
// ./lib/api.js
var request = require('request-promise');
// constructor
var API = function () {
var api = this;
api.endpoint = "https://api.example.com/v1";
//...
};
API.prototype.getItems = function () {
var api = this;
var endpoint = '/items';
return new Promise(function (resolve, reject) {
var request = makeRequest(api, endpoint).then(function (response) {
if (200 === response.statusCode) {
resolve(response.body.items);
}
}, function (err) {
reject(false);
});
});
};
function makeRequest(api, endpoint) {
var url = api.endpoint + endpoint;
var options = {
method: 'GET',
uri: url,
body: {},
headers: {},
simple: false,
resolveWithFullResponse: true,
json: true
};
return request(options)
.then(function (response) {
console.log(response.body);
return response;
})
.catch(function (err) {
return Error(err);
});
}
module.exports = new API();
Some more background:
At first I started to make API request with the request module, that works with callbacks. Since these were called async, the items never made it to the main logic and I used to handle it with Promises.
You are missing two things here:
That you can chain promises directly and
the way promise error handling works.
You can change the return statement in makeRequest() to:
return request(options);
Since makeRequest() returns a promise, you can reuse it in getItems() and you don't have to create a new promise explicitly. The .then() function already does this for you:
return makeRequest(api, endpoint)
.then(function (response) {
if (200 === response.statusCode) {
return response.body.items;
}
else {
// throw an exception or call Promise.reject() with a proper error
}
});
If the promise returned by makeRequest() was rejected and you don't handle rejection -- like in the above code --, the promise returned by .then() will also be rejected. You can compare the behaviour to exceptions. If you don't catch one, it bubbles up the callstack.
Finally, in index.js you should use getItems() like this:
API.getItems().then(function (response) {
// Here you are sure that everything worked. No additional checks required.
// Whatever you want to do with the response, do it here.
// Don't assign response to another variable outside of this scope.
// If processing the response is complex, rather pass it to another
// function directly.
}, function (err) {
// handle the error
});
I recommend this blog post to better understand the concept of promises:
https://blog.domenic.me/youre-missing-the-point-of-promises/