Structuring functions that encapsulate node.js request - javascript

I am making a call to an API within a function. At this point I get "undefined" as the returned value. I know the call to the API is successful since the URLs that I am trying to get in the call print to the term no problem. I am 99% sure that call to the encapsulating function gets triggered before the request function is done ("undefined" is returned before URLs are listed). Wanted to confirm this and ask if there is a tutorial or code snippet someone could point me to with a good description of a pattern I should follow in this case. <-- Obviously still struggling with the async nature of the beast :)
function GetPhotoURLs(url){
var photo_urls= new Array();
request({
url: url,
json: true
}, function (error, response, body) {
if (!error && response.statusCode === 200)
{
//console.log(body) // Print the json response
var inhale_jsonp=body;
var blog_stream= inhale_jsonp.substring(22,inhale_jsonp.length-2); //getting JSON out of the wrapper
blog_stream=JSON.parse(blog_stream);
for(var i=0;i<blog_stream.posts.length;i++)
{
photo_urls[i]=blog_stream['posts'][i]['photo-url-500'];
console.log(photo_urls[i]+"\n"); //checking that I am getting all the URLs
}
console.log("success!");
console.log(photo_urls[1]);
return photo_urls;
}
else
{
photo_urls[0]='none';
console.log("nope!");
console.log(photo_urls[0]);
return photo_urls;
}
});
}
output sequence -->
1. undefined
2. Listing of URLs
3. Success message + second element from URLs array

The request() function is asynchronous. As such, it finishes long after your original function has completed. Thus, you can't return the result from it (the result isn't even known yet when the function returns). Instead, you must process the result IN your callback or call a function from within that callback and pass the result to that function.
As for general design patterns for this type of work, you can either process the result in the callback as suggested above or switch to using promises to help you manage the asynchronous nature of the request. In all cases, you will be processing the result in some sort of callback.
You can read this answer for some more detail on handling async responses. That specific answer is written for client-side ajax calls, but the concepts for handling async responses are identical.
In your specific case, you may want to make getPhotoURLs() take a callback function which you can call with the result:
function GetPhotoURLs(url, callback){
.....
request(..., function(error, response, body) {
...
// when you have all the results
callback(photo_urls);
})
}
GetPhotoURLs(baseurl, function(urls) {
// process urls here
// other code goes here after processing the urls
});

Related

Saving a value through JavaScript's request method

I run into an issue when trying to use the request method in javascript where I can't save a value. I'll run a block of code like:
let savedData;
request({
url: url,
json: true
}, function (err, resp, body) {
if (err) {
return;
}
savedData = body.data;
});
console.log(savedData);
I know that request doesn't block or something, so I think it's run after the console.log or something like that? I just need to know how I can save the desired data for use later in the method.
Your code is working correctly, you're just neglecting the fact that the callback provided as a second parameter to request() is executed asynchronously.
At the time when your console.log() is executed, the network request may or may not have successfully returned the value yet.
Further Explanation
Take a look at the documentation for the request() function.
It states that the function call takes the following signature,
request(options, callback);
In JavaScript, a callback performs exactly as the name perscribes; it calls back by executing the provided function, after it has done what it first needs to.
This asynchronous behavior is especially prominent in making networking requests, since you wouldn't want your program to freeze and wait for the network request to retrieve or send what you've requested.
Example
function callback() {
console.log('I finished doing my asynchronous stuff!');
console.log('Just calling you back as I promised.');
}
console.log('Running some asynchronous code!');
request({...options}, callback);
console.log('Hi! I'm being called since I'm the next line of code; and the callback will be called when its ready.');
Output
Running some asynchronous code!
Hi! I'm being called since I'm the next line of code; and the callback
will be called when its ready.
I finished doing my asynchronous stuff!
Just calling you back as I promised.
You'll either need to do the rest of the code in the callback of the request function, or use a promise. Anything outside of that callback is going to execute before saveedData ever shows up.

module.exports returns an empty object

So,I have the following js files:
test_api.js:
var request = require("request")
//I actually have a url here.API call which returns JSON.
var url = "";
request({
url: url,
json: true
}, function (error, response, body) {
if (!error && response.statusCode === 200) {
module.exports = body;
console.log(body) // Prints the json response
}
});
test.js:
var api = require('./test_api.js');
console.log(api);
So,when I run node test.js I get:
console.log(body) // Prints the json response
console.log(api); //Prints an empty object
Any idea why I get an empty object back?
When you call request(), you pass it a callback function. That callback function is called sometime in the future (that's an asynchronous callback). Meanwhile the rest of your module continues to execute and your module initialize completes with nothing assigned to module.exports yet.
So, when the caller does:
var api = require('./test_api.js');
The module has finished loading and nothing was assigned to module.exports yet so therefore, it is still an empty object and thus api contains only an empty object.
Then, sometime later, your request() operation finishes and calls its callback. You then assign something to module.exports, but it's too late. The module was already loading and the caller already grabbed the old module.exports before you replaced it.
All network I/O in node.js is asynchronous. This means that the completion callback is called some indeterminate time in the future and the rest of your Javascript continues to execute. The only place you can process asynchronous results is inside the completion callback or in some other function that is called from that callback. Or, you can use promises to do that type of work for you.
So, basically you can't return results that were retrieved with asynchronous operations from the loading of a module and you can't assign them to module.exports. So instead, the modern way to design this is to export either a promise or a function that returns a promise and then the caller can use .then() on the promise to get access to the results.
Here would be a modern way to implement what it looks like you're trying to do using a promise.
var request = require("request")
//I actually have a url here.API call which returns JSON.
var url = "";
function requestP(options) {
return new Promise((resolve, reject) => {
request(options, (error, response, body) => {
if (error) {
reject(error);
} else if (response.statusCode !== 200) {
reject(new Error(`Network request returned status code ${response.statusCode}`));
} else {
resolve(body);
}
});
});
}
module.exports = requestP({url, json: true});
Then, the caller would use that like this:
let api = require('./test_api.js');
api.then(body => {
// process body here
}).catch(err => {
// process err here
});
For a more general discussion of returning asynchronous results, see How do I return the response from an asynchronous call?
You cannot assign module.exports or assign to exports asynchronously. Instead, you should consider exporting a function that accepts a callback and performs the request (caching/reusing the result if needed).

Using async.waterfall

I'm using node.js and the async package.
Here's the code I have:
async.waterfall(
[
function(callback) {
var data = getSomeData();
callback(null, data);
},
function(data, callback) {
someFunctionThatNeedsData(data);
callback(null, 'done');
}
],
function(err, result) {
}
);
getSomeData has an asynchronous HTTP request that grabs some data from a web service. I'd like to wait until I get a response, and then return that data and pass it to someFunctionThatNeedsData.
What I expected was that getSomeData -- including the callback inside of it -- would have to complete before moving on to invoke someFunctionThatNeedsData.
The problem is that, despite using the waterfall function here, data is undefined by the time it gets to someFunctionThatNeedsData.
Additionally, from console.log I can see that the end of getSomeData is reached before the callback inside of getSomeData even begins.
Am I using waterfall incorrectly, or is it just not the right tool here? If it's just not right, what can I use to achieve the desired effect?
Or do I have to resign to having deeply nested callbacks (which, with future work, I will) and have to just mitigate it by extracting inline code into named functions?
getSomeData() has an asynchronous http request that grabs some data from a web service.
This is the issue. The execution flow already continued to the callback and executed it. This is how asynchronous functions work!
You have to pass the callback to getSomeData, which calls it once the HTTP request finished. So yes: You may need to nest the callbacks.
If you have async operation. You don't necessary to use async.waterfall. You could just do that in a promise chain style.
getSomeData().then(function(data)
{
var changeData = changeYourData(data);
return changeData;
}).then(function(changedData)
{
// some more stuff with it. You can keep on forwarding to the next `then`
}).catch(function(err)
{
// if any error throw at any point will get catch here
}).finally(function()
{
// this one will guarantee get call no matter what,
// exactly the same like async.waterfall end of chain callback
});
This example will work with Q, When, and any promise lib that follow standard.
If you need to use async.waterfall (because you could drive it with an Array.map)
You just need to callback in your then
async.waterfall(
[
function(callback) {
// A
getSomeData().then(function(data)
{
callback(null, data);
});
// B - just throw the whole thing in
callback(null , getSomeData());
},
function(data, callback) {
// A
someFunctionThatNeedsData(data);
// B
data.then(function(resolvedData)
{
someFunctionThatNeedsData(resolvedData);
callback(null, 'done');
});
}
],
function(err, result) {
});
Hope this help.

Join thread in JavaScript

Probably asked before, but after the serious searching I'm still not able to find a proper solution. Please consider something like this:
function compute() {
asyncCall(args, function(err, result) {
});
/* 'join thread here' */
}
Even though asyncCall is asynchronous I'd like to use the result and return it from the function compute synchronously. asyncCall is a library call and I can't modify it in any way.
How to wait properly for the asynchronous result without setTimeout and watching a conditional variable? This is possible but suboptimal.
not sure how you can really use something that doesn't exist yet, but it's easy enough to return a slot where the result will be:
function compute() {
var rez=[];
asyncCall(args, function(err, result) {
rez[0]=result;
if(rez.onchange){ rez.onchange(result); }
});
/* 'join thread here' */
return rez;
}
now, you can refer to the [0] property of the return, and once the callback comes in, compute()[0] will have the result. It will also fire an event handler you can attach to the returned array that will fire when the data updates inside the callback.
i would use something more formal like a promise or secondary callback, but that's me...
EDIT: how to integrate a callback upstream:
// sync (old and busted):
function render(){
var myView=compute();
mainDiv.innerHTML=myView;
}
//async using my re-modified compute():
function render(){
var that=compute();
that.onchange=function(e){ mainDiv.innerHTML=e; }
}
see how making it wait only added a single wrapper in the render function?
There's no await syntax in browsers that is widely available. Your options are generally limited to Callback patterns or Promises.
NodeJS follows a callback pattern for most async methods.
function someAsyncMethod(options, callback) {
//callback = function(error, data)
// when there is an error, it is the first parameter, otherwise use null
doSomethingAsync(function(){
callback(null, response);
});
}
....
someAsyncMethod({...}, function(err, data) {
if (err) return alert("OMG! FAilZ!");
// use data
});
Another common implementation is promises, such as jQuery's .ajax() method...
var px = $.ajax({...});
px.data(function(data, xhr, status){
//runs when data returns.
});
px.fail(function(err,xhr, status){
//runs when an error occurs
});
Promises are similar to events...
Of the two methods above, the callback syntax tends to be easier to implement and follow, but can lead to deeply nested callback trees, though you can use utility patterns, methods like async to overcome this.

Trouble understanding Node.js callbacks

Today is my first foray into nodejs and I am particularly stumped trying to understand the way the following piece of logic flows. The logic is as follows:
request({ uri: db.createDbQuery('identifier:abcd1234') },
function(err, response, body) {
response.should.have.status(200);
var search = JSON.parse(body);
search.response.numFound.should.equal(1);
done();
});
});
At a higher level I do understand is that an http request is being made and the function is being called at some juncture that is taking the response and doing something to it. What I am trying to understand is the proper order of the calls and how does the binding of variables take place in the above given logic. How does the compiler know how to bind the return values from the request to the anonymous function? Basically, I want to gain an understanding on how things work under the hood for this snippet.
Thanks
Your question isnt specific to node.js, this is basically a feature of javascript.
Basically you are calling request() which is defined like function request(obj, callback)
Internally, the http request is being called, and once its completed, it calls callback which is actually a function pointer.
function request(obj, callback){
//http request logic...
var err = request_logic_internal_function();
var response = ...
var body = ...
callback(err, response, body)
}
Your code can actually be restructured as :
var options = { uri: db.createDbQuery('identifier:abcd1234') };
var request_callback = function(err, response, body) {
response.should.have.status(200);
var search = JSON.parse(body);
search.response.numFound.should.equal(1);
done();
};
request(options, request_callback);
What you're basically doing is sending in a function pointer as a variable.
I don't know what library(ies) you're using, and it looks like you may have anonymized them by assigning methods into your code's global scope like request, done, and db.
What I can say is this:
That indentation is horrible and initially misled me on what it was doing, please gg=G (vim syntax) your code so it's properly indented.
request takes two arguments, a configuration object and a callback.
db.createDbQuery must be a blocking method or the anonymous object you're creating won't have the proper value.
request uses that configuration value, makes a non-blocking I/O request of some kind, and later will call the callback function you provide. That means that the code immediately after that request call will execute before the callback you provide will execute.
Some time later the request data will come back, Node.js's event loop will provide the data to the library's registered event handler (which may or may not be your callback directly -- it could do something to it and then call your event handler afterwards, you don't know or really care).
Then the function does some checks that will throw errors if they fail, and finally calls a done function in its scope (defined somewhere else) that will execute and continue the logical stream of execution.

Categories