//Handling get request from the client
app.get('/watch', function(req, res) {
//A for loop goes here which makes multiple requests to an external API and stores the result in a variable data
console.log(data); //Gives an empty value!!
res.write(data); //Sends an empty value
}
Now when I try to log the data variable DURING the loop, it's value is as expected. However it sends data as an empty variable to the client. I am pretty sure this is because the for loop takes a while to execute and Node being non-blocking moves to the next part of the code. Is there a workaround for this or is something fundamentally wrong with the design of my code?
EDIT: Posting the for loop as requested
for(var num in data.items)
{
url ='/Product';
options={
host:"api.xyz.com",
path:url
};
http.get(options,function(response){
var responseData = "";
response.setEncoding('utf8');
//stream the data into the response
response.on('data', function (chunk) {
responseData+=chunk.toString();
});
//responseData=JSON.parse(responseData);
//write the data at the end
response.on('end', function(){
body=JSON.parse(responseData);
var discount=body.product[0].styles[0].percentOff.replace('%','');
if(discount>=20)
{
discounted_json.disc_items.push({"percentOff":discount});
}
});
});
}
When you want to call multiple asynchronous functions in order, you should call the first one, call the next one in it's callback and so on. The code would look like:
asyncFunction1(args, function () {
asyncFunction2(args, function () {
asyncFunction3(args, function () {
// ...
})
})
});
Using this approach, you may end up with an ugly hard-to-maintain piece of code.
There are various ways to achieve the same functionality without nesting callbacks, like using async.js or node-fibers.
Related
Is it possible, in node.js, to make an asynchronous call that times out if it takes too long (or doesn't complete) and triggers a default callback?
The details:
I have a node.js server that receives a request and then makes multiple requests asynchronously behind the scenes, before responding. The basic issue is covered by an existing question, but some of these calls are considered 'nice to have'. What I mean is that if we get the response back, then it enhances the response to the client, but if they take too long to respond it is better to respond to the client in a timely manner than with those responses.
At the same time this approach would allow to protect against services that simply aren't completing or failing, while allowing the main thread of operation to respond.
You can think of this in the same way as a Google search that has one core set of results, but provides extra responses based on other behind the scenes queries.
If its simple just use setTimout
app.get('/', function (req, res) {
var result = {};
// populate object
http.get('http://www.google.com/index.html', (res) => {
result.property = response;
return res.send(result);
});
// if we havent returned within a second, return without data
setTimeout(function(){
return res.send(result);
}, 1000);
});
Edit: as mentioned by peteb i forgot to check to see if we already sent. This can be accomplished by using res.headerSent or by maintaining a 'sent' value yourself. I also noticed res variable was being reassigned
app.get('/', function (req, res) {
var result = {};
// populate object
http.get('http://www.google.com/index.html', (httpResponse) => {
result.property = httpResponse;
if(!res.headersSent){
res.send(result);
}
});
// if we havent returned within a second, return without data
setTimeout(function(){
if(!res.headersSent){
res.send(result);
}
}, 1000);
});
Check this example of timeout callback https://github.com/jakubknejzlik/node-timeout-callback/blob/master/index.js
You could modify it to do action if time's out or just simply catch error.
You can try using a timeout. For example using the setTimeout() method:
Setup a timeout handler: var timeOutX = setTimeout(function…
Set that variable to null: timeOutX = NULL (to indicate that the timeout has been fired)
Then execute your callback function with one argument (error handling): callback({error:'The async request timed out'});
You add the time for your timeout function, for example 3 seconds
Something like this:
var timeoutX = setTimeout(function() {
timeOutX = null;
yourCallbackFunction({error:'The async request timed out'});
}, 3000);
With that set, you can then call your async function and you put a timeout check to make sure that your timeout handler didn’t fire yet.
Finally, before you run your callback function, you must clear that scheduled timeout handler using the clearTimeout() method.
Something like this:
yourAsyncFunction(yourArguments, function() {
if (timeOutX) {
clearTimeout(timeOutX);
yourCallbackFunction();
}
});
I've got a function on server like this
Meteor.methods({
"functionName": function(data) {
//calculations, http requests etc.
console.log(variable); //shows variable properly
return variable;
}
});
When I do console.log(variable) on server I can see data.
Now I do Meteor.call on client:
Meteor.call('functionName', data, function(err, res) {
console.log(res); //shows undefined
});
What's wrong here? Why can't I use response variable on client?
EDIT:
I've been struggling with this for couple of hours and I narrowed down errors to one, HTTP.get() function. Now it looks like this:
//some calculations
var variable = HTTP.get('url');
return variable.data; //should return an object
I tried to wrap it with Promise but then it didn't work. It looked like this:
//some calculations
function promise() {
var variable = HTTP.get('url');
resolve('Done.');
}
promise().then(function() {
console.log(variable); //doesn't even work on server this way
return variable;
});
Based on the comments and your question, my guess is that your return statement is happening within an async callback of an HTTP request. You can use Meteor.wrapAsync or Promises to make these requests synchronous and get a valid return statement.
The issue is occurring due to synchronous call of http.
What really happen here you function is not waiting for http call response.
You should use Meteor.wrapAsync;
I'll try to keep this as simple as possible. The following function seems to run in a very strange manner. The output I'm getting from my test-prints is in the order of 1, 4, 3, 2 (with an empty returnList).
This seems to suggest that the code within the interior of these code-blocks are getting executed last, which as you might guess creates some problem when I want to return the returnList.
var server = new mongo.Server('localhost', 27017);
var db = new mongo.Db('tdp013', server);
app.get('/getall', function (req, res) {
var returnList = [];
console.log("1");
db.open(function(err, client){
client.collection("messages", function(err, col){
col.find({}, function(err, cursor){
cursor.each(function(err, item){
if(item!=null){
console.log("2");
returnList.push(item);
}
});
console.log("3");
});
});
});
console.log("4");
console.log(returnList);
res.sendStatus(200);
});
My question is simply this, is there a good explanation(in lack of a better word) as to why/how these lines are not getting executed in the written order?
Or alternatively, is there a way to return the returnList without doing it in the iteration?
This is simply the asynchronous nature that node.js further fortifies through the use of JavaScript.
You're trying to read this code and reason through its execution sequentially (i.e. synchronously), but that is not how it works.
console.log('1');
db.open(function (err, db) {
// ...
});
console.log('4');
Take the above snippet, for instance. You expect the function in db.open() to complete before it moves on to write 4 to the console, but that is not the case at all. What happens is that node will asynchronously run that open() method and continue on to the next code (here, that'd be console.log('4');). When that open() method has completed running, it'll kick off the callback function defined in the parameter list.
This is async JavaScript. You can't expect it to run the code in your editor synchronously/sequentially written like this.
Developing logic that will flow the way (it appears) you want it to flow, it would have to be refactored like the following:
var server = new mongo.Server('localhost', 27017);
var db = new mongo.Db('tdp013', server);
app.get('/getall', function (req, res) {
var returnList = [];
console.log("1");
db.open(function(err, client){
client.collection("messages", function(err, col){
col.find({}, function(err, cursor){
cursor.each(function(err, item){
if(item!=null){
console.log("3");
returnList.push(item);
}
else {
console.log("4");
console.log(returnList);
res.sendStatus(200);
}
});
console.log("2");
});
});
});
});
Node.js is asynchronous in nature and runs on single threaded event loop.
Example:
Let’s say you have triggered a database query and you are waiting for query to execute and then proceed further, but in a same time what will happen to the piece of code which are not dependent on that database query, they won’t execute and wait for their turn.
Asynchronous is exactly opposite of this. While you are doing database query Node.js will continue executing code and when database query will be done, it will come back to process it.
So in your case you have added console.log("2"); and console.log("3"); in callback function, so those lines are executing when those method executed. And console.log("4"); will not wait for db.open method to be finished. Hence the results are 1, 4, 3, 2.
So to make sure returnList returning correct data you have to add that in the callback method itself.
Hope this will help you to understand the problem you are facing.
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 8 years ago.
EDIT: I want to explain the problem more clearly, my purpose is to make http response before "webot.waitRule()" returns. So should http request call back to webot.waitRule or the other way around?
below is my code which implements a module in a program, and my problem is
the function returns before the http request get its response. I know http requests in node
js is async, so there is any solution for it？ I am pretty new in javascript and nodejs. Thanks.
webot.waitRule('wait_class', function(info) {
var courseName='lol';
var options = {
host: 'api.uwaterloo.ca',
path: '/v2/courses/CS/486/examschedule.json'
};
callback = function(response) {
var str = '';
//another chunk of data has been recieved, so append it to `str`
response.on('data', function (chunk) {
str += chunk;
});
response.on('end',function(){
// console.log(str);
// return "kkk";
var data = JSON.parse(str);
console.log(data['data']['course']);
courseName = courseName+data['data']['course'];
console.log("finished");
// return "lolllll";
// return "nide ke shi "+ courseName;
});
}
var req = http.request(options, callback);
req.end();
console.log("i am finshed");
return courseName;
});
Since node is primarily asynchronous and uses callbacks (as opposed to promises), you'd likely want to define a function that will be called that takes a parameter which is the output of your asynchronous call.
For example (in your case): you get courseName in the callback for response.on('end', .... You could define a function such as:
function processCourseName(name) {
console.log('do something with', name);
}
And then after console.log('finished');, just add a call to that method
...
courseName = ...
console.log("finished");
processCourseName(courseName); //whenever the request is finished, then process the course name in a separate function
More explanation: Your return courseName statement are exiting from the function before the http.request call is finished, and therefore before courseName is even assigned to a proper value.
More edit: I took a look at the webot module and it looks like its written in a way that doesn't support async code? Usually libraries pass a callback function you can call when your async processing is done, like function(info, callback) {}, but webot does not appear to. Your best bet might be to assign courseName to the info object (info.courseName) you are given and notify your program to use courseName elsewhere.
Notes:
You should use somthing like mikeal/request to make requests outwards, as it's written specifically to do so.
I am using the node module jsonfile to read a json file and route it using the express module and res.json()
To my understanding I cant use the async read because manipulation of the json can only be handled in the callback making it effectively impossible to return the data and serve it using res.json()
app.get('/api/announcements', function(req, res) {
res.json(utils.getAnnouncements())
})
getAnnouncements: function() {
data = jsonfile.readFile('announcements.json', function(err, obj) {
//return obj
})
//return data
}
is what I want but in practice this either returns undefined or a promise depending on the implementation.
Would reading the file synchronously block the execution of the entire server or just the event loop of the app.get('/api/announcements')
Also, what would be the most correct way to go about doing this?
The entire Node process has a single event loop, so reading synchronously would block EVERYTHING. (clustering withstanding...)
You want to do something like:
app.get('/api/announcements', function(req, res) {
//define anonymous function that will be used when getAnnouncements is done.
utils.getAnnouncements(function(err,fileData){
// handle if(err)
res.json(fileData)
})
})
getAnnouncements: function(callback) {
//read file async, using callback function to handle results
jsonfile.readFile('announcements.json', function(err, fileData) {
callback(err, fileData)
})
//or just `jsonfile.readFile('announcements.json',callback)`
}