I'll try to keep this as simple as possible. The following function seems to run in a very strange manner. The output I'm getting from my test-prints is in the order of 1, 4, 3, 2 (with an empty returnList).
This seems to suggest that the code within the interior of these code-blocks are getting executed last, which as you might guess creates some problem when I want to return the returnList.
var server = new mongo.Server('localhost', 27017);
var db = new mongo.Db('tdp013', server);
app.get('/getall', function (req, res) {
var returnList = [];
console.log("1");
db.open(function(err, client){
client.collection("messages", function(err, col){
col.find({}, function(err, cursor){
cursor.each(function(err, item){
if(item!=null){
console.log("2");
returnList.push(item);
}
});
console.log("3");
});
});
});
console.log("4");
console.log(returnList);
res.sendStatus(200);
});
My question is simply this, is there a good explanation(in lack of a better word) as to why/how these lines are not getting executed in the written order?
Or alternatively, is there a way to return the returnList without doing it in the iteration?
This is simply the asynchronous nature that node.js further fortifies through the use of JavaScript.
You're trying to read this code and reason through its execution sequentially (i.e. synchronously), but that is not how it works.
console.log('1');
db.open(function (err, db) {
// ...
});
console.log('4');
Take the above snippet, for instance. You expect the function in db.open() to complete before it moves on to write 4 to the console, but that is not the case at all. What happens is that node will asynchronously run that open() method and continue on to the next code (here, that'd be console.log('4');). When that open() method has completed running, it'll kick off the callback function defined in the parameter list.
This is async JavaScript. You can't expect it to run the code in your editor synchronously/sequentially written like this.
Developing logic that will flow the way (it appears) you want it to flow, it would have to be refactored like the following:
var server = new mongo.Server('localhost', 27017);
var db = new mongo.Db('tdp013', server);
app.get('/getall', function (req, res) {
var returnList = [];
console.log("1");
db.open(function(err, client){
client.collection("messages", function(err, col){
col.find({}, function(err, cursor){
cursor.each(function(err, item){
if(item!=null){
console.log("3");
returnList.push(item);
}
else {
console.log("4");
console.log(returnList);
res.sendStatus(200);
}
});
console.log("2");
});
});
});
});
Node.js is asynchronous in nature and runs on single threaded event loop.
Example:
Let’s say you have triggered a database query and you are waiting for query to execute and then proceed further, but in a same time what will happen to the piece of code which are not dependent on that database query, they won’t execute and wait for their turn.
Asynchronous is exactly opposite of this. While you are doing database query Node.js will continue executing code and when database query will be done, it will come back to process it.
So in your case you have added console.log("2"); and console.log("3"); in callback function, so those lines are executing when those method executed. And console.log("4"); will not wait for db.open method to be finished. Hence the results are 1, 4, 3, 2.
So to make sure returnList returning correct data you have to add that in the callback method itself.
Hope this will help you to understand the problem you are facing.
Related
This question already has answers here:
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
(7 answers)
Closed 4 years ago.
I'm trying to send back results data from my API call in Node. I have an HTML form with a search bar set up that takes the search term and fires off a search with that term.
Right now the data gets sent back and is empty on the first click, but when I click again, the data from the previous call is sent back to my HTML page, so I know there's something going on in regards to callbacks. I could use a setTimeout on the res.json() but that seems a little...juvenile.
I am pretty new to Node and I know the solution is rather simple...but I can't quite seem to figure it out.
Promises, callbacks, async. Maybe I'm just a little confused on where they go...
var result;
var Questrade = require("questrade");
var qt = new Questrade("env.KEY")
qt.account = "env.ACCOUNT";
qt.on("ready", function(err) {
if (err) {
console.log(err);
}
});
function searchQt() {
qt.search(searchTerm, function(err, symbols) {
if (err) {
console.log(err);
}
result = symbols;
});
}
app.get("/search", function(req, res) {
searchTerm = req.query.symbol;
searchQt();
res.json(result);
});
I'm expecting to receive my data back on time.
Easiest and most incremental solution would be to move res.json(result) to the callback where the value of result is received.
function searchQt(res) { // <-- note added argument
qt.search(searchTerm, function(err, symbols) {
if (err) {
console.log(err);
}
result = symbols;
res.json(result); // here it is!
});
}
app.get("/search", function(req, res) {
searchTerm = req.query.symbol;
searchQt(res); <-- note added argument
// res.json(result); <-- commented out and moved above
});
You could also make things work with promises or async/await, but since you're already using callbacks, this is the most incremental way to get to working code.
While you're in there, you should remove result from the outer scope. You don't need or want that value preserved between requests. So just res.json(symbols); and get rid of the results variable entirely.
And passing res as an argument like that above is a bit of a code smell. Let's move the call to qt.search() directly into the callback for app.get() to clean things up a tiny bit. This eliminates the searchQt() function entirely:
app.get("/search", function(req, res) {
searchTerm = req.query.symbol;
qt.search(searchTerm, function(err, symbols) {
if (err) {
// handle error here somehow. maybe something like this?
console.log(`An error occurred: ${err}`);
res.status(500).end('An error occurred');
return;
}
res.json(symbols);
});
});
There are still some other improvements possible here. (I'm not sure if the ready event needs to fire before a search is tried, for example. So there might be a race condition there.) Promises or async/await might make this more readable. (People seem to like those better than deeply-nested callbacks. I personally don't mind the nesting, but I get it.) But that hopefully gets you going in the right direction. (And hopefully I didn't add any bugs that weren't there before!)
I am trying to write a script that will query a local database and send the results to a Google Spreadsheet. Where I am currently stuck and looking for help is how do I get the results of my query.
var myQuery = [];
function runQuery(query, callback) {
client.connect(function (err) {
if (err) throw err;
client.query(query, function (err, result) {
if (err) {
console.log(err);
}
callback({ result: result });
client.end(function (err) {
if (err) throw err;
});
});
});
}
runQuery('SELECT item_number FROM item WHERE item_number LIKE \'11-%\' LIMIT 10', function (resultsObject) {
myQuery = JSON.stringify(resultsObject.result.rows);
console.log('Did stringify work?');
console.log(myQuery);
});
console.log(myQuery);
My output:
Info: Start process (3:46:11 PM)
[]
Did stringify work?
[{"item_number":"11-0011"},{"item_number":"11-0012"},{"item_number":"11-1255"},{"item_number":"11-0052"},{"item_number":"11-0060"},{"item_number":"11-1256"},{"item_number":"11-1281"},{"item_number":"11-0659"},{"item_number":"11-0660"},{"item_number":"11-0054"}]
Info: End process (3:46:12 PM)
I think I understand what's happening, the scope of the variable myQuery is set and printed within the runQuery function just fine, but outside of that function it's not being actually set. How do I get around this?
Very new to JavaScript and NodeJS so I hope I'm using the correct terminology.
Thanks in advance!
The empty myQuery has already been consoled before you actually put values into it. This issue is caused by Javascript's Asynchronous mechanism. If you want to do something with the result, you have to use a callback function. Just like what you did when you accessed result from runQuery.
Another way to do it is promise. You can return a promise in some function and use the promise to access result.
If you decide to use promise, here is the code:
function getresult(parameters) {
return new Promise((resolve, reject) => {
runQuery(parameter, resolve, reject);
});
}
Using this function, you passed resolve as callback to runQuery. You do want to add another err callback function to catch err in runQuery. In promise you use reject as err callback.
When you want to do something with the result:
getresult(parameters)
.then((result) => {put your code here. result variable is your result})
.catch((err) => {do whatever with err});
This is not a scope issue. It is a timing issue. The query takes some time to execute. JavaScript is 'non-blocking' meaning it will start an asynchronous process (like doing a DB query - really virtually all I/O operations) and return the result later. In your code you are using a callback that is run once the data becomes available.
So, with that in mind, the way to think about your code is that you are doing things in this order:
1) declaring your function runQuery - this happens first since in JavaScript function declarations are moved to the top of the script they are in (a process called hoisting)
2) your variable myQuery is established and initialized (to an empty array)
3) your query is kicked off by a call to runQuery; this only starts the process
4) your console.log(myQuery) at the end of your script is run
5) some time later the query completes and your callback is called with the data where your other console.log() shows the returned data
I have an event triggering a Metor.call():
Meteor.call("runCode", myCode, function(err, response) {
Session.set('code', response);
console.log(response);
});
But my runCode function inside the server's Metheor.methods has inside it a callback too and I can't find a way to make it return something to response in the above code.
runCode: function(myCode) {
var command = 'pwd';
child = exec(command, function(error, stdout, stderr) {
console.log(stdout.toString());
console.log(stderr.toString());
// I Want to return stdout.toString()
// returning here causes undefined because runCode doesn't actually return
});
// I can't really return here because I don't have yet the valuer of stdout.toString();
}
I'd like a way to have the exec callback return something as runCode without setInterval which would work, but as a hacky way in my opinion.
You should use Future from fibers.
See docs here : https://npmjs.org/package/fibers
Essentially, what you want to do is wait until some asynchronous code is run, then return the result of it in a procedural fashion, this is exactly what Future does.
You will find out more here : https://www.eventedmind.com/feed/Ww3rQrHJo8FLgK7FF
Finally, you might want to use the Async utilities provided by this package : https://github.com/arunoda/meteor-npm, it will make your like easier.
// load future from fibers
var Future=Npm.require("fibers/future");
// load exec
var exec=Npm.require("child_process").exec;
Meteor.methods({
runCode:function(myCode){
// this method call won't return immediately, it will wait for the
// asynchronous code to finish, so we call unblock to allow this client
// to queue other method calls (see Meteor docs)
this.unblock();
var future=new Future();
var command=myCode;
exec(command,function(error,stdout,stderr){
if(error){
console.log(error);
throw new Meteor.Error(500,command+" failed");
}
future.return(stdout.toString());
});
return future.wait();
}
});
I wrote an example code following.
//test.js
var fs = require('fs');
fs.readdir(__dirname+'/app/img/', function(err, files) {
console.log(files);
});
someFunction(function(output){
console.log(output);
});
console.log('This is the final console call');
function someFunction(callback) {
callback(' I am some funciton. ');
}
Then, after runing 'node test.js', I got following result.
$ node test2.js
I am some funciton.
This is the final console call
[ 'photo1', 'photo2' ]
someFunction() is created by myself for simulating the callback function in fs.readdir().
And as my original assumption, the result should be
[ 'photo1', 'photo2' ]
I am some funciton.
This is the final console call
or
This is the final console call
[ 'photo1', 'photo2' ]
I am some funciton.
But now, logs of fs.readdir() and someFunction() appear before and after the log of final call log seperately.
hmmm.... I totally can't figure out it.
Therefore, my problem is that:
Why is the log from fs.readdir() is the last log?
Is it jsut because that fs.readdir() needs more time to execute?
Or, it's that I'm lack of some important concept about this issue?
Thanks you~!
The whole point of the async NodeJS interfaces is that you can keep running your program without needing to wait for long-latency functions such as readdir to return the results. You call eraddir to tell it to start processing, continue running your program and when readdir its done it will arrange to call the callback function you passed. If you call more than one async function at the same time there is no guarantee as to which one will finish first.
If you wanted all those console.logs to functions to execute sequentially you would need to nest the callbacks
var fs = require('fs');
function someFunction(callback) {
callback(' I am some funciton. ');
}
fs.readdir(__dirname+'/app/img/', function(err, files) {
console.log(files);
someFunction(function(output){
console.log(output);
console.log('This is the final console call');
});
});
A way with slightly better latency is to do all the calls at once and then wait until everything is done to print the results.
// receives an array of computations to run in parallel
// calls a callback with an array with the results when all computations are done
// each computation is a function that takes a return callback as its first parameter.
function parallel(computations, callback){
var results = [];
var n = 0;
computations.forEach(function(comp, i){
comp(function(result){
n++;
results[i] = result;
if(n === computations.length){
callback(results);
}
});
});
}
var fs = require('fs');
function someFunction(callback) {
callback(' I am some funciton. ');
}
parallel([
function(cb){
fs.readdir(__dirname+'/app/img/', function(err, files) {
cb(files);
});
},
function(cb){
someFunction(function(output){
cb(output);
});
},
], function(results){
console.log(results[0]);
console.log(results[1]);
console.log("This is the final console call");
});
I wouldn't recommend coding this kind of parallel synchronization by hand though. There are many async programming libraries out there to make the process more pleasant that offer this and much more.
The fs.readdir function is executed async while the other two are not. This means that the function in fs.readdir will be executed later when the all the directory has been read(which could take a while).
The other two statements are executed sync which means they will run directly. Therefor they are printed before the files list.
I'm building a very simple client/server app using node.js and redis for the first time. After successfully firing up my redis client and my http server, I'm trying to do a simple SET/GET with my redis client.
I first do:
client.set('apple', 10, redis.print);
which returns a Reply:Ok.
Immediately after, I execute:
client.get('apple', function(err, reply) {
count = parseInt(reply);
return count;
});
Strangely, upon printing out count, I get undefined. But, if I use redis.print, such as:
client.get('apple', redis.print);
A Reply: 10 is returned in the console. 10 also shows up if I do a console.log(count).
I thought maybe I was misusing the set functions in the module, but after referring to these both, I am unsure of what's causing my errors:
node_redis - github readme
redis.io API - GET
upon printing out count, I get undefined.
client.get('apple', function(err, reply) {
count = parseInt(reply);
return count;
});
console.log(count); // is this where you get undefined?
// count at this point is, in fact, undefined,
// because it's set in a callback which was not yet fired.
I don't program in node.js, so I'm not sure about benefits of returning count from the callback. Where would it return it to? Isn't it supposed to be something like this?
client.get('apple', function(err, reply) {
count = parseInt(reply);
// do something with the count here. Print it or whatever.
});
Or you could trap it in a closure, but then you'll need to somehow wait for callback to return and set the variable. How it can be done - I have no idea.
var count = null;
client.get('apple', function(err, reply) {
count = parseInt(reply);
});
// TODO: wait for callback
console.log(count);
This is asynchonous, meaning the rest of the code is executed while waiting for the callback.
if you want to check if your set worked just print it :
client.get('apple', function(err, reply) {
console.log(reply);
});