Im building an app with Nodejs. Im fairly fluent with Front end javascript where asynchronous events rarely get too complex and don't go that deep. But now that I'm using Node which is all event driven, making a lot of calls to different servers and databases that all rely on each other becomes rather clustered.
It seems to be common place to have a next() function passed as a parameter that gets called once the first event has finished. This works great however I'm struggling to keep readable code when needing to have next functions after next functions.
Let me explain through example.
Lets say I have a route defined like so:
app.use('/fetchData', function(req, res) {
});
So before we can return the data I need to make a few async calls.
First to the database to retrieve login details.
Then using the login details i need to make another call to an external server to login in and retrieve the raw information.
Then third I need to go back to the database to do some checks.
And then finally return the data to the user.
How would you do that? Im trying like this but cant get it right nor looking readable:
app.use('/fetchData', function(req, res) {
//First I create a user object to pass information around to each function
var user = {...};
var third = database.doSomeChecks;
var second = server.externalCall(user, third);
//first
database.getLoginDetails(user, second);
});
Obviously second actually runs the function and sets second as the returned value. But I can seem to pass the right information through to second.
One Option i thought could be to pass through an array of callbacks and to always call the last function in the array and remove it.
app.use('/fetchData', function(req, res) {
//First I create a user object to pass information around to each function including the req and res object to finally return information
var user = {...};
var third = database.doSomeChecks;
var second = server.externalCall;
//first
database.getLoginDetails(user, [third, second]);
});
What are your techniques? Is the array idea as pointed out above the best solution?
I'd recommend you to use promises as a personal preference I like to use bluebird it's easy to implement, it has a very nice performance and also it has some cool features to play with.
with promises, it's easier to read the control flow execution (at least to me), a lot of people complain about the callback hell and promises it's one of the possible solutions.
you can do something like this:
from:
var user = {...};
var third = database.doSomeChecks;
var second = server.externalCall(user, third);
to:
var user = {...};
checkDB(query).then(
function(data){
//data checks
return value
}).then(
function(value){
// value returned from the previous promise
return server.externalCall(value);
});
you can take a look to this answer and see how you can deal with nested promises which are far easier than callbacks.
I hope that helps.
Related
I looked over the internet looking for an answer but I didn't find.
I studied a bit about the event loop thanks to this https://www.youtube.com/watch?v=8aGhZQkoFbQ amazing video.
still my problem is not solved. I tried with promises and callback but my problem is over that. because I call a function that inside there is asynchronous function so if I put the callback, it will wait for the end of the function not the asynchrounous function within it.
So I ask you help. I beg help XD I am getting know more and more javascript so thats my question:
I have a nodejs express angular app. I have a form in the front end. I make a post request with this form to my node server than I grab this data I process it and I start to make 6 request to a server. at the end of this 6 request I need to send back the data to the front end so now I tell you what I dont get
app.post('/api/todos', function(req, res) {
var from = (new Date(req.body.from)).getTime();
var to = (new Date(req.body.to)).getTime();
var partner = req.body.partner;
var POS = req.body.POS;
Request.request(from,to,partner,POS);
res.send("data")
}
this is about my handling of the form as you can see the problem is that when I call "Request.request" even if I put a call back will it will finish before the data come since "request.request" is a synchronous function and all the 6 request are asynchronous.
I give you an hint of are are made my requests its quite a lot of once but is easier if I sum it.
Request.request(){
res.end(){
savedata**;
if (recursionIndex<y)
indexRequest();
else
bodyRequest()
} //async request
req.end(){}
}
basically once you define the number of time you want to repeat the function you are done. Now what I need is to wait till the end of everything and than give back the whole data to angular
bodyRequest(){
// Same logic of before I didnt use any callback I just used recursion
}
now my question is since I am just using recursion how can I make the whole thing async how can I wrap up everything and give back the data once the last recursion happen?
Guys I got the answer,
app.post('/api/todos', function(req, res) {
var from = (new Date(req.body.from)).getTime();
var to = (new Date(req.body.to)).getTime();
var partner = req.body.partner;
var POS = req.body.POS;
Request.request(from,to,partner,POS,res);
//res.send("data")
}
I pass "res" to my Request.request and through all the subfunction and the trick is done!
since res is an object that possesses the function to answer the POST request coming from angular doesn't matter when you call it but you need the object.
thanks everyone for their attention!
I’m creating a Facebook game using HTML and Javascript, and I’ve just finished building a leaderboard table which lists every player’s name and rank number. This table is populated with data returned from Facebook's game scores API.
This is working perfectly, but I also want to reward players for improving their rank in the table.
This is how I plan to do this:
When the game loads, I run a function called updateTable();, this
populates the leaderboard with the scores and ranks of the players
received from an API call to Facebook's database.
When the player starts to play the game, I store a copy of their rank inside a separate hidden div.
When the game ends, if the player has achieved a new high score, then
it gets entered into the database. After this happens, I run
updateTable(); again to update the leaderboard.
I then run a function called compareRanks();, this compares the
player’s new rank with the rank that I’ve stored in the hidden div.
If the new ranking is a lower number than the stored rank, then they’ve moved up the leaderboard and I reward them 100 coins for every place they move up.
For example:
Player A starts the game and is ranked 5th (so “5” gets stored in a hidden div).
When Player A finishes the game, the leaderboard is updated, and Player A is now ranked 2nd (so the player has jumped 3 places).
To work out what the reward should be, I want to subtract the first variable from the second (5-2 = 3), Player A overtook 3 other players, so their reward will be 3 x 100 gold coins.
The problem I’m having is that when I run compareRanks();, the new rank keeps showing up as the same number as the stored rank, even though I know that the player has improved their rank.
I’m pretty sure this is due to the new rank being grabbed before updateTable(); has fully interacted with the database. I’ve tested this by separating the functions, by making compareRanks(); run on click of a button, when I did this, I completed a game, improved my rank, waited a few seconds after updateTable(); ran, then clicked the button, and the two ranks showed up differently, which is correct. So I think compareRanks(); just needs to wait for updateTable(); to fully complete before it runs.
This is how my functions are laid out:
updateTable(){
//code here interacts with the database/makes a call using Facebook's API,
//and populates the leaderboard table with the returned data
}
On start of a new game, the player’s current rank is stored in the hidden div.
When the game completes updateTable(); is run again, followed by compareRanks();:
compareRanks(){
//code here grabs the stored rank from the hidden div
//code here grabs the newly updated rank and compares the two.
}
I’ve read answers about using callbacks, but I couldn’t get them to work. And I’ve tried doing something like this:
updateTable(){
{
//code here interacts with the database/makes a call using Facebook's API,
//and populates the leaderboard table with the returned data
}
compareRanks();
};
But the new rank is still showing up the same as the old rank when compareRanks(); runs. updateTable(); is changing the ranks correctly on the leaderboard when it runs, so I think compareRanks(); is just running before updateTable(); fully completes.
I’d really appreciate any help in fixing this problem, thank you in advance!
A good way of approaching this would be the use of Javascript Promises. They allow you to do async stuff without nesting multiple callback functions.
function first (parameter){
return new Promise(function(resolve,reject){
//Do async stuff, maybe some ajax
//When async stuff is finished:
resolve(async_data);
//Or when something went wrong:
reject(error);
}
}
function second(parameter){
return new Promise(function(resolve,reject){
//Do async stuff, maybe some ajax
//When async stuff is finished:
resolve(async_data);
//Or when something went wrong:
reject(error);
}
}
//You can then use:
first(data).then(second(async_data)).then(function(async_data){
//Here would be the point where both functions finished after eachother!
}).catch(function(error){
//Hey, one of my promises was rejected! Maybe I should handle that error :)
});
This comes with a few advantages. You can put as many functions and operations as you want into that chain of .thens without nesting big amounts of callback functions. You can also access the reject() call by using .catch(). You should concider reading the docs for Promises as there are many more features that should be interesting for you.
If you don't want to get involved with Promises (They make your code alot cleaner because they're composable so you can create very clear chains of promises) you can look into some of the other awnsers that work with Callbacks (Not that bad for such a small usecase).
Here is a great article about it: Article: JavaScript Promises
Basically a callback is a function passed in as a parameter to another function. JavaScript can do this because functions are first-class objects.
Now, because updateTable will call a db/FB API, you need to invoke the callback within the callback for that operation. I don't know the correct syntax for that operation, so my example uses pseudocode.
function updateTable(callback) {
FBAPI.get(something, function (data) {
// do things
callback();
});
}
updateTable(compareRanks);
Note, if compareRanks needs access to the data from the API you would pass in the data to the callback too:
callback(data);
Better go with new Promise object of javascript-
The Promise object is used for deferred and asynchronous computations. A Promise represents an operation that hasn't completed yet, but is expected in the future.
new Promise(executor);
new Promise(function(resolve, reject) { ... });
Check this link for more help-
https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Promise
The comment from Aless80 holds the key to your answer I think. I don't know how the Facebook API looks like, but whenever you interact with a database or web-server like that it is common for the API call to have a callback in which you can handle anything the server might return to you.
For instance, a simple web-server I am working on handles requests send via AJAX calls from a browser. I am using Jquery to do this, but the structure should be roughly the same.
var dataObject = {};
var dataID = "1234";
$.ajax({
type: "GET", // basic http get request
url: "http://www.mywebserver.com/get-data/" + dataID,
crossDomain: true
}).done(function(response){
// this function runs after the webserver has received our GET request and handled it
// response contains the data we got back from mywebserver
dataObject = response;
});
DoSomething(dataObject);
What happens here is that "DoSomething()" will fire before the dataObject contains any data returned by the database! So if we want to do something with the returned data we should call the function in the "callback" of our ajax request, like so:
var dataObject = {};
var dataID = "1234";
$.ajax({
type: "GET", // basic http get request
url: "http://www.mywebserver.com/get-data/" + dataID,
crossDomain: true
}).done(function(response){
//dataObject = response;
//DoSomething(dataObject);
DoSomething(response);
});
The commented out stuff in this example is for clarity, of course needlessly passing around variables is something we'd like to avoid :)
I also highly recommend looking into JavaScript callbacks. They are tricky to grasp at first but Node.js is basically build upon the concept so it is well worth becoming familiar with them.
I have the following setup, and I'm curious if this is the correct way to do it. It works correctly, but I'm just making sure that I'm doing it right, or if there is a better way to accomplish the same task.
//custom ajax wrapper
var pageLoadPromise = ajax({
url: //call to webmethod
});
//this is the way I have been doing it
pageLoadPromise.done(cB1)
.done(cB2)
.done(cB3)
.done(cB4)
.done(function(){cB5(args);});
//this function requires that cB1 has been completed
//I tried this and it worked as well
pageLoadPromise.done(cB1,cB2,cB3,cB4)
.done(function(){cB5(agrs)});
Doing it both ways works, but like I said, am I wondering if it is this the correct way to accomplish this?
UPDATE:
I have made a small adjustment to my code, specifically for cB1 and the callback to cB5
pageloadPromise.done(
function(data){
cB1(data).done(function(){
cB5(args);
});
},cB2,cB3,cB4
);
function cB1(data){
var cB1Promise = $.Deferred();
...
cB1Promise.resolve();
return cB1Promise;
}
As pointed out by #Bergi, regardless of how you add the callbacks, they are all run in the order they are attached using done. So, promise.done(cb1, cb2, cb3, cb4).done(cb5) is the same as promise.done(cb1).done(cb2).done(cb3).done(cb4).done(cb5).
To make sure cb5 runs after cb1 use:
promise.done( function(data) {cb1(data).done(cb5);}, cb2, cb3, cb4);
Remove data if you don't need it.
I played around with the scenarios in http://jsbin.com/moqiko/4/edit?js,console,output
Doing it both ways works
Yes, they are pretty much equivalent (except for the .done(function(){cB5}); which doesn't work).
I am wondering if it is this the correct way to accomplish this?
Use the one you like better. This is more a design question than one of "correctness". However, both ways look quite odd in my eyes, and I've seen lots of promise code. I would recommend two different structures, depending on how your app is structured:
You use the pageLoadPromise as a global cache for your initial data. It is then consumed in very different places, possibly at different times, for multiple different things (or maybe even repeatedly for the same thing). Then use pageLoadPromise repeatedly in each module:
var pageLoadPromise = ajax({url: …}); // initialisation
pageLoadPromise.done(cB1); // somewhere
…
pageLoadPromise.done(cB2); // somewhere else
…
pageLoadPromise.done(cB3); // other place or time
…
You use the pageLoadPromise in one place only, and want to basically do one thing when it's loaded, except that it is structured in multiple subtasks; and each needs only a part of, not the whole structure. Then use a single callback only:
ajax({url: …}).then(function(data) {
cb1(data.d.cb1data);
cb2(data.d.cb2data);
cb3(data.d.cb3data);
cb4(data.d.cb4data);
cb5(data.d.cb5data, some_additional_data);
});
I have made a small adjustment to my code, specifically for cB1 and the callback to cB5
You should not make cb1 return a promise when it doesn't do anything asynchronous. Don't modify it. If you want to express explicitly that cb5 needs to be executed with the result of cb1, then you should use .then for chaining:
var pageLoadPromise = ajax({url: …}); // initialisation
var cB1promise = pageLoadPromise.then(cB1);
cB1promise.done(cb5); // does get called with the return value of cB1
or
ajax({url: …}).then(function(data) {
var res1 = cb1(data.d.cb1data);
…
cb5(data.d.cb5data, some_additional_data, res1);
});
Update. Thanks to #Bergi who pointed out that jQuery's done() returns in fact the same promise. I've updated the answer based on that.
If cB2,cB3,cB4 are not interconnected and all of them process the same data from the ajax call, then you can add them to the same promise (pageloadPromise).
With the above assumption in mind, your second version of code can be simplified without involving a new promise to be created in cB1(), and without having to go one extra indentation level:
pageloadPromise.then(cB1).done(cB5);
pageloadPromise.done(cB2, cB3, cB4);
function cB1(data){
// ...
//data2 would be the argument value passed when resolving
// your original cB1Promise
return data2;
}
What happens here is that the .then() call creates a new promise that gets resolved with whatever data cB1 returns, allowing cB5 to receive that data without creating an extra callback and without involving another promise (as we already have one in hand).
However if cB1 needs another ajax then your original implementation of cB1 would be more appropriate (the callback scheduling remains the same though).
And one final note, I didn't noticed any failure handlers, in case the ajax call fails.
My userlist table in mongo is set up like this:
email: email#email.com, uniqueUrl:ABC, referredUrl: ...
I have the following code where I query all of the users in my database, and for each of those users, find out how many other's users' referredUrl's equal the current user's unique url:
exports.users = function(db) {
return function(req, res) {
db.collection('userlist').find().toArray(function (err, items) {
for(var i = 0; i<= items.length; i++) {
var user = items[i];
console.log(user.email);
db.collection('userlist').find({referredUrl:user.uniqueUrl}).count(function(err,count) {
console.log(count);
});
}
});
};
};
Right now I'm first logging the user's email, then the count associated with the user. So the console should look as such:
bob#bob.com
1
chris#chris.com
3
grant#grant.com
2
Instead, it looks like this:
bob#bob.com
chris#chris.com
grant#grant.com
1
3
2
What's going on? Why is the nested query only returning after the first query completes?
Welcome to asynchronous programming and callbacks.
What you are expecting to happen is that everything works in a linear order, but that is not how node works. The whole subject is a little too broad for here, but could do with some reading up on.
Luckily the methods invoked by the driver all key of process.nextTick, which gives you something to look up and search on. But there is a simple way to remedy the code due to the natural way that things are queued.
db.collection('userlist').find().toArray(function(err,items) {
var processing = function(user) {
db.collection('userlist').find({ referredUrl: user.uniqueUrl })
.count(function(err,count) {
console.log( user.email );
console.log( count );
});
};
for( var i = 0; i < items.length; i++) {
var user = items[i];
processing(user);
}
});
Now of course that is really an oversimplified way of explaining this, but understand here that you are passing parameters through to your repeating .find() and then doing all the output there.
As said, fortunately some of the work is done for you in the API functions and the event stack is maintained as you added the calls. But mostly, now the output calls are made together and are not occurring within different sets of events.
For a detailed explanation of event loops and callbacks, I'm sure there a much better ones out there than I could write here.
Callbacks are asynchronous in node.js. So, your count function (function(err,count) { console.log(count); }) is not executed immediately after console.log(user.email);. Therefore, the output is normal, nothing wrong with it. What the wrong is the coding style. You shouldn't call callbacks consecutively to get same result when you call functions in same manner in python (in single thread). To get desired result, you should do all work in single callback. But before doing that, I recommend you to understand how callbacks work in nodejs. This will significantly help your coding in nodejs
I've been playing with jsdom, which is a module for node.js. This following code on the bottom is from their documentation page. My problem is how to return something from asynchronous functions.
I know that this is a question that is asked a lot, probably by me as well. I also know that callbacks are a good friend when it comes to these kind of problems. My goal here is to find a workaround which might act like a cookie or a Session variable in PHP, in order to transfer that little bit of data to the outer scope, outside the asynchronous function. Then it should be accessible once the data is set from the outer scope.
First thing I want to know is:
Is there already a way to store data somewhere like a cookie or session that exists in the outer scope and accessible once I've done what I had to do?
If I was to write the data to a file, in point B in the code, and read it at point C, wouldn't I have to write a some sort of timeout function to wait for a few seconds before reading the file? My experience in working with Asynchronous functions in nodejs has sometimes shown that I had to wait a few seconds before the writing process was done before trying to read it. Would this be the case here too? If yes, wouldn't it mean it would have to happen if where I saved the data was memory?
If I was to write a c++ plugin for this purpose, that acted as a separate data bay where we could save(data) at point B to the memory and retrieve(data) at point C from the memory; would this work?
Honestly, I do not like writing temporary files to work around the asynchronous functions. I have been looking for a simple yet effective way to pass data around, but I need some guidance from experienced programmers like you to surpass the unnecessary approaches to this problem.
If you could toss around some ideas for me, stating what might work and what might not work, I'd appreciate it.
Here's the example code:
// Print all of the news items on hackernews
var jsdom = require("jsdom");
// var result;
// A) Outer Scope: Since the function is async in done, storing the result here and echoing in point C is pointless.
jsdom.env({
html: "http://news.ycombinator.com/",
scripts: ["http://code.jquery.com/jquery.js"],
done: function (errors, window) {
var $ = window.$;
console.log("HN Links");
$("td.title:not(:last) a").each(function() {
console.log(" -", $(this).text());
});
// B) let's say I want to return something I've scavenged here.
// result = $("a");
}
});
// C)
// console.log(result)
You need to clear your head of your synchronous experience that thinks code lower in the file happens later in time. It does not necessarily do that in node, ever. Here's the deal. In node, you place orders like at a restaurant, and you don't do it like:
1. Order a salad
2. Wait 11 minutes
3. Eat the salad
You do it like this
1. Order a salad
2. Wait for the server to serve you the salad
3. Eat the salad
The first example is a race condition and a terrible bug in your program that will cause either the salad waiting around to be eaten for no reason or trying to eat a salad that isn't there yet.
Don't think "I want to return something here", think "this data is ready". So you can have:
function eatSalad() {...}
placeOrder("salad", eatSalad);
Where eatSalad is the callback for the placeOrder routine, which does the necessary I/O to get the salad. Notice how even though eatSalad is earlier in the file, it happens later chronologically. You don't return things, you invoke callbacks with data you have prepared.
Here's your snippet made asynchronous.
// Print all of the news items on hackernews
var jsdom = require("jsdom");
// var result;
// A) Outer Scope: Since the function is async in done, storing the result here and echoing in point C is pointless.
jsdom.env({
html: "http://news.ycombinator.com/",
scripts: ["http://code.jquery.com/jquery.js"],
done: function (errors, window) {
var $ = window.$;
console.log("HN Links");
$("td.title:not(:last) a").each(function() {
console.log(" -", $(this).text());
});
// B) let's say I want to return something I've scavenged here.
// result = $("a");
resultIsReady($("a"));
}
});
function resultIsReady(element) {
console.log(element);
}
EDIT TO ADD to answer your question from the comments, node code will generally be built up not from functions that return things, but from functions that invoke callback functions with their "return value". The return keyword is only used to actually return a value for in-memory code that does not do any I/O. So finding the mean of an in-memory array can just return it, but finding the mean from a database result set must invoke a callback function. The basic paradigm is built up your programs from functions like this (pseudocode DB library):
function getUser(email, callback) {
db.users.where({email: email}, function (error, user) {
if (error) {
//This return statement is just for early termination
//of the function. The value returned is discarded
return callback(error);
}
callback(null, user);
});;
}
So that's how you do things. And typically functions like this do a very limited number of IO calls (1 or 2 are common, then you start falling into nesting hell and need to refactor).
I personally write lots of functions like that and then use the async library to describe the higher-order sequence of things that needs to happen. There are lots of other popular flow control libraries as well, and some people like the promises pattern. However, at the moment some of the core node community members seem to be advocating callbacks as the one true way and promises seem to be falling out of favor.
Avoid using synchronous code in any place where there is a chance for blocking the execution, such as database operations, file IO, network data retrieval, long calculations etc.
In your example use callback when you finish your computation to continue with execution, you could also take a look at async https://npmjs.org/package/async library which is de facto standard for hairier calls :
function sendData(result) {
res.json(result);
}
var jsdom = require("jsdom");
// var result;
// A) Outer Scope: Since the function is async in done, storing the result here
// and echoing in point C is pointless.
jsdom.env({
html: "http://news.ycombinator.com/",
scripts: ["http://code.jquery.com/jquery.js"],
done: function (errors, window) {
var $ = window.$;
console.log("HN Links");
$("td.title:not(:last) a").each(function () {
console.log(" -", $(this).text());
});
// B) let's say I want to return something I've scavenged here.
var result = $("a");
sendData(result);
}
});