We Keep Coding

In Parse.com's Cloud code, asynchronous code is giving variables in for-loop the wrong value

I'm trying to save different food names without duplicates on parse.com. However, when I run the code, the database consists of the same 2 or 3 foods over and over, instead of 200 or so unique names.
Below is my function. I tried logging the name of the food at two different points, and I get different values. The first point gives the correct name of the food, but the second point only shows either flaxseed muffins or raspberry pie. I think the problem has to do with the code running asynchronously, but I'm not sure how to resolve the issue.
Parse.Cloud.define("recordFavorite", function(request, response) {
var foodList = request.params.foodList; //string array of food names
var Food = Parse.Object.extend("Food");
var query = new Parse.Query(Food);
for (i = 0; i < foodList.length; i++ ) {
var name = foodList[i];
console.log("before name is " + name);
var query = new Parse.Query(Food);
query.exists("name", name);
query.find({
success: function(results) {
if(results.length == 0){
var food = new Food();
food.set("name", name);
food.save(null, {
success: function(food) {
console.log("saved with name " + name);
},
error: function(food, error) {
}
});
} else {
//don't create new food
}
},
error: function(error) {
}
});
}
});
EDIT:
I was able to make some progress by modifying it to the code pasted below. Now it saves all the objects, including duplicates. I noticed that the lines
var query = new Parse.Query(Food);
query.exists("name", name);
returns an array of all the foods and doesn't filter out the objects containing "name". (To be clear, this was probably still occurring in the original code, but I hadn't noticed.)
Parse.Cloud.define("recordFavorite", function(request, response) {
var foodList = request.params.foodList; //string array of food names
var foodListCorrected = new Array();
var Food = Parse.Object.extend("Food");
// Wrap your logic in a function
function process_food(i) {
// Are we done?
if (i == foodList.length) {
Parse.Object.saveAll(foodListCorrected, {
success: function(foodListCorrected) {
},
error: function(foodListCorrected) {
}
});
return;
}
var name = foodList[i];
var query = new Parse.Query(Food);
query.exists("name", name);
query.find({
success: function(results) {
console.log(results.length);
if(results.length == 0){
//console.log("before " + foodListCorrected.length);
var food = new Food();
food.set("name", name);
foodListCorrected.push(food);
// console.log(foodListCorrected.length);
} else {
//don't create new food
}
process_food(i+1)
},
error: function(error) {
console.log("error");
}
});
}
// Go! Call the function with the first food.
process_food(0);
});

I think you're right about the problem being the async logic. The problem is that the outer loop completes as quickly as it can, firing off the various, slower async calls for your food lookup queries as it goes. The outer loop doesn't wait and because of what's know as 'variable hoisting' when you access 'name' inside your success function, its value will be the latest value of 'name' in the outer loop. So when the success function is called, the value of name has moved on to a different food to when you first initiated the exists/save query sequence.
Here's a really simple example:
Say your foodList looked like ['Muffin'], ['Cheesecake']. When you enter the loop for the first time, you have name='Muffin'. You fire off your exists query for name='Muffin' and that now happens asynchronously. Meanwhile, the outer loop happily moves on and sets name='Cheesecake' and fires off another exists query. Meanwhile. your first exists query completes and you are now ready to save the first food. But, because of hoisting, the value of name within your success function is now 'Cheesecake'. So it saves 'Cheesecake' when it should have saved 'Muffin' Then the second set of async queries complete, and this one also saves 'Cheesecake'. So you get two foods, representing your two unique foods, but both are called 'Cheesecake'!
Here's the classic article on variable hoisting, it is well worth a read:
http://www.adequatelygood.com/JavaScript-Scoping-and-Hoisting.html
A way of solving this would be to only trigger the processing of the next food once all the async calls for the current food have completed. You can do this like this:
Parse.Cloud.define("recordFavorite", function(request, response) {
var foodList = request.params.foodList; //string array of food names
var Food = Parse.Object.extend("Food");
var query = new Parse.Query(Food);
// Wrap your logic in a function
function process_food(i) {
// Are we done?
if (i == foodList.length) return;
var name = foodList[i];
console.log("before name is " + name);
var query = new Parse.Query(Food);
query.exists("name", name);
query.find({
success: function(results) {
if(results.length == 0){
var food = new Food();
food.set("name", name);
food.save(null, {
success: function(food) {
console.log("saved with name " + name);
// Move onto the next food, only after all the async operations
// have completed.
process_food(i+1)
},
error: function(food, error) {
}
});
} else {
//don't create new food
}
},
error: function(error) {
}
});
}
// Go! Call the function with the first food.
process_food(0);
});
(Note, I've not tested this code, so there might be syntax errors).
I've not come across Parse before... I saw your question, went off to read about it, and thought it looked very interesting! I will remember it for my next PHP API project. I think there are some smarter things you can try to do. For example, your approach requires 2 async calls per food, one to see if it exists, and one to save it. For 200 foods, that's 400 async calls. However, the Parse API looks very helpful, and I think it will offer tools to help you cut this down. You could probably try something along the following lines:
You already have an array of strings of the names you want to save:
var foodList = request.params.foodList; //string array of food names
Say it looks like ["Cupcakes", "Muffins", "Cake"].
Now build a Parse query that gets all food names already on the server. (I don't know how to do this!). But you should get back an array, let's say ["Cupcakes", "Cheesecake"].
Now you an strip the duplicates in JavaScript. There'll be some nice questions here on StackOverflow to help with this! The result will be that "Cupcake" is a duplicate, so we are left with the array ["Muffins", "Cake"]
Now it looks like in Parse you can Batch some operations:
https://parse.com/docs/rest#objects-batch
so your goal is to save this array of ["Muffins", "Cake"] with one API call.
This approach will scale well with the number of foods, so even with 200 foods, you should be able to do it in one query, and one batch update per 50 foods (I think 50 is a batch limit, from the Parse docs), so at most you will need 5 API calls.

I believe this (https://www.parse.com/docs/js_guide#promises-series) is the solution you're looking for. You need to utilize promises to force synchronicity.

Delay this Javascript FOR Loop

This LOOP queries the Parse.com server & then plays with the results if any. The problem is that when nArray is greater than 100, the function exceeds the query/burst limit of Parse.com CloudCode & it fails.
One idea would be to delay the LOOP for a second after every 100 LOOPS, but I'm not sure how to do that. Any other solutions would be greatly appreciated.
Thanks in Advance,
for (var k = 1; k < nArray.length; k++) {
(function (k, mArray) { // <-- define an inline function
query2.equalTo("username", nArray[k]); // BURST LIMIT EXCEEDS
query2.find({
success: function (results) {
if (results.length !== 0) {
var object = results[0];
var compareUserEmail = object.get('email');
if (compareUserEmail !== userEmail) {
// alert("The result is equal to" + object.get('Name'));
mArray.push({
name: object.get('Name'),
email: object.get('email'),
bloxID: object.get('bloxID')
});
gameScore.set("filtered", mArray);
gameScore.save(null, {
success: function (gameScore) {
response.success("Success!");
alert('New object created with objectId: ' + gameScore.id);
},
error: function (gameScore, error) {
alert('Failed to create new object, with error code: ' + error.description);
}
});
}
};
},
error: function () {}
});
})(k, mArray);
// <-- call it after definition using (k)
};

You've got a couple of issues to deal with.
The reason Parse.com doesn't support setInterval is because that would be inviting disaster. It terminates your Cloud Code if it takes too long, so letting you add delays would just increase the chance your code is terminated before completion.
The reason Parse.com has a burst limit is that this usually suggest "you are doing it wrong (tm)". In your case you are looping through an array and running a query for each item in the array. Instead you should be using the containedIn method to get all records for the array in one go. If you are getting more than 100 items in your array you can choose to increase the record limit to 1000, but first consider carefully if this is really what you need.
Given that you are modifying a lot of objects and saving them all, consider using the saveAll method to save them all in one hit too.
You might want to consider batching these operations, but be aware of the restrictions on overall duration for Cloud Code.

You can use a setInterval:
var i = 0;
var intervalId = setInterval(function() {
if(i < nArray.length) {
... your code ...
i++;
} else {
clearInterval(intervalId);
}
}, 100); //every 100ms; change it to what you need

Best method to string together variety of DB calls in Node js

I basically need to make about 3 calls to get the data for a json object.. It basically JSON array of JSON object which have some attributes, one of which is an array of other values selected using a second query, then that one also has an array inside which is selected with another db call.
I tried using asyn.concatSeries so that I can dig down into the bottom call and put together all the information I collected for one root json object but that's creating a lot of unexpected behaviour..
Example of JSON
[
{
"item" : "firstDbCall"
"children" : [ {
"name" : "itemDiscoveredWithSecondDBCall"
"children" : [ itemsDiscoveredwith3rdDBCall]
},
]
}
]
This is really difficult using node.js. I really need to figure out how to do this properly since I have to do many of these for different purposes.
EDIT
This is the code i have. There's some strange behaviour with async.concatSeries. The results get called multiple times after each one of the functions finish for each array. So i had to put a check in place. I know it's very messy code but i've been just putting band-aids all over it for the past 2 hours to make it work..
console.log("GET USERS HAREDQARE INFO _--__--_-_-_-_-_____");
var query = "select driveGroupId from tasks, driveInformation where agentId = '"
+ req.params.agentId + "' and driveInformation.taskId = tasks.id order by driveInformation.taskId desc;";
connection.query(query, function(err, rows) {
if (rows === undefined) {
res.json([]);
return;
}
if(rows.length<1) { res.send("[]"); return;}
var driveGroupId = rows[0].driveGroupId;
var physicalQuery = "select * from drives where driveGroupId = " + driveGroupId + ";";
connection.query(physicalQuery, function(err, rows) {
console.log("ROWSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS");
console.log(rows);
async.concatSeries(rows, function(row, cb) {
console.log("-------------------------------SINGLE ROW-------------------------------------");
console.log(row);
if(row.hasLogicalDrives != 0) {
console.log("HAS LOGICAL DRIVES");
console.log(row.id);
var query = "select id, name from logicalDrives where driveId = " + row.id;
connection.query(query, function(error, drives) {
console.log("QUERY RETURNED");
console.log(drives);
parseDriveInfo(row.name, row.searchable, drives, cb);
});
}
else
var driveInfo = { "driveName" : row.name, "searchable" : row.searchable};
console.log("NO SUB ITEMS");
cb(null, driveInfo);
}, function(err, results) {
console.log("GEETTTTINGHERE");
console.log(results);
if(results.length == rows.length) {
console.log("RESULTS FOR THE DRIVE SEARCH");
console.log(results);
var response = {"id": req.params.agentId};
response.driveList = results;
console.log("RESPONSE");
console.log(response);
res.json(response);
}
});
});
});
};
parseDriveInfo = function(driveName, searchable, drives, cb) {
async.concatSeries(drives, function(drive,callback) {
console.log("SERIES 2");
console.log(drive);
console.log("END OF DRIVE INFO");
var query = "select name from supportedSearchTypes where logicalDriveId = " + drive.id;
connection.query(query, function(error, searchTypes) {
drive.searchTypes = searchTypes;
var driveInfo = { "driveName" :driveName,
"searchable" : searchable,
"logicalDrives" : drive
};
callback(null, driveInfo);
});
}, function (err, results) {
console.log("THIS IS ISISIS ISISISSISISISISISISISISISIS");
console.log(results);
if(results.length === drives.length) {
console.log("GOTHERE");
cb(null, results);
}
});
}

Getting good enough with async to use exactly the right combination of methods under the right circumstances takes a fair amount of experience. Most likely your case in particular can be handled with async.waterfall if its query1 then query2(dataFoundByQuery1) then query3(dataFoundByQuery2). But depending on the circumstances you need to mix and match async methods appropriately and sometimes have 2 levels - for example a "big picture" async.waterfall where some of the steps in the waterfall do async.parallel or async.series as needed. I've never used async.concat and given your needs I think you have chosen the wrong method. The workhorses are async.each, async.eachSeries, async.waterfall, and async.map, at least for the web app & DB query use cases I mostly encounter, so make sure you really have those understood before exploring the more specific convenience methods.

EDIT: This is a more in depth example based on use of the connection library you seem to be using. Please note, some of this is javascript psuedo code. Things like adding objects to the resultsArray are clearly not complete, the only thing I took time to make sure was correct is the "flow of logic" as it pertains to callbacks. Everything else is for you to implement. In order to support multiple calls to the same callback function and maintain state from call to call, the best way is to wrap the set of callbacks in a closure. This allows the callbacks to share some state with the main event loop. This allows you to pass arguments to the callbacks, without actually having to pass them as arguments, much like class variables in c++, or even globals in javascript, but we haven't poluted the global scope :)
function queryDataBase(query) {
//wrap the whole query in a function so the callbacks can share some
//variables with similar scope. This is called a closure
int rowCounter = 0;
var dataRowsFromStep2;
var resultsArray = {};
connection.query(query, dataBaseQueryStep2);
function dataBaseQueryStep2(err, rows) {
//do something with err and rows
dataRowsFromStep2 = rows;
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);//Always zero the first time. Might need to double check rows isn't empty!
connection.query(query, dataBaseQueryStep3);
}
function dataBaseQueryStep3(err, rows) {
//do something with err and rows
if(rowCounter < dataRowsFromStep2.size) {
resultsArray.add(rows);//Probably needs to be more interesting, but you get the idea
//since this is within the same closure, rowCounter maintains it's state
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);
//recursive call query using dataBaseQueryStep3 as it's callback repeatedly until
//we run out of rows to call it on.
connection.query(query, dataBaseQueryStep3)
} else {
//when the if statement fails we have no more rows to run queries on so return to main program flow
returnToMainProgramLogic(resultsArray);
}
}
}
function returnToMainProgramLogic(results) {
//continue running your program here
}
I personally like the above logic better than the syntax async produces... I believe the heart of your problem rests in your nested calls to async, and the fact that ASYN itself, runs the series of functions asynchronously, but in order(confusing I know). If you write your program like this, you won't have to worry about it!

I would strongly suggest using sequelize.js It provides a really powerful orm that allows you to chain queries together. It also allows you to directly load your data into js objects, write dynamic sql, and connect to many different databases. Picture ActiveRecord from the Ruby world for Node.

How to return value from a for loop which is querying the database in Javascript?

geturls(data,function(urls){
var data = {
"data": [
{ "userProfile": userP },
{ "urls": urls }
]
};
res.send(data);
});
function getUrls(data,done){
links = new Array();
for (var i=0; i<data.length; i++){
user = data[i]
Url.find({where:{data.id}}).success(function(url){
links.push({
"url": ur.text,
"date": data.syncedTime
});
if (urls.length == data.length){
done(links);
}
});
}
}
My problem with my code is this:
I'm returning the response through a callback once data collected in my array equals the length of the parent array. This is obviously a very dangerous and not so elegant solution. As, suppose I get a .failure from Url database, then my urls.length won't be equal with data.length. So, I'm a bit confused how to go about this.
Any help?

It will be easy for you, if you use async.js.
I used mapSeries here. It takes 3 parameters.
collection/array
iterator, which will be called for each item in the passed collection/array with 2 arguments. 1. item in collection, 2. callback. After completing the job in iterator, You should call the callback in node style(err first, results follows).
Final callback, which will be called after all the items in the collection mapped.
function getUrls(data,done){
var async = require('async');
async.mapSeries(data, function(user, cb) {//If you want it to be async `async.map`
Url.find({where:{user.id}}).success(function(url){
cb(null, {
"url": url.text,
"date": user.syncedTime
});
});
}, function(err, results) {
//results is an array. Its the same as `links` in your old code.
done(results);
});
}
geturls(data,function(urls){
var data = {
"data": [
{ "userProfile": userP },
{ "urls": urls }
]
};
res.send(data);
});

Use recursion:
function getUrls(data,done) {
var links = new Array();
function doGetUrl(i) {
var user = data[i];
Url.find({where:{data.id}}).
success(function(url){
links.push({
"url": ur.text,
"date": data.syncedTime
});
if (links.length == data.length){
done(links);
} else {
doGetUrl(i + 1); // get next url
}
}).
failure(function(err) {
doGetUrl(i); // on error, try to get current url again
// other error handling code
});
}
doGetUrl(0);
}

I would probably make use of the complete callback, in jQuery terms. Have a counter that records how many records have been processed and update this in complete, as this executes on success or failure. Then when that counter is >= the length of the data array you can exit.
As an aside, I would always do a >= rather than an == for the comparison you are doing there, that way, if for any crazy reason, the count is upped more than it should you still exit.

If alll you want to do is avoif the problem of checking links.length to determine when you are done then I think its just a matter of adding a separate counter that gets incremented even if the urk database fails. If you do that you can continue using your current stype where the async requests are run in parallel.
var nreq = 0;
for (var i=0; i<data.length; i++){
doTheAsyncOperation(function(){
//Run this part in both the success and error cases
nreq = nreq + 1;
if(nreq >= data.length){ done(links) }
})
}
On the other hand, if you want to run one query after the other you will need to rewrite the for to use recursion. This time, you don't need to worry about keeping a separate counter since you know when the final request runs:
function loop(i){
if(i >= data.length){
done(links);
}else{
doTheAsyncOperation(function(){
loop(i+1);
})
}
}
loop(0);
Finally, its good to know how to code this sort of patterns yourself but in the long run I highly recommend using a control flow library to keep things cleaner.

Synchronous query to Web SQL Database

I'm working on a bit of JavaScript that interacts with a client-side SQLite database, via the newish window.openDatabase(...), database.transaction(...) and related APIs. As most of you know when you execute a query in this way it is an asynchronous call, which is typically good. You can make the call and handle the results as appropriate with callbacks.
In my current situation I'm working on an algo for a client that does some hierarchy walking in the locally stored database. The part of the algo I'm having trouble with requires starting at some row, which has a reference to a "parent" (by id) that is another row further up in the table. I have to keep walking up this tree until I reach the root.
The problem is that I'm at a point where I'm not sure how to use an asynchronous style query with a callback to keep feeding the loop parent ids. Ideally I could get the query to block so that I can do it all in the loop. Here's the key parts of my current setup:
for (i in search.searchResults.resultsArray)
{
hierarchyArr = new Array();
pageHierarchyArr = new Array();
id = search.searchResults.resultsArray[i].ID;
while (id != null && id != "")
{
var hierarchySql = "SELECT ID, parentID, type, content FROM content WHERE ID = " + id;
// This is a prettied up call to database.transaction(...)
var rs = db.getRS(hierarchySql);
// Ideally the code below doesn't execute until rs is populated
hierarchyArr.push(rs[0]);
if (rs[0].type == "page")
{
pageHierarchyArr.push(rs[0]);
// Do some additional work
}
id = rs[0].parentID;
}
}
As you might imagine, it doesn't work well. hierarchyArr gets an "undefined" pushed into it, and then the script crashes when it tries to check the type of rs[0].
When I try to set it up with a callback (db.getRSAndCallback(sql, callbackFunc), which I used for the earlier, non-interdependent queries just fine) it's worse: the inner loop takes off like crazy because id isn't getting updated; presumably because the loop is keeping the JavaScript interpreter so busy that it never actually fills rs. In some artificial testing where I forced the inner loop to break after a few iterations all the callbacks started coming through all at the end, after the loop finished.
The "standard" (such as it is right now) at http://dev.w3.org/html5/webdatabase/#synchronous-database-api seems to indicate that there is a synchronous API, but I haven't seen any sign of it on any WebKit based browsers.
Can anyone offer suggestions on how I might either, a. properly formulate these iterative, interdependent queries using callbacks or, b. somehow get the call to actually happen in a synchronous or apparently synchronous manner.
Many thanks in advance for anyone who takes a crack at this seemingly tricky little problem.
Naim
P.S. Here's the client's implementation of db.getRS for reference:
.
.
.
getRS: function(sql)
{
var output = [];
db.database.transaction(function(tx)
{
tx.executeSql(sql, [], function(tx,rs)
{
for(i = 0; i < rs.rows.length; i++)
{
output.push(rs.rows.item(i));
}
},
function(tx, error) { ... }
)});
return output;
},
.
.
.

I used callbacks and a closure to solve a similar problem, consider:
function getFolder(id, callback) {
var data = [];
ldb.transaction(function (tx) {
tx.executeSql('SELECT * FROM folders where id=?',
[id],
function (tx, results) {
if (results.rows && results.rows.length) {
for (i = 0; i < results.rows.length; i++) {
data.push(results.rows.item(i));
}
}
if (typeof(callback) == 'function')
callback(data);
},
function (tx, error) {
console.log(error);
});
});
}
In the continuation of this example, folder has a property parent to define it's relation to other folders. As does a document. The following will get you the path of a document using a closure (success):
function getDocPath(doc, callback) {
var path = [];
var parent = doc.parent;
var success = function(folder) {
var folder = folder[0];
parent = folder.parent;
path.push({'id':folder.id,'name':folder.name});
if (parent != "undefined")
getFolder(parent, success);
else
if ( typeof(callback) == 'function' ) callback(path.reverse());
}
getFolder(parent, success);
}

You could use callbacks with a closure to your stack of remaining queries. Or you could use recursion, passing the stack as parameters.