We Keep Coding

How to force sequential execution for asynchronously called functions

I'm new to the idea of asynchronous code, and am still trying to wrap my brain around how everything works.
I'm building a Node Express application which will interface with a database. When running in a development environment I want it to interface with a Sqlite database. (The production database will not use Sqlite. This only applies to creating a small development environment.)
My problem is I'm having trouble controlling the execution order and timing of queries to the database.
I would like to build my SqliteDatabase.js file such that it can only execute queries sequentially, despite the fact that functions in this file will be called by other parts of the program that are running asynchronously.
How can I acheive this?
For reference, here is how I currently have my SqliteDatabase.js file set up:
var debug = require('debug')('app:DATABASE');
var sqlite = require('sqlite3').verbose();
open = function() {
var db = new sqlite.Database('./test-database.db', sqlite.OPEN_READWRITE | sqlite.OPEN_CREATE, function(err) {
if (err) {
debug("We've encountered an error opening the sqlite database.");
debug(err);
} else {
debug("Sqlite database opened successfully.");
}
});
return db;
}
executeSQLUpdate = function(sql, next) {
var db = open();
db.serialize(function() {
console.log("executing " + sql);
db.run(sql);
db.close();
next();
});
}
exports.executeSQLUpdate = executeSQLUpdate;
Is there some way to build a queue, and make it so when the "executeSQLUpdate" function is called, the request is added to a queue, and is not started until all previous requests have been completed?
To give an example, take a look at this code which utilises my SqliteDatabase.js file:
ar database = require('../../bin/data_access/SqliteDatabase.js');
var createTestTableStmt = "CREATE TABLE IF NOT EXISTS Test(\n" +
"Name TEXT PRIMARY KEY NOT NULL UNIQUE,\n" +
"Age INT NOT NULL,\n" +
"Gender TEXT NOT NULL\n" +
");";
var clearTestTableStmt = "DELETE FROM Test;";
var testInsertStmt = "INSERT INTO Test (Name, Age, Gender)\n" +
"VALUES (\"Connor\", 23, \"Male\");";
createTable = function() {
database.executeSQLUpdate(createTestTableStmt, clearTable);
}
clearTable = function() {
database.executeSQLUpdate(clearTestTableStmt, insertRow);
}
insertRow = function() {
database.executeSQLUpdate(testInsertStmt, function() {
console.log("Done!");
});
}
createTable();
9 times out of 10 the above code works fine, but every once in a while, the "insert row" function is called before the "clearTable" function is called, which throws an error because of a violated database constraint.
How can I change my implementation of the SqliteDatabase.js file to avoid this issue?

You can use async to do this using await. This code will wait for each asynchronous database call to complete before executing the next line.
async function createTable() {
await database.executeSQLUpdate(createTestTableStmt);
await database.executeSQLUpdate(clearTestTableStmt);
await database.executeSQLUpdate(testInsertStmt);
console.log("Done!");
}
Your console.log statement will only execute once all three have completed.
I should also mention that you need a try...catch block around the three database calls to trap any errors and provide an alternate exit point if something should go wrong.

I realized why the callback function next() was sometimes being called before db.run(sql)
It turns out that db.run() is itself an asychronous function. I updated my code, and added a callback to the db.run() line to make sure we don't skip ahead until it's done.
Here's what it looks like now:
executeSQLUpdate = function(sql, next) {
var db = open();
db.run(sql, function(err) {
db.close(function() {
if (next) next(err);
});
});
}
Nesting each asynchronous function in the previous function's callback, makes each function execute in order.
Thanks to everyone who gave me hints that helped me figure out what the problem was.

What does response.on mean? Node.js

I am having trouble with a Node.js http request. (A bigger question I will ask about later if I can't figure it out).
I have code that I modified from and an example and I don't understand what the response.on means. As read more about http in Node.js (Anatomy of an HTTP Transaction) I don't see people using examples with 'response.on'. I think I know but I want to clarify. Oh I am using Express.js too.
Thanks
Below is the code I am using to try and call the BART api I need a response from.
...
// Real Time Departure from a given station
router.route('/departTimeStation')
.get(function(req, res) {
vCmd = 'etd';
vOrig = req.query.vOriginStation;
vDir = 'n'; // [NOTE] - 'n' or 's', north or south, OPTIONAL
vPlat = 1; // [NOTE] - 1 to 4, number of platform, OPTIONAL
var xoptions = {
host: 'api.bart.gov',
path: '/api/etd.aspx?cmd=' + vCmd + '&orig=' + vOrig + '&key=' + config.bart.client
};
var xcallback = function(response) {
response.on('data', function(chunk) {
vParsed += chunk;
});
response.on('end', function() {
parseString(vParsed, function(err, result) {
vShow = JSON.stringify(result);
vTemp = result;
});
});
};
http.request(xoptions, xcallback).end();
return res.send (vTemp)
});
...

response.on can take a few 'events' that happen during the response life cycle,
The two you have above are response.on('data'...) which is when you get the data
and response.on('end'...) is at the end of the response, simple as that!
Node Docs on response: https://nodejs.org/api/http.html#http_class_http_incomingmessage

response.on is a function, or rather method, since it is a property.

Functions that can't execute together

I was recently building a scraper module to get some information with nodejs until I encountered this "little" problem. The modules that I'm using are cheeriojs and request.
Actually the module works like a charm if I call only one method at a time. It contains three function and only two of them are exported, this is the code:
'use strict';
var request = require('request'),
cheerio = require('cheerio'),
counter = 0;
function find(term, cat, callback) {
// All the check for the parameters
scrape("http://.../search.php?search=" + encodeURIComponent(term), cat, callback);
}
function last(cat, callback) {
// All the check for the parameters
scrape("http://google.com/", cat, callback);
}
function scrape(url, cat, callback) {
request(url, function (error, response, body) {
if (!error && response.statusCode == 200) {
var $ = cheerio.load(body);
var result = [];
var items = $('.foo, .foo2').filter(function() {
// Condition to filter the resulted items
});
items.each(function(i, row) {
// Had to do another request inside here to scrape other information
request( $(".newpagelink").attr("href"), function(error, response, body) {
var name = $(".selector").text(),
surname = $(".selector2").text(),
link = cheerio.load(body)('.magnet').attr('href'); // This is the only thing that I'm scraping from the new page, the rest comes from the other "cheerio.load"
// Push an object in the array
result.push( { "name": name, "surname": surname, "link": link } );
// To check when the async requests are ended
counter++;
if(counter == items.length-1) {
callback(null, result);
}
});
});
}
});
}
exports.find = find;
exports.last = last;
The problem now, as I was saying, is that if I create a new node script "test.js" and I call only last OR find, it works perfectly! But if I call both the methods consecutively like this:
var mod = require("../index-tmp.js");
mod.find("bla", "blabla", function(err, data) {
if (err) throw err;
console.log(data.length + " find");
});
mod.last(function(err, data) {
console.log(data.length + " last");
});
The results are completely messed up, sometimes the script doesn't even print something, other times print the result of only "find" or "last", and other times returns a cheeriojs error (I won't add here to not mess you up, because probably it's my script's fault). I thought also to repeat the same function two times for both the methods but nothing, the same problems occur... I don't know what else to try, I hope you'll tell me the cause of this behavior!

Your counter variable is global, not specific to each scrape call. It wouldn't work if you called find twice at the same time either, or last.
Move the declaration and initialisation of var counter = 0; into the scrape function, or even better right next to the result and items declarations.

From scanning your code quickly, this is probably due to the variable counter being global. These are asynchronous functions, so they will both act on counter at the same thing. Move the declaration inside of the scrape function.
If you need more information about asynchronous programming, refer to Felix's great answer in this question.

Parse CloudCode order of execution

Im trying to send a push message to everyone with read access every time a new note is saved.
In pseudocode it should get the ACL. Evaluate each member in the ACL and return an array of all users with read access. Then send a push notification to each member.
I've tried running separate task one by one and it works properly. However when I put everything together in the following code I get strange results. Looking at the log I can see it not executing in order as I expect. I first though the getACL call was an asynchronous call so I tried to implement promises with no luck. Then after help from stackoverflow I find out that getACL is not asynchronous therefore the code should just work, right?
This is what I'm trying:
Parse.Cloud.afterSave("Notes", function(request) {
var idsToSend = [];
var i = 0;
console.log("1 start");
var objACL = request.object.getACL();
var ACLinJSON = objACL.toJSON();
console.log("2 ACL = " + ACLinJSON);
for (var key in ACLinJSON) {
if (ACLinJSON[key].read == "true") {
idsToSend[i] = key.id;
console.log("3 i = " + i + " = " + idsToSend[i]);
i++;
}
}
console.log("4 idsToSend = " + idsToSend);
//lookup installations
var query = new Parse.Query(Parse.Installation);
query.containedIn('user', idsToSend);
Parse.Push.send({
where: query,
data: {
alert: "note updated"
}
}, {
success: function() {
console.log("Success sent push");
},
error: function(error) {
console.error("can’t find user"); //error
}
});
});
And this is the response I see from parse log
I2014-08-04T08:08:06.708Z]4 idsToSend =
I2014-08-04T08:08:06.712Z]2 ACL = [object Object]
I2014-08-04T08:08:06.714Z]1 start
I2014-08-04T08:08:06.764Z]Success sent push
Everything is completely out of order??
How can I execute the above function in the way it's written?

I've found the logs are not in order when I run things too, could be a timing issue or something, ignore the order when they're in the same second, I have done other tests to confirm things really do run in order on my own Cloud Code... had me completely confused for a while there.
The issue you're having is that log #3 is never being hit... try tracing ACLinJSON on it's own to see the actual structure. When you append it to a string it outputs [object Object] as you have seen, so do console.log(ACLinJSON); instead.
Here's the structure I've seen:
{
"*":{"read":true},
"Administrator":{"write":true}
}
Based on that I would expect your loop to work, but it may have a different level of wrapping.
UPDATE:
Turns out the issue was looking for the string "true" instead of a boolean true, thus the fix is to replace the following line:
// replace this: if (ACLinJSON[key].read == "true") {
if (ACLinJSON[key].read == true) {

Calling asynchronous function in callback

I'm having some trouble understanding asynchronous functions. I've read the chapter in Mixu's Node Book but I still can't wrap my head around it.
Basically I want to request a ressource (using the node package cheerio), parse it for valid URLs and add every match to my redis set setname.
The problem is that in the end it's only adding the first match to the redis set.
function parse(url, setname)
{
request(url, function (error, response, body)
{
if (!error && response.statusCode == 200)
{
$ = cheerio.load(body)
// For every 'a' tag in the body
$('a').each(function()
{
// Add blog URL to redis if not already there.
var blog = $(this).attr('href')
console.log("test [all]: " + blog);
// filter valid URLs
var regex = /http:\/\/[^www]*.example.com\//
var result = blog.match(regex);
if(result != null)
{
console.log("test [filtered]: " + result[0]);
redis.sismember(setname, result[0], function(err, reply)
{
if(!reply)
{
redis.sadd(setname, result[0])
console.log("Added " + result[0])
}
redis.quit()
})
}
})
}
})
}
I'd be very grateful for pointers on how I'd have to restructure this so the redis.sadd method is working with the correct result.
The output of the current implementation looks like:
test [all]: http://test1.example.com/
test [filtered]: http://test1.example.com/
...
Added http://test2.example.com/
So it's adding the test1.example.com but not printing the "added" line, and it's not adding the test2.example.com but it's printing the "added" line for it.
Thank you!

The first issue is caused by redis.sismember() being asynchronous: when its callback is called, you have already overwritten the result variable so it will point to the last value it had, and not the value at the moment at which you called redis.sismember().
One way to solve that is to create a new scoped variable by wrapping the asynchronous function in a closure:
(function(result) {
redis.sismember(setname, result[0], function(err, reply) {
...
});
})(result);
Another option is to create a partial function that's used as callback:
redis.sismember(setname, result[0], function(result, err, reply) {
...
}.bind(this, result));
The second issue is, I think, caused by redis.quit() being called, which closes the Redis connection after the first sadd(). You're not checking err, but if you do it might tell you more.