We Keep Coding

How to force sequential execution for asynchronously called functions

I'm new to the idea of asynchronous code, and am still trying to wrap my brain around how everything works.
I'm building a Node Express application which will interface with a database. When running in a development environment I want it to interface with a Sqlite database. (The production database will not use Sqlite. This only applies to creating a small development environment.)
My problem is I'm having trouble controlling the execution order and timing of queries to the database.
I would like to build my SqliteDatabase.js file such that it can only execute queries sequentially, despite the fact that functions in this file will be called by other parts of the program that are running asynchronously.
How can I acheive this?
For reference, here is how I currently have my SqliteDatabase.js file set up:
var debug = require('debug')('app:DATABASE');
var sqlite = require('sqlite3').verbose();
open = function() {
var db = new sqlite.Database('./test-database.db', sqlite.OPEN_READWRITE | sqlite.OPEN_CREATE, function(err) {
if (err) {
debug("We've encountered an error opening the sqlite database.");
debug(err);
} else {
debug("Sqlite database opened successfully.");
}
});
return db;
}
executeSQLUpdate = function(sql, next) {
var db = open();
db.serialize(function() {
console.log("executing " + sql);
db.run(sql);
db.close();
next();
});
}
exports.executeSQLUpdate = executeSQLUpdate;
Is there some way to build a queue, and make it so when the "executeSQLUpdate" function is called, the request is added to a queue, and is not started until all previous requests have been completed?
To give an example, take a look at this code which utilises my SqliteDatabase.js file:
ar database = require('../../bin/data_access/SqliteDatabase.js');
var createTestTableStmt = "CREATE TABLE IF NOT EXISTS Test(\n" +
"Name TEXT PRIMARY KEY NOT NULL UNIQUE,\n" +
"Age INT NOT NULL,\n" +
"Gender TEXT NOT NULL\n" +
");";
var clearTestTableStmt = "DELETE FROM Test;";
var testInsertStmt = "INSERT INTO Test (Name, Age, Gender)\n" +
"VALUES (\"Connor\", 23, \"Male\");";
createTable = function() {
database.executeSQLUpdate(createTestTableStmt, clearTable);
}
clearTable = function() {
database.executeSQLUpdate(clearTestTableStmt, insertRow);
}
insertRow = function() {
database.executeSQLUpdate(testInsertStmt, function() {
console.log("Done!");
});
}
createTable();
9 times out of 10 the above code works fine, but every once in a while, the "insert row" function is called before the "clearTable" function is called, which throws an error because of a violated database constraint.
How can I change my implementation of the SqliteDatabase.js file to avoid this issue?

You can use async to do this using await. This code will wait for each asynchronous database call to complete before executing the next line.
async function createTable() {
await database.executeSQLUpdate(createTestTableStmt);
await database.executeSQLUpdate(clearTestTableStmt);
await database.executeSQLUpdate(testInsertStmt);
console.log("Done!");
}
Your console.log statement will only execute once all three have completed.
I should also mention that you need a try...catch block around the three database calls to trap any errors and provide an alternate exit point if something should go wrong.

I realized why the callback function next() was sometimes being called before db.run(sql)
It turns out that db.run() is itself an asychronous function. I updated my code, and added a callback to the db.run() line to make sure we don't skip ahead until it's done.
Here's what it looks like now:
executeSQLUpdate = function(sql, next) {
var db = open();
db.run(sql, function(err) {
db.close(function() {
if (next) next(err);
});
});
}
Nesting each asynchronous function in the previous function's callback, makes each function execute in order.
Thanks to everyone who gave me hints that helped me figure out what the problem was.

Twitch TV JSON API Issue

So,I am trying to use the twitch API:
https://codepen.io/sterg/pen/yJmzrN
If you check my codepen page you'll see that each time I refresh the page the status order changes and I can't figure out why is this happening.
Here is my javascript:
$(document).ready(function(){
var ur="";
var tw=["freecodecamp","nightblue3","imaqtpie","bunnyfufuu","mushisgosu","tsm_dyrus","esl_sc2"];
var j=0;
for(var i=0;i<tw.length;i++){
ur="https://api.twitch.tv/kraken/streams/"+tw[i];
$.getJSON(ur,function(json) {
$(".tst").append(JSON.stringify(json));
$(".name").append("<li> "+tw[j]+"<p>"+""+"</p></li>");
if(json.stream==null){
$(".stat").append("<li>"+"Offline"+"</li>");
}
else{
$(".stat").append("<li>"+json.stream.game+"</li>");
}
j++;
})
}
});

$.getJSON() works asynchronously. The JSON won't be returned until the results come back. The API can return in different orders than the requests were made, so you have to handle this.
One way to do this is use the promise API, along with $.when() to bundle up all requests as one big promise, which will succeed or fail as one whole block. This also ensures that the response data is returned to your code in the expected order.
Try this:
var channelIds = ['freecodecamp', 'nightblue3', 'imaqtpie', 'bunnyfufuu', 'mushisgosu', 'tsm_dyrus', 'esl_sc2'];
$(function () {
$.when.apply(
$,
$.map(channelIds, function (channelId) {
return $.getJSON(
'https://api.twitch.tv/kraken/streams/' + encodeURIComponent(channelId)
).then(function (res) {
return {
channelId: channelId,
stream: res.stream
}
});
})
).then(function () {
console.log(arguments);
var $playersBody = $('table.players tbody');
$.each(arguments, function (index, data) {
$playersBody.append(
$('<tr>').append([
$('<td>'),
$('<td>').append(
$('<a>')
.text(data.channelId)
.attr('href', 'https://www.twitch.tv/' + encodeURIComponent(data.channelId))
),
$('<td>').text(data.stream ? data.stream.game : 'Offline')
])
)
})
})
});
https://codepen.io/anon/pen/KrOxwo
Here, I'm using $.when.apply() to use $.when with an array, rather than list of parameters. Next, I'm using $.map() to convert the array of channel IDs into an array of promises for each ID. After that, I have a simple helper function with handles the normal response (res), pulls out the relevant stream data, while attaching the channelId for use later on. (Without this, we would have to go back to the original array to get the ID. You can do this, but in my opinion, that isn't the best practice. I'd much prefer to keep the data with the response so that later refactoring is less likely to break something. This is a matter of preference.)
Next, I have a .then() handler which takes all of the data and loops through them. This data is returned as arguments to the function, so I simply use $.each() to iterate over each argument rather than having to name them out.
I made some changes in how I'm handling the HTML as well. You'll note that I'm using $.text() and $.attr() to set the dynamic values. This ensures that your HTML is valid (as you're not really using HTML for the dynamic bit at all). Otherwise, someone might have the username of <script src="somethingEvil.js"></script> and it'd run on your page. This avoids that problem entirely.

It looks like you're appending the "Display Name" in the same order every time you refresh, by using the j counter variable.
However, you're appending the "Status" as each request returns. Since these HTTP requests are asynchronous, the order in which they are appended to the document will vary each time you reload the page.
If you want the statuses to remain in the same order (matching the order of the Display Names), you'll need to store the response data from each API call as they return, and order it yourself before appending it to the body.

At first, I changed the last else condition (the one that prints out the streamed game) as $(".stat").append("<li>"+jtw[j]+": "+json.stream.game+"</li>"); - it was identical in meaning to what you tried to achieve, yet produced the same error.
There's a discrepancy in the list you've created and the data you receive. They are not directly associated.
It is a preferred way to use $(".stat").append("<li>"+json.stream._links.self+": "+json.stream.game+"</li>");, you may even get the name of the user with regex or substr in the worst case.
As long as you don't run separate loops for uploading the columns "DisplayName" and "Status", you might even be able to separate them, in case you do not desire to write them into the same line, as my example does.
Whatever way you're choosing, in the end, the problem is that the "Status" column's order of uploading is not identical to the one you're doing in "Status Name".

This code will not preserve the order, but will preserve which array entry is being processed
$(document).ready(function() {
var ur = "";
var tw = ["freecodecamp", "nightblue3", "imaqtpie", "bunnyfufuu", "mushisgosu", "tsm_dyrus", "esl_sc2"];
for (var i = 0; i < tw.length; i++) {
ur = "https://api.twitch.tv/kraken/streams/" + tw[i];
(function(j) {
$.getJSON(ur, function(json) {
$(".tst").append(JSON.stringify(json));
$(".name").append("<li> " + tw[j] + "<p>" + "" + "</p></li>");
if (json.stream == null) {
$(".stat").append("<li>" + "Offline" + "</li>");
} else {
$(".stat").append("<li>" + json.stream.game + "</li>");
}
})
}(i));
}
});
This code will preserve the order fully - the layout needs tweaking though
$(document).ready(function() {
var ur = "";
var tw = ["freecodecamp", "nightblue3", "imaqtpie", "bunnyfufuu", "mushisgosu", "tsm_dyrus", "esl_sc2"];
for (var i = 0; i < tw.length; i++) {
ur = "https://api.twitch.tv/kraken/streams/" + tw[i];
(function(j) {
var name = $(".name").append("<li> " + tw[j] + "<p>" + "" + "</p></li>");
var stat = $(".stat").append("<li></li>")[0].lastElementChild;
console.log(stat);
$.getJSON(ur, function(json) {
$(".tst").append(JSON.stringify(json));
if (json.stream == null) {
$(stat).text("Offline");
} else {
$(stat).text(json.stream.game);
}
}).then(function(e) {
console.log(e);
}, function(e) {
console.error(e);
});
}(i));
}
});

Best method to string together variety of DB calls in Node js

I basically need to make about 3 calls to get the data for a json object.. It basically JSON array of JSON object which have some attributes, one of which is an array of other values selected using a second query, then that one also has an array inside which is selected with another db call.
I tried using asyn.concatSeries so that I can dig down into the bottom call and put together all the information I collected for one root json object but that's creating a lot of unexpected behaviour..
Example of JSON
[
{
"item" : "firstDbCall"
"children" : [ {
"name" : "itemDiscoveredWithSecondDBCall"
"children" : [ itemsDiscoveredwith3rdDBCall]
},
]
}
]
This is really difficult using node.js. I really need to figure out how to do this properly since I have to do many of these for different purposes.
EDIT
This is the code i have. There's some strange behaviour with async.concatSeries. The results get called multiple times after each one of the functions finish for each array. So i had to put a check in place. I know it's very messy code but i've been just putting band-aids all over it for the past 2 hours to make it work..
console.log("GET USERS HAREDQARE INFO _--__--_-_-_-_-_____");
var query = "select driveGroupId from tasks, driveInformation where agentId = '"
+ req.params.agentId + "' and driveInformation.taskId = tasks.id order by driveInformation.taskId desc;";
connection.query(query, function(err, rows) {
if (rows === undefined) {
res.json([]);
return;
}
if(rows.length<1) { res.send("[]"); return;}
var driveGroupId = rows[0].driveGroupId;
var physicalQuery = "select * from drives where driveGroupId = " + driveGroupId + ";";
connection.query(physicalQuery, function(err, rows) {
console.log("ROWSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS");
console.log(rows);
async.concatSeries(rows, function(row, cb) {
console.log("-------------------------------SINGLE ROW-------------------------------------");
console.log(row);
if(row.hasLogicalDrives != 0) {
console.log("HAS LOGICAL DRIVES");
console.log(row.id);
var query = "select id, name from logicalDrives where driveId = " + row.id;
connection.query(query, function(error, drives) {
console.log("QUERY RETURNED");
console.log(drives);
parseDriveInfo(row.name, row.searchable, drives, cb);
});
}
else
var driveInfo = { "driveName" : row.name, "searchable" : row.searchable};
console.log("NO SUB ITEMS");
cb(null, driveInfo);
}, function(err, results) {
console.log("GEETTTTINGHERE");
console.log(results);
if(results.length == rows.length) {
console.log("RESULTS FOR THE DRIVE SEARCH");
console.log(results);
var response = {"id": req.params.agentId};
response.driveList = results;
console.log("RESPONSE");
console.log(response);
res.json(response);
}
});
});
});
};
parseDriveInfo = function(driveName, searchable, drives, cb) {
async.concatSeries(drives, function(drive,callback) {
console.log("SERIES 2");
console.log(drive);
console.log("END OF DRIVE INFO");
var query = "select name from supportedSearchTypes where logicalDriveId = " + drive.id;
connection.query(query, function(error, searchTypes) {
drive.searchTypes = searchTypes;
var driveInfo = { "driveName" :driveName,
"searchable" : searchable,
"logicalDrives" : drive
};
callback(null, driveInfo);
});
}, function (err, results) {
console.log("THIS IS ISISIS ISISISSISISISISISISISISISIS");
console.log(results);
if(results.length === drives.length) {
console.log("GOTHERE");
cb(null, results);
}
});
}

Getting good enough with async to use exactly the right combination of methods under the right circumstances takes a fair amount of experience. Most likely your case in particular can be handled with async.waterfall if its query1 then query2(dataFoundByQuery1) then query3(dataFoundByQuery2). But depending on the circumstances you need to mix and match async methods appropriately and sometimes have 2 levels - for example a "big picture" async.waterfall where some of the steps in the waterfall do async.parallel or async.series as needed. I've never used async.concat and given your needs I think you have chosen the wrong method. The workhorses are async.each, async.eachSeries, async.waterfall, and async.map, at least for the web app & DB query use cases I mostly encounter, so make sure you really have those understood before exploring the more specific convenience methods.

EDIT: This is a more in depth example based on use of the connection library you seem to be using. Please note, some of this is javascript psuedo code. Things like adding objects to the resultsArray are clearly not complete, the only thing I took time to make sure was correct is the "flow of logic" as it pertains to callbacks. Everything else is for you to implement. In order to support multiple calls to the same callback function and maintain state from call to call, the best way is to wrap the set of callbacks in a closure. This allows the callbacks to share some state with the main event loop. This allows you to pass arguments to the callbacks, without actually having to pass them as arguments, much like class variables in c++, or even globals in javascript, but we haven't poluted the global scope :)
function queryDataBase(query) {
//wrap the whole query in a function so the callbacks can share some
//variables with similar scope. This is called a closure
int rowCounter = 0;
var dataRowsFromStep2;
var resultsArray = {};
connection.query(query, dataBaseQueryStep2);
function dataBaseQueryStep2(err, rows) {
//do something with err and rows
dataRowsFromStep2 = rows;
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);//Always zero the first time. Might need to double check rows isn't empty!
connection.query(query, dataBaseQueryStep3);
}
function dataBaseQueryStep3(err, rows) {
//do something with err and rows
if(rowCounter < dataRowsFromStep2.size) {
resultsArray.add(rows);//Probably needs to be more interesting, but you get the idea
//since this is within the same closure, rowCounter maintains it's state
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);
//recursive call query using dataBaseQueryStep3 as it's callback repeatedly until
//we run out of rows to call it on.
connection.query(query, dataBaseQueryStep3)
} else {
//when the if statement fails we have no more rows to run queries on so return to main program flow
returnToMainProgramLogic(resultsArray);
}
}
}
function returnToMainProgramLogic(results) {
//continue running your program here
}
I personally like the above logic better than the syntax async produces... I believe the heart of your problem rests in your nested calls to async, and the fact that ASYN itself, runs the series of functions asynchronously, but in order(confusing I know). If you write your program like this, you won't have to worry about it!

I would strongly suggest using sequelize.js It provides a really powerful orm that allows you to chain queries together. It also allows you to directly load your data into js objects, write dynamic sql, and connect to many different databases. Picture ActiveRecord from the Ruby world for Node.

New to asynchronous programming in JavaScript. Any advice?

Asynchronous callbacks are great but when one callback depends on the result of another I have callbacks with api calls that have callbacks, and so on.
apiCall(function () { apiCall(function () { apiCall(function () ...
I can name the callback functions instead of including them inline. That looks prettier and has less nesting but I do not find it any easier to read.
Here is an example. I need to query the local sqlite database, use the result to query a server, then use the response to update the local database.
function sync() {
db.transaction(
function (transaction) {
execute(transaction, 'SELECT max(server_time) AS server_time FROM syncs;', [],
function (transaction, results) { // Query results callback
var t = results.rows.item(0).server_time;
$.post('sync.json', { last_sync_time: (t || '1980-01-01') },
function (data) { // Ajax callback
db.transaction(
function(transaction) {
$(data.thing).each(function () {
var thing = new Thing(this.thing);
thing.insert(transaction);
});
});
});
});
});
}
Is there a way to untagle this (other than naming the callbacks)?

I think you're too quick to discard un-nesting things by naming your functions rather than writing them inline. This is pretty much the only way to clean up that mess.
Instead of:
do_a(
function () {
// more nesting...
}
);
Use names to provide a bit of clarity and purpose to each function:
function on_a_complete() {
}
do_a(on_a_complete);

I think you answered your own question. Naming your callbacks is really the only way to clean this up anymore. Something like:
execute(transaction, 'SELECT max(server_time) AS server_time FROM syncs;', [],handleLocalResults, errorHandler);
handleLocalResults = function (transaction, results)...
handleServerResults = func...

Naming the callbacks is one thing you can do, but the other thing that you need to do is to have non-overlapping SQL transactions. (1)
Your first transaction should be like this:
// The whole thing starts here
db.transaction(selectTimeCB, null, ajaxPost);
The transaction starts with the callback to select the time, and when the transaction is complete, make a call to the ajaxPost operation.
// Initial transaction to get server_time
var selectTimeCB = function(t) {
var query = 'SELECT max(server_time) AS server_time FROM syncs';
t.executeSql(query, [], postLastSyncCB);
};
// This saves the results from the above select, and nothing else.
var server_time;
var postLastSyncCB = function(t, results) {
server_time = results.rows.item(0).server_time;
};
var ajaxPost = function() {
$.post('sync.json', { last_sync_time: (server_time || '1980-01-01') }, nextDbTransaction);
};
If you have overlapped SQL transactions, that can really kill the performance of the database. I recently did a test of 200 mixed transactions on a database with a bit over 500 rows, and found that keeping the transactions separate reduced a run time of over 90 seconds to 3-5 seconds.

Synchronous query to Web SQL Database

I'm working on a bit of JavaScript that interacts with a client-side SQLite database, via the newish window.openDatabase(...), database.transaction(...) and related APIs. As most of you know when you execute a query in this way it is an asynchronous call, which is typically good. You can make the call and handle the results as appropriate with callbacks.
In my current situation I'm working on an algo for a client that does some hierarchy walking in the locally stored database. The part of the algo I'm having trouble with requires starting at some row, which has a reference to a "parent" (by id) that is another row further up in the table. I have to keep walking up this tree until I reach the root.
The problem is that I'm at a point where I'm not sure how to use an asynchronous style query with a callback to keep feeding the loop parent ids. Ideally I could get the query to block so that I can do it all in the loop. Here's the key parts of my current setup:
for (i in search.searchResults.resultsArray)
{
hierarchyArr = new Array();
pageHierarchyArr = new Array();
id = search.searchResults.resultsArray[i].ID;
while (id != null && id != "")
{
var hierarchySql = "SELECT ID, parentID, type, content FROM content WHERE ID = " + id;
// This is a prettied up call to database.transaction(...)
var rs = db.getRS(hierarchySql);
// Ideally the code below doesn't execute until rs is populated
hierarchyArr.push(rs[0]);
if (rs[0].type == "page")
{
pageHierarchyArr.push(rs[0]);
// Do some additional work
}
id = rs[0].parentID;
}
}
As you might imagine, it doesn't work well. hierarchyArr gets an "undefined" pushed into it, and then the script crashes when it tries to check the type of rs[0].
When I try to set it up with a callback (db.getRSAndCallback(sql, callbackFunc), which I used for the earlier, non-interdependent queries just fine) it's worse: the inner loop takes off like crazy because id isn't getting updated; presumably because the loop is keeping the JavaScript interpreter so busy that it never actually fills rs. In some artificial testing where I forced the inner loop to break after a few iterations all the callbacks started coming through all at the end, after the loop finished.
The "standard" (such as it is right now) at http://dev.w3.org/html5/webdatabase/#synchronous-database-api seems to indicate that there is a synchronous API, but I haven't seen any sign of it on any WebKit based browsers.
Can anyone offer suggestions on how I might either, a. properly formulate these iterative, interdependent queries using callbacks or, b. somehow get the call to actually happen in a synchronous or apparently synchronous manner.
Many thanks in advance for anyone who takes a crack at this seemingly tricky little problem.
Naim
P.S. Here's the client's implementation of db.getRS for reference:
.
.
.
getRS: function(sql)
{
var output = [];
db.database.transaction(function(tx)
{
tx.executeSql(sql, [], function(tx,rs)
{
for(i = 0; i < rs.rows.length; i++)
{
output.push(rs.rows.item(i));
}
},
function(tx, error) { ... }
)});
return output;
},
.
.
.

I used callbacks and a closure to solve a similar problem, consider:
function getFolder(id, callback) {
var data = [];
ldb.transaction(function (tx) {
tx.executeSql('SELECT * FROM folders where id=?',
[id],
function (tx, results) {
if (results.rows && results.rows.length) {
for (i = 0; i < results.rows.length; i++) {
data.push(results.rows.item(i));
}
}
if (typeof(callback) == 'function')
callback(data);
},
function (tx, error) {
console.log(error);
});
});
}
In the continuation of this example, folder has a property parent to define it's relation to other folders. As does a document. The following will get you the path of a document using a closure (success):
function getDocPath(doc, callback) {
var path = [];
var parent = doc.parent;
var success = function(folder) {
var folder = folder[0];
parent = folder.parent;
path.push({'id':folder.id,'name':folder.name});
if (parent != "undefined")
getFolder(parent, success);
else
if ( typeof(callback) == 'function' ) callback(path.reverse());
}
getFolder(parent, success);
}

You could use callbacks with a closure to your stack of remaining queries. Or you could use recursion, passing the stack as parameters.