We Keep Coding

How to force sequential execution for asynchronously called functions

I'm new to the idea of asynchronous code, and am still trying to wrap my brain around how everything works.
I'm building a Node Express application which will interface with a database. When running in a development environment I want it to interface with a Sqlite database. (The production database will not use Sqlite. This only applies to creating a small development environment.)
My problem is I'm having trouble controlling the execution order and timing of queries to the database.
I would like to build my SqliteDatabase.js file such that it can only execute queries sequentially, despite the fact that functions in this file will be called by other parts of the program that are running asynchronously.
How can I acheive this?
For reference, here is how I currently have my SqliteDatabase.js file set up:
var debug = require('debug')('app:DATABASE');
var sqlite = require('sqlite3').verbose();
open = function() {
var db = new sqlite.Database('./test-database.db', sqlite.OPEN_READWRITE | sqlite.OPEN_CREATE, function(err) {
if (err) {
debug("We've encountered an error opening the sqlite database.");
debug(err);
} else {
debug("Sqlite database opened successfully.");
}
});
return db;
}
executeSQLUpdate = function(sql, next) {
var db = open();
db.serialize(function() {
console.log("executing " + sql);
db.run(sql);
db.close();
next();
});
}
exports.executeSQLUpdate = executeSQLUpdate;
Is there some way to build a queue, and make it so when the "executeSQLUpdate" function is called, the request is added to a queue, and is not started until all previous requests have been completed?
To give an example, take a look at this code which utilises my SqliteDatabase.js file:
ar database = require('../../bin/data_access/SqliteDatabase.js');
var createTestTableStmt = "CREATE TABLE IF NOT EXISTS Test(\n" +
"Name TEXT PRIMARY KEY NOT NULL UNIQUE,\n" +
"Age INT NOT NULL,\n" +
"Gender TEXT NOT NULL\n" +
");";
var clearTestTableStmt = "DELETE FROM Test;";
var testInsertStmt = "INSERT INTO Test (Name, Age, Gender)\n" +
"VALUES (\"Connor\", 23, \"Male\");";
createTable = function() {
database.executeSQLUpdate(createTestTableStmt, clearTable);
}
clearTable = function() {
database.executeSQLUpdate(clearTestTableStmt, insertRow);
}
insertRow = function() {
database.executeSQLUpdate(testInsertStmt, function() {
console.log("Done!");
});
}
createTable();
9 times out of 10 the above code works fine, but every once in a while, the "insert row" function is called before the "clearTable" function is called, which throws an error because of a violated database constraint.
How can I change my implementation of the SqliteDatabase.js file to avoid this issue?

You can use async to do this using await. This code will wait for each asynchronous database call to complete before executing the next line.
async function createTable() {
await database.executeSQLUpdate(createTestTableStmt);
await database.executeSQLUpdate(clearTestTableStmt);
await database.executeSQLUpdate(testInsertStmt);
console.log("Done!");
}
Your console.log statement will only execute once all three have completed.
I should also mention that you need a try...catch block around the three database calls to trap any errors and provide an alternate exit point if something should go wrong.

I realized why the callback function next() was sometimes being called before db.run(sql)
It turns out that db.run() is itself an asychronous function. I updated my code, and added a callback to the db.run() line to make sure we don't skip ahead until it's done.
Here's what it looks like now:
executeSQLUpdate = function(sql, next) {
var db = open();
db.run(sql, function(err) {
db.close(function() {
if (next) next(err);
});
});
}
Nesting each asynchronous function in the previous function's callback, makes each function execute in order.
Thanks to everyone who gave me hints that helped me figure out what the problem was.

Using findOne in a loop takes too long in Node.js

I'm using Node.js with MongoDB, I'm also using Monk for db access. I have the below code :
console.time("start");
collection.findOne({name: "jason"},
function(err, document) {
for(var i = 0; i < document.friends.length; i++) // "friends is an array contains ids of the user's friends"
{
collection.findOne({id: document.friends[i]}, function(err, doc)
{
console.log(doc.name);
});
}
});
console.log("The file was saved!");
console.timeEnd("start");
I have two questions regarding this code :
I see the execution time and "The file was saved!" string first, then I see the names of the friends coming in the console. Why is that? Shouldn't I see the names first then the execution time? Is it because the async nature of Node.js?
Names are printing very slowly in the console, the speed is like one name in two seconds. Why is it so slow? Is there a way to make the process faster?
EDIT:
Is it a good idea to break friends list to smaller pieces and call friends asynchronously? Would it make the process faster?
EDIT 2:
I changed my code to this :
collection.find({ id: { "$in": document.friends}}).then(function(err, doc)
{
console.log(doc.name);
if(err) {
return console.log(err);
}
}
This doesn't give an error, but this doesn't print anything either.
Thanks in advance.

Answer for question 1:
Yes, you are right.
Is it because the async nature of Node.js.
And to prevent that Node.js provides some mechanism for that you can use it otherwise you can do it on your own manually by setting one flag.
Answer for question 2:
you can use $in instead of findOne, it will be ease and fast.
e.g. .find({ "fieldx": { "$in": arr } })
arr :- In this you need to provide whole array.

yes, it's because javascript's async nature.
As you have called db from for loop javascript will not wait for it's response and continue the execution so it will print the file was saved first.
about your ans 2
It's making a dbCall for every friend then it's obvious that it will take some time that's why it's taking 1 or 2 secs for every friend.
console.time("start");
collection.findOne({name: "jason"},
function(err, document) {
for(var i = 0; i < document.friends.length; i++) // "friends is an array contains ids of the user's friends"
{
console.log("InsideforLoop Calling " + i + " friend");
collection.findOne({id: document.friends[i]}, function(err, doc)
{
console.log(doc.name);
});
console.log("Terminating " + i + "-----");
}
});
console.log("The file was saved!");
console.timeEnd("start");
This will make your async and db doubts more clear.
As you will see it will print all console in line.
InsideforLoop Calling 0 friend
Terminating 0 -----
and so on....Like this
console.log(doc.name);
but this will be printed asynchronusly
Added
collection.findOne({name: "jason"},
function(err, document) {
//you can do this
collection.find({id: $in:{document.friends}, function(err, doc)
{
console.log(doc);
});
});
Find All Details in one call

collection.aggregate([
{
$match:{
id :{ "$in" : document.friends},
}
}
]).exec(function ( e, d ) {
console.log( d )
if(!e){
// your code when got data successfully
}else{
// your code when you got the error
}
});

collection.findOne({name: "jason"},
function(err, document) {
if(document != undefined){
collection.find({ id: { "$in": document.friends}}).then(function(err, doc)
{
console.log(doc.name);
if(err) {
return console.log(err);
}
}
}
});

Answer to 1: Yes, it is because node is async. The part where it logs names is executed only when the first findOne returns, whereas the file was saved is executed straight away.

Inserting into Collection after Promises in a Meteor Method

I'm using this Gumroad-API npm package in order to fetch data from an external service (Gumroad). Unfortunately, it seems to use a .then() construct which can get a little unwieldy as you will find out below:
This is my meteor method:
Meteor.methods({
fetchGumroadData: () => {
const Gumroad = Meteor.npmRequire('gumroad-api');
let gumroad = new Gumroad({ token: Meteor.settings.gumroadAccessKey });
let before = "2099-12-04";
let after = "2014-12-04";
let page = 1;
let sales = [];
// Recursively defined to continue fetching the next page if it exists
let doThisAfterResponse = (response) => {
sales.push(response.sales);
if (response.next_page_url) {
page = page + 1;
gumroad.listSales(after, before, page).then(doThisAfterResponse);
} else {
let finalArray = R.unnest(sales);
console.log('result array length: ' + finalArray.length);
Meteor.call('insertSales', finalArray);
console.log('FINISHED');
}
}
gumroad.listSales(after, before, page).then(doThisAfterResponse); // run
}
});
Since the NPM package exposes the Gumorad API using something like this:
gumroad.listSales(after, before, page).then(callback)
I decided to do it recursively in order to grab all pages of data.
Let me try to re-cap what is happening here:
The journey starts on the last line of the code shown above.
The initial page is fetched, and doThisAfterResponse() is run for the first time.
We first dump the returned data into our sales array, and then we check if the response has given us a link to the next page (as an indication as to whether or not we're on the final page).
If so, we increment our page count and we make the API call again with the same function to handle the response again.
If not, this means we're at our final page. Now it's time to format the data using R.unnest and finally insert the finalArray of data into our database.
But a funny thing happens here. The entire execution halts at the Meteor.call() and I don't even get an error output to the server logs.
I even tried switching out the Meteor.call() for a simple: Sales.insert({text: 'testing'}) but the exact same behaviour is observed.
What I really need to do is to fetch the information and then store it into the database on the server. How can I make that happen?
EDIT: Please also see this other (much more simplified) SO question I made:
Calling a Meteor Method inside a Promise Callback [Halting w/o Error]

I ended up ditching the NPM package and writing my own API call. I could never figure out how to make my call inside the .then(). Here's the code:
fetchGumroadData: () => {
let sales = [];
const fetchData = (page = 1) => {
let options = {
data: {
access_token: Meteor.settings.gumroadAccessKey,
before: '2099-12-04',
after: '2014-12-04',
page: page,
}
};
HTTP.call('GET', 'https://api.gumroad.com/v2/sales', options, (err,res) => {
if (err) { // API call failed
console.log(err);
throw err;
} else { // API call successful
sales.push(...res.data.sales);
res.data.next_page_url ? fetchData(page + 1) : Meteor.call('addSalesFromAPI', sales);
}
});
};
fetchData(); // run the function to fetch data recursively
}

wd waitForElementBy... and asserter usage

I'm using the library provided here successfully in several tests, but am now stuck trying to use asserters as per their docs as a passed function in the waitForElementByLinkText API method.
Here is where I've defined the Asserter:
function Asserter(_assert){
this.assert = _assert;
}
/**
* asserters.isVisible
*
* #asserter
*/
var isDisplayed = new Asserter(
function(el,cb) {
el.isDisplayed(function(err, displayed) {
if(err) { return cb(err); }
cb(null, displayed);
});
}
);
module.exports = {
Asserter: Asserter,
isDisplayed: isDisplayed
};
Then in my chained script, I am attempting to use it as follows, but the console.log executes before the element is visible:
.get('http://mydomain.com/mypage')
.elementByLinkText('Reset', function(err, el){
browser.next('clickElement', el, noop);
})
.waitForElementByLinkText('This is the link text', isDisplayed, 10000, 100, function(err){
console.log('The page has updated!');
})
I believe my code is using a deprecated version of the chaining syntax which is needed to support legacy code coming out of SeBuilder, but makes it hard to follow the samples which all use the new method.

You don't need to redefine commonly used asserters, please refer to the new example here:
https://github.com/admc/wd/blob/master/examples/deprecated/wait-for-simple.js
If you need more help, please provide some html/js sample, otherwise it's hard to figure out what you are actually trying to achieve.

JavaScript Seems to Stop After AJAX Call

The question title is rather vague, but here's my situation. I have roughly 700+ lines of jQuery for a web application, each function and "major point of interest" in the code noted by a log to the console when it fires. For example, I have a few functions that use an AJAX call to a servlet to retrieve some information. I log when the AJAX request begins, if it's succeeded (then print what data it gathered), etc. So, by the look of what my console has logged when I open the page, it seems to stop after the first AJAX call. Granted, the call seemed to work just fine, and the data it returned was perfect. As you'll see, it even populated the select box as intended. However, the console logs stop shortly after, making me believe that for some reason, the other functions are not being called...
jQuery
$(document).ready(function() {
Initialize();
});
function Initialize() {
console.log("Initializing...");
User();
Widgets();
if($.cookie("fogbugzId") != null) {
console.log("Stored ID: " + $.cookie("fogbugzId"));
$("#userSelect").val($.cookie("fogbugzId")).trigger("change");
$("#userSelect").hide();
} else console.log("No ID Stored!");
}
function User() {
console.log("Initializing User...");
$.each(FetchUsers(), function(index, user) {
$("#userSelect").append($("<option>").val(user.id).text(user.name));
});
$("#userSelect").change(function() {
if($("#userSelect").val() != "") {
console.log("User Changed to " + $("#userSelect").val() + ": " + $("#userSelect").text());
$.cookie("fogbugzId", $("#userSelect").val(), { expires: 365 });
}
Update();
});
console.log("User Initialized!");
}
function FetchUsers() {
console.log("Loading Users...");
$("#loading").show();
$.get(servlet, { command: "getUsers" }, function(data) {
var users = new Array();
$(data).find("user").each(function() {
users.push({
id: $(this).find("id").text(),
name: $(this).find("name").text()
});
});
$.each(users, function(index, user) {
console.log(">> " + user.id + ": " + user.name);
});
console.log("Users Loaded!");
return(users);
}, "xml").complete(function() {
$("#loading").hide();
}).error(function() {
console.log("Loading Users Failed!");
});
}
function Widgets() {
console.log("Initializing Widgets...");
// More Code
console.log("Widgets Initialized!");
}
Console
Initializing...
Initializing User...
Loading Users...
>> 267: Alex Molthan
>> 35: Bill Brinkoetter
>> 100: Bob Yoder
>> 189: Brian Cutler
>> 559: Brian Ormond
>> 400: Corey Nakamura
Users Loaded!
But the logging stops right there. So the AJAX call to fetch the users from the database works fine, but apparently the User() function doesn't manage to finish properly. The only error that the JavaScript console gives me is one within my jquery.min.js file:
Uncaught TypeError: Cannot read property 'length' of undefined jquery.min.js:16
f.e.extend.each jquery.min.js:16
User modifytime.js:14
Initialize modifytime.js:3
(anonymous function) modifyTime.jsp:21
f.extend._Deferred.e.resolveWith jquery.min.js:16
f.e.extend.ready jquery.min.js:16
f.c.addEventListener.B jquery.min.js:16
It looks as though it is breaking on the $.each() that iterates through the array of users returned by the FetchUsers() function. I know the function returns usable array, so I'm not sure what it's getting stuck on. Can anyone see something I'm missing right off the bat? I tried assigning the users[] returned by the FetchUsers() function into a variable first, then passing that into the $.each(), but it still didn't work. Any suggestions?
Edit: After replacing the minified version of jQuery with the uncompressed version, it seems as though the array of users that I pass into the $.each() function has now .length property, which is why it's breaking. Just to check, before I call that particular $.each() function, I placed a log of the users[].length returned from the FetchUsers() function to see that it still had no .length property. I then went to the FetchUsers() function itself and placed a log of the users[].length just before I return it. This log, however, works perfectly fine (though my example doesn't show it, it returns 40 users). So is my users[] not being returned as an array or something?

FetchUsers does not return anything, it does not even have a return statement. Additionally, $.get is an asynchronous function, so you cannot return the value it passes to its callback from the FetchUsers function. Instead, you could make FetchUsers take a callback it calls when it has received data of a user (and in this case doing that change would be relatively trivial):
function User() {
console.log("Initializing User...");
FetchUsers(function(user) { // Changed!
$("#userSelect").append($("<option>").val(user.id).text(user.name));
});
<...>
}
function FetchUsers(callback) { // Changed!
console.log("Loading Users...");
$("#loading").show();
$.get(servlet, { command: "getUsers" }, function(data) {
//var users = new Array(); No longer necessary.
$(data).find("user").each(function() {
callback({ // Changed!
id: $(this).find("id").text(),
name: $(this).find("name").text()
});
});
<...>
}
Changing those three lines with the "Changed!" comment should be enough to make it work correctly. (Though your logging of the users gotten will need to be slightly altered as they are no longer pushed into an array.)

I confess I have not read and understood every part of your source (nor checked if all the braces are closed), but FetchUsers clearly does NOT return anything (contrary to your claim) - so a call to FetchUsers() evaluates to 'undefined'. Fixing it will require some rewriting as in Javascript you cannot really return a result of asynchronous operation (like $.get) from a synchronous function (like FetchUsers()) - this would require multithreading (some kind of blocking, waiting etc).