We Keep Coding

How do I wait for the first mysql connection query to resolve before the code moves to the second connection query in same func

Im trying to query a MySQL database and see if a record exists in a table
if it does then render page without inserting to a table
if it does not then call MySQL with another query to write to a table and then render page
What I believe is happening is that the first connection.query runs and before it renders the page when the record exists it tries to insert to table and errors with the below, maybe due to trying to render at the same time but not sure? Any help on solving this will be appreciated.
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client at ServerResponse.setHeader (_http_outgoing.js:558:11)
exports.follow = async (req, res) => {
try {
pool.getConnection(function (error, connection) {
if (error) {
console.log(error);
return;
}
connection.query(checkExists, async (error, results) => {
if (error)
throw error;
return res.status(200).render('search', {
});
})
connection.query(insertIfDoesNotExist, async (error, results) => {
if (error) throw error;
if (loggedin) {
return res.status(200).render('search', {
});
}
})
}
})
} catch (error) {
console.log(error);
}
}

You're right, connection.query() is asynchronous, so you've end up with race condition. checkExists and insertIfDoesNotExist will be queried synchronously, but it will only run its callback when it gets a reply from the database (this is the async part).
So most probably, you end up calling both call back, and trying to res.render twice, which is not correct. Each HTTP request can only have one response.
So how to solve this? You should nest your callback or use await (if you use a promise version of SQL driver) to something like this
exports.follow = async (req, res) => {
try {
pool.getConnection(function (error, connection) {
if (error) {
console.log(error);
return;
}
connection.query(checkExists, async (error, results) => {
if (error) throw error;
if (!results) // condition to check if it exists here!
// Only insert this after you've confirmed that it does not exists
connection.query(insertIfDoesNotExist, async (error, results) => {
if (error) throw error;
if (loggedin) {
return res.status(200).render('search', {});
}
});
return res.status(200).render('search', {});
});
});
} catch (error) {
console.log(error);
}
};

Event triggerred but not all the code is executed

i am working on a project on github named lan-info.
i have the following code
let arpSweepCommand = './arpSweep.sh';
app.get('/arp', (req, res) => {
console.log('we have a working signal!');
executeCommand(arpSweepCommand, req, res, false);
fs.readFile('command_output/arpedHosts.txt', 'utf-8', (error, data) => {
if (error) {
res.send(error);
} else {
res.send(data);
}
})
})
and arpSweep.sh contains:
timeout 10s netdiscover -P > command_output/arpedHosts.txt
and in my frontend i have a Jquery AJAX call:
arpButton.click(function () {
loader.show();
$.ajax({
method: 'GET',
url: '/arp',
success: function (data) {
console.log(data);
commandOutput.append(data);
loader.hide();
}
})
})
i know that there are no syntactical errors because webpack compiles the frontend code without complaints; and i know that the backend does catch the request as whenever i click arpButton it prints 'we have a signal!' on the server.
but the problem is loader.show()/hide() seem to do nothing in only this ajax request i know that this is specific to this ajax request because i have similar requests that function perfectly.
the problem for me is the information isn't automatically appended to commandOutput.
i need to reclick arpButton another time for the output to be appended to commandOutput.
but the loader wont show up for me i know that the loader isn't appearing and disappearing very fast because the ajax call takes at least 10 seconds to complete.
the other problem is the data doesn't appear automatically after 10 seconds in commandOutput i tested that only the loader is malfunctioning by waiting for 30 minutes after clicking arpButton but nothing happens; when i proceed to click arpButton again the output is shown instantly.
so why isnt the page updating itself? it instead forces me to reclick the button.
if you need more information you can click the project link above the files are:
src/index.js
index.js
arpSweep.sh
NOTE: be sure to change the subnet in the top of index.js if its not the subnet you use (automatic subnet detection is a feature i plan on implementing later).
EDIT: after trying to read the file as a callback for executeCommand:
function executeCommand (command, req, res, sendRes = true, callback) {
exec(command, (error, stdout, stderr) => {
console.log(`stdout: ${stdout}`)
console.log(`stderr: ${stderr}`)
if (error !== null) {
console.log(`Error ${error}`)
} else if (sendRes === false) {
} else if (typeof callback === 'function') {
callback()
} else {
res.send(stdout + stderr)
}
})
}
app.get('/arp', (req, res) => {
console.log('we have a working signal!')
executeCommand(arpSweepCommand, req, res, false, function () {
fs.readFile('command_output/arpedHosts.txt', 'utf-8', (error, data) => {
if (error !== null) {
res.send(error)
} else {
res.send(data)
}
})
})
})
i am encountering a new problem:
when arpButton is clicked now the loader shows up but after 10 seconds (the time allocated to the command in arpSweep.sh before netdiscover is killed) seems to make no difference and the loader seems to stay there forever now.
and on the server i am catching the following error.
error: Error can't execute command './arpSweep.sh'

the problem for me is the information isn't automatically appended to commandOutput
The issue you have is because the browser doesn't know how to append json (I assume it is not always json response).
Try to append responseText instead of data:
success: function (data, status, xhr) {
console.log(data)
commandOutput.append(xhr.responseText + '\r\n')
}
As of loader - everything OK with it, maybe you just need to hide it not on success, but on complete callback.
In this part:
executeCommand(arpSweepCommand, req, res, false)
fs.readFile('command_output/arpedHosts.txt', 'utf-8', (error, data) => {
You calling executeCommand asynchronously, so you can't be sure that the output is from the current call (that is probably the core of the issue when you need to click twice). So you should move reading file part of the code inside of the exec callback inside of the executeCommand method like this:
function executeCommand (command, req, res, cb) {
exec(command, (error, stdout, stderr) => {
console.log(`stdout: ${stdout}`)
console.log(`stderr: ${stderr}`)
if (error !== null) {
console.log(`Error ${error}`)
} else {
if (typeof cb === 'function') {
cb()
} else {
res.send(stdout + stderr)
}
}
}
}
And
executeCommand(arpSweepCommand, req, res, () => {
fs.readFile('command_output/arpedHosts.txt', 'utf-8', (error, data) => {
if (error) {
res.send(error)
} else {
res.send(data)
}
})
})

The problem is that executeCommand is asynchronous (it uses child_process.exec), so you are reading the file content BEFORE has been written, and that's why you need to click the button again to see the content in the next request. See the example here and how to make it sync-like with promisify and async/await: https://nodejs.org/api/child_process.html#child_process_child_process_exec_command_options_callback
Something like this (not tested):
const exec = util.promisify(require('child_process').exec);
async function executeCommand (command, req, res, sendRes = true) {
return exec(command, (error, stdout, stderr) => {
console.log(`stdout: ${stdout}`)
console.log(`stderr: ${stderr}`)
if (error !== null) {
console.log(`Error ${error}`)
} else if (sendRes === false) {
} else {
res.send(stdout + stderr)
}
})
}
and then, in your handler:
await executeCommand(arpSweepCommand, req, res, false);

Nodejs express and promises not doing what I expect

I am trying to build a login API using NodeJS, but my code is not doing what I expect it to. I am very new to js, promises and all so please simplify any answer if possible.
From what I can see in the output of my code, the first promise part does not wait until the function findUsers(...) is finished.
I have a routes file where I want to run a few functions sequentially:
Find if user exist in database
if(1 is true) Hash and salt the inputted password
... etc
The routes file now contains:
var loginM = require('../models/login');
var loginC = require('../controllers/login');
var Promise = require('promise');
module.exports = function(app) {
app.post('/login/', function(req, res, next) {
var promise = new Promise(function (resolve, reject) {
var rows = loginM.findUser(req.body, res);
if (rows.length > 0) {
console.log("Success");
resolve(rows);
} else {
console.log("Failed");
reject(reason);
}
});
promise.then(function(data) {
return new Promise(function (resolve, reject) {
loginC.doSomething(data);
if (success) {
console.log("Success 2");
resolve(data);
} else {
console.log("Failed 2");
reject(reason);
}
});
}, function (reason) {
console.log("error handler second");
});
});
}
And the findUser function contains pooling and a query and is in a models file:
var connection = require('../dbConnection');
var loginC = require('../controllers/login');
function Login() {
var me = this;
var pool = connection.getPool();
me.findUser = function(params, res) {
var username = params.username;
pool.getConnection(function (err, connection) {
console.log("Connection ");
if (err) {
console.log("ERROR 1 ");
res.send({"code": 100, "status": "Error in connection database"});
return;
}
connection.query('select Id, Name, Password from Users ' +
'where Users.Name = ?', [username], function (err, rows) {
connection.release();
if (!err) {
return rows;
} else {
return false;
}
});
//connection.on('error', function (err) {
// res.send({"code": 100, "status": "Error in connection database"});
// return;
//});
});
}
}
module.exports = new Login();
The output i get is:
Server listening on port 3000
Something is happening
error handler second
Connection
So what I want to know about this code is twofold:
Why is the first promise not waiting for findUser to return before proceeding with the if/else and what do I need to change for this to happen?
Why is error handler second outputed but not Failed?
I feel like there is something I am totally misunderstanding about promises.
I am grateful for any answer. Thanks.

Issues with the code
Ok, there are a lot of issues here so first things first.
connection.query('...', function (err, rows) {
connection.release();
if (!err) {
return rows;
} else {
return false;
}
});
This will not work because you are returning data to the caller, which is the database query that calls your callback with err and rows and doesn't care about the return value of your callback.
What you need to do is to call some other function or method when you have the rows or when you don't.
You are calling:
var rows = loginM.findUser(req.body, res);
and you expect to get the rows there, but you won't. What you'll get is undefined and you'll get it quicker than the database query is even started. It works like this:
me.findUser = function(params, res) {
// (1) you save the username in a variable
var username = params.username;
// (2) you pass a function to getConnection method
pool.getConnection(function (err, connection) {
console.log("Connection ");
if (err) {
console.log("ERROR 1 ");
res.send({"code": 100, "status": "Error in connection database"});
return;
}
connection.query('select Id, Name, Password from Users ' +
'where Users.Name = ?', [username], function (err, rows) {
connection.release();
if (!err) {
return rows;
} else {
return false;
}
});
//connection.on('error', function (err) {
// res.send({"code": 100, "status": "Error in connection database"});
// return;
//});
});
// (3) you end a function and implicitly return undefined
}
The pool.getConnection method returns immediately after you pass a function, before the connection to the database is even made. Then, after some time, that function that you passed to that method may get called, but it will be long after you already returned undefined to the code that wanted a value in:
var rows = loginM.findUser(req.body, res);
Instead of returning values from callbacks you need to call some other functions or methods from them (like some callbacks that you need to call, or a method to resolve a promise).
Returning a value is a synchronous concept and will not work for asynchronous code.
How promises should be used
Now, if your function returned a promise:
me.findUser = function(params, res) {
var username = params.username;
return new Promise(function (res, rej) {
pool.getConnection(function (err, connection) {
console.log("Connection ");
if (err) {
rej('db error');
} else {
connection.query('...', [username], function (err, rows) {
connection.release();
if (!err) {
res(rows);
} else {
rej('other error');
}
});
});
});
}
then you'll be able to use it in some other part of your code in a way like this:
app.post('/login/', function(req, res, next) {
var promise = new Promise(function (resolve, reject) {
// rows is a promise now:
var rows = loginM.findUser(req.body, res);
rows.then(function (rowsValue) {
console.log("Success");
resolve(rowsValue);
}).catch(function (err) {
console.log("Failed");
reject(err);
});
});
// ...
Explanation
In summary, if you are running an asynchronous operation - like a database query - then you can't have the value immediately like this:
var value = query();
because the server would need to block waiting for the database before it could execute the assignment - and this is what happens in every language with synchronous, blocking I/O (that's why you need to have threads in those languages so that other things can be done while that thread is blocked).
In Node you can either use a callback function that you pass to the asynchronous function to get called when it has data:
query(function (error, data) {
if (error) {
// we have error
} else {
// we have data
}
});
otherCode();
Or you can get a promise:
var promise = query();
promise.then(function (data) {
// we have data
}).catch(function (error) {
// we have error
});
otherCode();
But in both cases otherCode() will be run immediately after registering your callback or promise handlers, before the query has any data - that is no blocking has to be done.
Summary
The whole idea is that in an asynchronous, non-blocking, single-threaded environment like Node.JS you never do more than one thing at a time - but you can wait for a lot of things. But you don't just wait for something and do nothing while you're waiting, you schedule other things, wait for more things, and eventually you get called back when it's ready.
Actually I wrote a short story on Medium to illustrate that concept: Nonblacking I/O on the planet Asynchronia256/16 - A short story loosely based on uncertain facts.

Variable not retaining value outside of function

I am attempting to pass the data variable from the function to the app.get call.
But the global variable 'data' in my function is not retaining the value returned from the database query. I believe this to be a scoping issue. I set the value for 'data' to null at the beginning to make the variable accessible throughout the function. I am not hitting either of the error conditions.
function getCommands(query) {
var data = null;
try {
pg.connect(cString, function(err, client, done) {
// Catch any connection errors, and display them to the screen
if(err) {
return console.error('could not connect to postgres', err);
}
client.query(query, function(q_err, result) {
// Release the client back to the pool
done();
if(q_err) {
return console.error('error running query', q_err);
}
// Data object with an empty array to hold the row information
data = {"data":[]};
for (var row in result.rows)
{
//console.log(result.rows[row]);
data.data.push(result.rows[row]);
}
//Here, data, has the correct return values.
console.log(data);
});
});
}
catch( e )
{
console.log(e);
}
//Here, data, is null.
console.log(data);
return data;
}
app.get('/clients/', function(req, res) {
res.send(getCommands('SELECT clientid, clientname FROM hourglass.clients ORDER BY clientid ASC'));
});
Could someone help me determine why 'data' is not retaining value outside of the pg.connect function?

I think your problem is not that the variable is not retaining the value outside of the function, but rather that you console.log(data) is executing before the variable is set.
If you were to put console.log('step X') in your code as in the following example you'll see the sequence in which your code is executed.
function getCommands(query) {
var data = null;
console.log('STEP 1');
pg.connect(cString, function(err, client, done) {
console.log('STEP 3');
});
console.log('STEP 2');
}

function getCommands(query, callback) {
try {
pg.connect(cString, function(err, client, done) {
// Catch any connection errors, and display them to the screen
if(err) {
return console.error('could not connect to postgres', err);
}
client.query(query, function(q_err, result) {
// Release the client back to the pool
done();
if(q_err) {
return console.error('error running query', q_err);
}
// Data object with an empty array to hold the row information
var data = {"data":[]};
for (var row in result.rows)
{
data.data.push(result.rows[row]);
}
callback(data); //After data is set, the value is passed back
});
});
}
catch( e )
{
console.log(e);
}
}
app.get('/clients', function(req, res) {.....])
Using a callback function mentioned by #dystroy worked perfectly.

can't submit a form on express/angular app

in my program there's a validation function on it, if there's an error it will prevent the form to submit and display error msg else it will console.log("Success") but my form cannot be submitted even without any error. is there anyway to enable status code 200 when there is no error ? because now the form prevent me to submit because of status code 400
express
function validateSignup(data,callback) {
"use strict";
var USER_RE = /^[a-zA-Z0-9_-]{2,25}$/;
var PASS_RE = /^.{6,100}$/;
var EMAIL_RE = /^[\S]+#[\S]+\.[\S]+$/;
if (!USER_RE.test(data.publicUsername)) {
callback(new Error('Invalid Public Username try just letters and numbers, e.g: Ed, 69, Kelvin and etc'), null);
}
if (!PASS_RE.test(data.password)) {
callback(new Error('Password must be at least 6 characters long'), null);
}
if (data.password != data.confirmPassword) {
callback(new Error('Password must match'), null);
}
if (!EMAIL_RE.test(data.email)) {
callback(new Error('Invalid email address'), null);
}
if (data.email != data.confirmEmail) {
callback(new Error('Email must match'), null);
}
return true;
}
handlesignup
this.handleSignup = function(req, res, next) {
"use strict";
validateSignup(req.body, function(error, data) {
if(error) {
res.send(400, error.message);
} else {
console.log("success");
}
})
}
Angular
function RegisterCtrl($scope, $http, $location) {
$scope.form = {};
$scope.errorMessage = '';
$scope.submitPost = function() {
$http.post('/register', $scope.form).
success(function(data) {
$location.path('/');
}).error(function(err) {
$scope.errorMessage = err;
});
};
}

You have multiple issues in your code.
Your validateSignup function doesn't always call its callback. If the input passes the validation, it shouldn't return true but instead call its callback with no error and the data:
function validateSignup(data,callback) {
// ...
callback(null, data);
}
You don't always answer the client's request:
validateSignup(req.body, function(error, data) {
if(error) {
res.send(400, error.message);
} else {
console.log("success");
res.send(200);
}
})
Edit: As a side note, a callback should aways be called asynchronously (ie. using process.setImmediate, process.nextTick or setTimeout), but that isn't an issue in your specific case as the callback will always be called synchronously. As noted in Effective JS, item 67:
Never call an asynchronous callback synchronously, even if the data is immediately available.
That's why my advice is to always call callbacks asynchronously, which will free you from weird bugs later on. There are a number of reasons as why you shouldn't do it, but the most obvious is that you can easily blow the stack.
Here's how you you can defer the callback execution:
function validateSignup(data,callback) {
// ...
process.setImmediate(function() {
callback(null, data);
});
}