We Keep Coding

Return value inside a collection is returned as undefined but works as a response send

I have the following GET call which works as intended, connecting to a couchbase db and performing some updates.
databaseRouter.put('/update/:id', (req, res) => {
updateDocument(req, res);
});
export const updateDocument = (req, res) => {
collection.get(req.params.id, (err, result) => {
if (err) {
res.status(404).send(err);
} else {
const document = result.value;
document.product_id = req.body.id || document.product_id;
collection.replace(req.params.id, document, (err) => {
if (err) {
res.status(500).send(err);
}
}).then(() => res.json(document));
}
}).catch(e => console.log(e));
}
This is for external clients to use.
But I want this to logic to be reusable within the project in another instance for batch processing. Not a rest call.
Thus I am looking to refactor the updateDocument function to return the document value or errors instead of performing res.send();
But I can't just modify as follows. result is undefined.
And I am also not gonna be able to maintain the status codes for errors.
Unless I explicitly return like a object with a key called status.
export const updateDocument = (req, res) => {
.....
}).then(() => document); // instead of }).then(() => res.json(document));
.....
}
databaseRouter.put('/update/:id', (req, res) => {
const result = updateDocument(req, res); // result is undefined
res.send(result);
});
Is there a way I could elegantly extract the logic so that I can continue to achieve what I have for the GET call for clients
but also be able to reuse the same logic internally within the project?

Node.js MSSQL package query function not returning data

I'm writing my first Node.js REST API. I'm using MS SQL Server as my database. I am using the npm package mssql to work with my SQL server DB.
I took the code below directly from the mssql docs example page. I simply wrapped it into a function.
function getServices() {
sql
.connect(config)
.then((pool) => {
// Query
return pool
.request()
.input("SID", sql.Int, 1)
.query(
"select * from [dbo].[Services] where ServiceId = #SID"
);
})
.then((result) => {
//console.dir(result); //this has data.
return result;
})
.catch((err) => {
console.log(err);
return err;
});
}
The above code works just fine and gets the data from my DB. The issue happens when I try to make this code into a function that I can use on my express route, as shown below.
router.get("/", (req, res, next) => {
const data = getServices();
console.log("data: ", data); //this comes back as undefined
res.status(200).json({
message: "Handling GET request from /services router.",
});
});
From reading the docs and all the others posts on stackoverflow. I am using the .then() promise mechanism, so why is my "result" not getting back to the function on my express route? The "data" const on the express route is always undefined. What am I missing here?

Rule of thump: calling a function const data = getServices(); means that this function doesn't contain any asynchronous work like async/await/.then/.catch/Promise.
Once a function containes one of those, it should always be called with one of the above.
SO #1 you must change
router.get("/", async (req, res, next) => {
const data = await getServices();
console.log("data: ", data); //this comes back as undefined
res.status(200).json({
message: "Handling GET request from /services router.",
});
});
Then the function itself
function getServices() {
try {
return sql
.connect(config)
.then((pool) => {
// Query
return pool
.request()
.input("SID", sql.Int, 1)
.query(
"select * from [dbo].[Services] where ServiceId = #SID"
);
})
.then((result) => {
//console.dir(result); //this has data.
return result;
})
} catch(ex){
return ex;
}
}

I kept digging through some more SO posts and found my working solution. Updated, working code is posted below.
router.get("/", (req, res, next) => {
getServices().then((result) => {
res.status(200).json(result.recordset);
});
});
function getServices() {
return new Promise((resolve) => {
sql
.connect(config)
.then((pool) => {
// Query
return pool
.request()
.input("SID", sql.Int, 1)
.query(
"select * from [dbo].[Services] where ServiceId = #SID"
);
})
.then((result) => {
//console.dir(result);
resolve(result);
})
.catch((err) => {
console.log(err);
resolve(err);
});
});
}

How to run request sequentially in a loop?

This is a pseudo code of what I am trying to achieve. First I need to get a list of URLs from the request body then pass those URLs to request function (using request module) which will get the data from each url and then save those data to MongoDB. After all the requests are finished including saving data to the server only then it should send a response.
app.post('/', (req, resp) => {
const { urls } = req.body;
urls.forEach((url, i) => {
request(url, function (err, resp, body) {
if (err) {
console.log('Error: ', err)
} else {
// function to save data to MongoDB server
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
})
});
// Should be called after all data saved from for loop
resp.send('All data saved')
})
I have tried this code and of course the resp.send() function will run without caring if the request has completed. Using this code I get a result on the console like this:
Data saved for URL number - 3
Data saved for URL number - 1
Data saved for URL number - 5
Data saved for URL number - 2
Data saved for URL number - 4
I could write them in nested form but the variable urlscan have any number of urls and that's why it needs to be in the loop at least from my understanding. I want the requests to run sequentially i.e. it should resolve 1st url and then second and so on and when all urls are done only then it should respond. Please help!

app.post('/', async (req, resp) => {
const {
urls
} = req.body;
for (const url of urls) {
try {
const result = await doRequest(url)
console.log(result)
} catch (error) {
// do error processing here
console.log('Error: ', err)
}
}
})
function doRequest(url) {
return new Promise((resolve, reject) => {
request(url, function(err, resp, body) {
err ? reject(err) ? resolve(body)
})
})
}
using async await

You should look at JavaScript Promises
Otherwise, you can do a recursive request like so:
app.post('/', (req, resp) => {
const { urls } = req.body;
sendRequest(urls, 0);
})
function sendRequest(urlArr, i){
request(urlArr[i], function (err, resp, body) {
if (err) {
console.log('Error: ', err)
}
else {
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
i++;
if(i == urlArr.length) resp.send('All data saved') //finish
else sendRequest(urlArr, i); //send another request
})
}
All I had to do is create a separate function I can call over and over again, passing the url array and a base index 0 as arguments. Each success callback increments the index variable which I pass in the same function again. Rinse and repeat until my index hits the length of the url array, I'll stop the recursive loop from there.

You want to wait till all api response you get and stored in db, so you should do async-await and promisify all the response.
You can use Request-Promise module instead of request. So you will get promise on every requested api call instead of callback.
And use promise.all for pushing up all request(module) call inside array.
Using async-await you code execution will wait till all api call get response and stored in db.
const rp = require('request-promise');
app.post('/', async (req, res) => {
try{
const { urls } = req.body;
// completed all will have all the api resonse.
const completedAll = await sendRequest(urls);
// now we have all api response that needs to be saved
// completedAll is array
const saved = await saveAllData(completedAll);
// Should be called after all data saved from for loop
res.status(200).send('All data saved')
}
catch(err) {
res.status(500).send({msg: Internal_server_error})
}
})
function sendRequest(urlArr, i){
const apiCalls = [];
for(let i=0;i < urlArr.length; i++){
apiCalls.push(rp(urlArr[i]));
}
// promise.all will give all api response in order as we pushed api call
return Promise.all(apiCalls);
}
You can refer these links:
https://www.npmjs.com/package/request-promise
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

Looking at the intention(a crawler) you can use Promise.all because the urls are not dependant upon each other.
app.post('/', (req, resp) => {
const { urls } = req.body;
const promises = urls.map((url, i) => {
return new Promise((resolve, rej)=>{
request(url, function (err, resp, body) {
if (err) {
rej(err);
} else {
resolve(body);
}
})
})
.then((body)=>{
//this should definitely be a promise as you are saving data to mongo
return saveUrlData(body);
})
});
// Should be called after all data saved from for loop
Promise.all(promises).then(()=>resp.send('All data saved'));
})
Note: Need to do error handling as well.

there are multiple ways to solve this.
you can use async/await
Promises
you can also use the async library
app.post('/', (req, res, next) => {
const { urls } = req.body;
async.each(urls, get_n_save, err => {
if (err) return next(err);
res.send('All data saved');
});
function get_n_save (url, callback) {
request(url, (err, resp, body) => {
if (err) {
return callback(err);
}
saveUrlData(body);
callback();
});
}
});

Node.js res.render() with Request not working

I am trying to render an EJS template and pass in data to it with the node package Request. I have got this to work with no problem using node-fetch on my last project.
Here's a quick snippet of that:
const fetchCurrentWeather = (url, res) => {
fetch(url)
.then(res => res.json())
.then(data => res.render('pages/weather', {
currentId: data.list
}))
.catch(err => {
console.log(err);
res.sendStatus(500);
});
}
And this will render the EJS weather page template with the data from the API response.
With this latest project however, I am trying to use the Node Package Request to do the same thing and I am failing. Here's the code for that:
app.post('/', (req, res, next) => {
const paramOptions = {
url: 'https://api.foursquare.com/v2/venues/search',
method: 'GET',
qs: {
client_id: 'W033PDIYFI2TSAUO4L5ANUFOAUMZV32NUWNOF0NL0JS2E5W4',
client_secret: 'SM1ZI11XOMPVMEGMBZXSN3LGTLOBQCVGIM1SN4QAO0QTCSM1',
near: 'xxxxxx',
intent: 'browse',
radius: '15000',
query: 'pizza',
v: '20170801',
limit: 1
}
};
request(paramOptions, function(err, res, body) {
if(err) {
console.log(err)
} else {
res.render('pages/search'); //THIS WILL NOT WORK
console.log(body) // RETURNS DATA FROM API ENDPOINT
}
});
res.render('pages/search'); // THIS WILL WORK
});
The res.render() that does work is outside of the scope of the Request function, so I can not access the returned data from this Request function. When I console.log the body of the Request function, I do get JSON data returned in my terminal view, so I know Request is working, but I am unsure of how to pass this data into an EJS template as I did with my fetch() example.

There are two separate res parameters and the inner one is hiding the outer one. Change the name of the one in this line:
request(paramOptions, function(err, res, body) {
to something else like this:
request(paramOptions, function(err, response, body) {
And, then when you call res.render(), it will use the higher scoped one you want.
You probably also need to pass some data to res.render(filename, data) as the second argument so you can feed the results of the request() to your render operation.
And, in your if (err) condition, you should send an error status, perhaps res.sendStatus(500).

Creating error/result function in node

I'm having trouble understanding how to create functions that would return in the format of (err, result) for an Express app.
My current db query function is:
pool.query(
'SELECT id FROM users WHERE email = ? LIMIT 1',
[email],
(results) => { // I'd like this to be (err, results)
if(results instanceof Error){...}
}
})
In my db.js file, pool looks like this:
module.exports = {
query: (query, args, cb) => {
pool.getConnection( (err, connection) => {
if(err){
new Error('No database connections available in pool')
} else {
connection.query(query, args, (error, results, fields) => {
connection.release()
// I got a MySQL error here and I'd like to handle it in my callback function
if(error){
new Error('Bad query')
} else {
cb(results)
}
})
}
})
}
}
For this and other functions, I'd like to return a proper Error if there is one, and have my callback listen for err, result as parameters.
I tried using new Error('Bad query') but that came back as the first variable in my callback no matter what (which is how I ended up with instanceof Error.
How do you structure a callback and response so that your callback can be in the err, result format and check for/handle errors properly on functions you're creating? (I understand how to use it for modules already in this format - I'm talking about writing/formatting your own code.)
Thanks!

You can do it like this:
module.exports = {
query: (query, args, cb) => {
pool.getConnection( (err, connection) => {
if(err){
cb(new Error('No database connections available in pool'));
} else {
connection.query(query, args, (error, results, fields) => {
connection.release();
// I got a MySQL error here and I'd like to handle it in my callback function
if(error){
cb(new Error('Bad query'));
} else {
cb(null, results);
}
});
}
});
}
}
You always pass the error value as the first argument to the callback and, if there is a result, you pass it as the second. Then, within the callback, you check to see if err is non-null and, if so, there is an error. If it's null, then the second argument contains the result.
Note that by not returning or including the actual err value that the database gave you, you may be hiding useful information (like why the query failed).
Then, where you use this, you do something like this:
let query = 'SELECT id FROM users WHERE email = ? LIMIT 1';
pool.query(query, [email], (err, result) => {
if (err) {
// handle error here
} else {
// process result here
}
});