We Keep Coding

where to should write res.send() in node.js app

In my app I have a category collection that saves the category title with its image ID. The image is saved in another collection with its meta data like path , type and so on. So for retrieving category I should retrieve category image in image collection by its ID and add image path to category object that is retrieved from category collection and send it to client...But I don't know where should I send categories to client?
When I send the response face this error :
throw er; // Unhandled 'error' event
^
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at ServerResponse.setHeader (_http_outgoing.js:561:11)
at ServerResponse.header (H:\node.js\online-store\app\node_modules\express\lib\response.js:771:10)
at ServerResponse.send (H:\node.js\online-store\app\node_modules\express\lib\response.js:170:12)
at ServerResponse.json (H:\node.js\online-store\app\node_modules\express\lib\response.js:267:15)
at ServerResponse.send (H:\node.js\online-store\app\node_modules\express\lib\response.js:158:21)
at H:\node.js\online-store\app\controllers\pcategory.controller.js:123:19
at H:\node.js\online-store\app\node_modules\mongoose\lib\model.js:4845:18
at processTicksAndRejections (internal/process/task_queues.js:77:11)
Emitted 'error' event on Function instance at:
at H:\node.js\online-store\app\node_modules\mongoose\lib\model.js:4847:15
at processTicksAndRejections (internal/process/task_queues.js:77:11) {
code: 'ERR_HTTP_HEADERS_SENT'
}
This is my code :
exports.getAll = async (req, res) => {
try{
const categories = await ProductCategory.find({});
categories.map(async(category)=>{
await File.findById(category.imageID).exec(function(err,file){
if(err){
console.log(err)
}else if(file) {
category.imagePath = file.file_path;
tempCategories.push(category)
}
res.send(tempCategories);
})
})
return res.send(tempCategories);
}catch {
res.json(err =>{
console.log(err);
res.status(500).send({
message:
err.message || "There is an error in retrieving category"
});
})
}
}

The problem is that nothing in your code is waiting for the asynchronous operations you're doing in your map callback to complete, so it does the res.send at the end right away — and then does res.send again within the map callback later when the async operations complete. Instead, wait for them to finish and send the result.
Also, you're using res.send where I suspect you want res.json, and using res.json later incorrectly (it doesn't take a callback).
See comments:
exports.getAll = async (req, res) => {
try {
// Get the categories
const categories = await ProductCategory.find({});
// Get the files for the categories, wait for the result
const result = await Promise.all(categories.map(async (category) => {
const file = await File.findById(category.imageID).exec();
// You probably can't modify the `category` object, so let's create
// and return a new object
return {...category, imagePath: file.file_path};
}));
// Send the result converted to JSON
return res.json(tempCategories);
} catch (err) { // Accept the error
// Send an error response
res.status(500).json({
message: err.message || "There is an error in retrieving category"
});
}
};
Side note: Your original code was using map without using the array it creates. That's an antipattern (sadly it seems to be one someone somewhere is teaching). I wrote up why and what to do instead here. (In my update to your code, I still use map, but I use the array it creates, passing it to Promise.all so we can wait for all those promises to settle.)

Your Code Like this,
Now Issue is You are sending two times Headers.
You can use like this, Firstly Declare array and push into it what you need and then last of your logic return it or send it.
exports.getAll = async (req, res) => {
try {
const categories = await ProductCategory.find({});
let tempCategories = []; // New Line
await Promise.all(categories.map(async (category) => {
await File.findById(category.imageID).exec(function (err, file) {
if (err) {
console.log(err)
} else if (file) {
category.imagePath = file.file_path;
tempCategories.push(category)
}
});
return category;
}));
res.send(tempCategories);
} catch {
res.json(err => {
console.log(err);
res.status(500).send({
message:
err.message || "There is an error in retrieving category"
});
})
}
}

Node.js MSSQL package query function not returning data

I'm writing my first Node.js REST API. I'm using MS SQL Server as my database. I am using the npm package mssql to work with my SQL server DB.
I took the code below directly from the mssql docs example page. I simply wrapped it into a function.
function getServices() {
sql
.connect(config)
.then((pool) => {
// Query
return pool
.request()
.input("SID", sql.Int, 1)
.query(
"select * from [dbo].[Services] where ServiceId = #SID"
);
})
.then((result) => {
//console.dir(result); //this has data.
return result;
})
.catch((err) => {
console.log(err);
return err;
});
}
The above code works just fine and gets the data from my DB. The issue happens when I try to make this code into a function that I can use on my express route, as shown below.
router.get("/", (req, res, next) => {
const data = getServices();
console.log("data: ", data); //this comes back as undefined
res.status(200).json({
message: "Handling GET request from /services router.",
});
});
From reading the docs and all the others posts on stackoverflow. I am using the .then() promise mechanism, so why is my "result" not getting back to the function on my express route? The "data" const on the express route is always undefined. What am I missing here?

Rule of thump: calling a function const data = getServices(); means that this function doesn't contain any asynchronous work like async/await/.then/.catch/Promise.
Once a function containes one of those, it should always be called with one of the above.
SO #1 you must change
router.get("/", async (req, res, next) => {
const data = await getServices();
console.log("data: ", data); //this comes back as undefined
res.status(200).json({
message: "Handling GET request from /services router.",
});
});
Then the function itself
function getServices() {
try {
return sql
.connect(config)
.then((pool) => {
// Query
return pool
.request()
.input("SID", sql.Int, 1)
.query(
"select * from [dbo].[Services] where ServiceId = #SID"
);
})
.then((result) => {
//console.dir(result); //this has data.
return result;
})
} catch(ex){
return ex;
}
}

I kept digging through some more SO posts and found my working solution. Updated, working code is posted below.
router.get("/", (req, res, next) => {
getServices().then((result) => {
res.status(200).json(result.recordset);
});
});
function getServices() {
return new Promise((resolve) => {
sql
.connect(config)
.then((pool) => {
// Query
return pool
.request()
.input("SID", sql.Int, 1)
.query(
"select * from [dbo].[Services] where ServiceId = #SID"
);
})
.then((result) => {
//console.dir(result);
resolve(result);
})
.catch((err) => {
console.log(err);
resolve(err);
});
});
}

How to run request sequentially in a loop?

This is a pseudo code of what I am trying to achieve. First I need to get a list of URLs from the request body then pass those URLs to request function (using request module) which will get the data from each url and then save those data to MongoDB. After all the requests are finished including saving data to the server only then it should send a response.
app.post('/', (req, resp) => {
const { urls } = req.body;
urls.forEach((url, i) => {
request(url, function (err, resp, body) {
if (err) {
console.log('Error: ', err)
} else {
// function to save data to MongoDB server
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
})
});
// Should be called after all data saved from for loop
resp.send('All data saved')
})
I have tried this code and of course the resp.send() function will run without caring if the request has completed. Using this code I get a result on the console like this:
Data saved for URL number - 3
Data saved for URL number - 1
Data saved for URL number - 5
Data saved for URL number - 2
Data saved for URL number - 4
I could write them in nested form but the variable urlscan have any number of urls and that's why it needs to be in the loop at least from my understanding. I want the requests to run sequentially i.e. it should resolve 1st url and then second and so on and when all urls are done only then it should respond. Please help!

app.post('/', async (req, resp) => {
const {
urls
} = req.body;
for (const url of urls) {
try {
const result = await doRequest(url)
console.log(result)
} catch (error) {
// do error processing here
console.log('Error: ', err)
}
}
})
function doRequest(url) {
return new Promise((resolve, reject) => {
request(url, function(err, resp, body) {
err ? reject(err) ? resolve(body)
})
})
}
using async await

You should look at JavaScript Promises
Otherwise, you can do a recursive request like so:
app.post('/', (req, resp) => {
const { urls } = req.body;
sendRequest(urls, 0);
})
function sendRequest(urlArr, i){
request(urlArr[i], function (err, resp, body) {
if (err) {
console.log('Error: ', err)
}
else {
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
i++;
if(i == urlArr.length) resp.send('All data saved') //finish
else sendRequest(urlArr, i); //send another request
})
}
All I had to do is create a separate function I can call over and over again, passing the url array and a base index 0 as arguments. Each success callback increments the index variable which I pass in the same function again. Rinse and repeat until my index hits the length of the url array, I'll stop the recursive loop from there.

You want to wait till all api response you get and stored in db, so you should do async-await and promisify all the response.
You can use Request-Promise module instead of request. So you will get promise on every requested api call instead of callback.
And use promise.all for pushing up all request(module) call inside array.
Using async-await you code execution will wait till all api call get response and stored in db.
const rp = require('request-promise');
app.post('/', async (req, res) => {
try{
const { urls } = req.body;
// completed all will have all the api resonse.
const completedAll = await sendRequest(urls);
// now we have all api response that needs to be saved
// completedAll is array
const saved = await saveAllData(completedAll);
// Should be called after all data saved from for loop
res.status(200).send('All data saved')
}
catch(err) {
res.status(500).send({msg: Internal_server_error})
}
})
function sendRequest(urlArr, i){
const apiCalls = [];
for(let i=0;i < urlArr.length; i++){
apiCalls.push(rp(urlArr[i]));
}
// promise.all will give all api response in order as we pushed api call
return Promise.all(apiCalls);
}
You can refer these links:
https://www.npmjs.com/package/request-promise
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

Looking at the intention(a crawler) you can use Promise.all because the urls are not dependant upon each other.
app.post('/', (req, resp) => {
const { urls } = req.body;
const promises = urls.map((url, i) => {
return new Promise((resolve, rej)=>{
request(url, function (err, resp, body) {
if (err) {
rej(err);
} else {
resolve(body);
}
})
})
.then((body)=>{
//this should definitely be a promise as you are saving data to mongo
return saveUrlData(body);
})
});
// Should be called after all data saved from for loop
Promise.all(promises).then(()=>resp.send('All data saved'));
})
Note: Need to do error handling as well.

there are multiple ways to solve this.
you can use async/await
Promises
you can also use the async library
app.post('/', (req, res, next) => {
const { urls } = req.body;
async.each(urls, get_n_save, err => {
if (err) return next(err);
res.send('All data saved');
});
function get_n_save (url, callback) {
request(url, (err, resp, body) => {
if (err) {
return callback(err);
}
saveUrlData(body);
callback();
});
}
});

How catch error in the front-end from response of expressjs?

My problem is the next:
//express server
app.post('/register', (req, res) => {
const {
password,
passwordConfirm
} = req.body;
if (password === passwordConfirm) {
//...
} else {
res.status(400).json("Passwords aren't matching")
}
})
//react function
onSubmitSignIn = () => {
const { password, passwordConfirm } = this.state;
let data = new FormData();
data.append('password', password);
data.append('passwordConfirm', passwordConfirm);
fetch('http://localhost:3001/register', {
method: 'post',
body: data
})
.then(response => response.json())
.then(user => {
//logs error message here
console.log(user)
})
//but I want to catch it here, and set the message to the state
.catch(alert => this.setState({alert}))
}
When I send the status code, and the message from express as the response, the front-end obviously recognize it as the response, that's why it logs the message to the console as "user". But how to send error which goes to the catch function?

fetch will really only error if it for some reason can't reason the API. In other words it will error on network errors. It will not explicitly error for non 2XX status codes.
You need to check the ok property as described here:
https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#Checking_that_the_fetch_was_successful
https://developer.mozilla.org/en-US/docs/Web/API/Response/ok
--
fetch('http://localhost:3001/register', {
method: 'post',
body: data
})
.then(response => {
if (!response.ok) {
throw new Error('my api returned an error')
}
return response.json()
})
.then(user => {
console.log(user)
})
.catch(alert => this.setState({alert}))

The problem is, that fetch is not recognizing the HTTP errors as Promise rejections.
The Promise returned from fetch() won't reject on HTTP error status even if the response is an HTTP 404 or 500. Instead, it will resolve normally, and it will only reject on network failure or if anything prevented the request from completing.
(Source)
You can checkout the linked source of the fetch repo which also states a suggestion for handling HTTP error statuses.

What if you throw an error :
app.get("/", function (req, res) {
throw new Error("BROKEN"); // Express will catch this on its own.
});
And then catch this error in the front end ?
See here for reference
EDIT
Maybe should you return the error with return next() so that the rest of the code is not processed in the server method :
app.get("/", function (req, res) {
return next(new Error('BROKEN'));
});

//express server
app.post('/register', (req, res) => {
try {
const {
password,
passwordConfirm
} = req.body;
if (password === passwordConfirm) {
//...
} else {
res.status(400).json("Passwords aren't matching")
}
} catch (error) {
res.status(500).send(error);
}
})
//react function
onSubmitSignIn = () => {
const {
password,
passwordConfirm
} = this.state;
let data = new FormData();
data.append('password', password);
data.append('passwordConfirm', passwordConfirm);
fetch('http://localhost:3001/register', {
method: 'post',
body: data
})
.then(response => response.json())
.then(user => {
//logs error message here
console.log(user)
})
//but I want to catch it here, and set the message to the state
.catch(alert => this.setState({
alert
}))
}

Node.js / Express.js can't get body response from API call via Request middleware

I'm trying to learn Node.js via Express.js framework. Currently I need to make an API call to get some data usefull for my app.
The API call is made with Request middleware, but when I'm out of the request my variable become undefined ... Let me show you :
var request = require('request');
var apiKey = "FOOFOO-FOOFOO-FOO-FOO-FOOFOO-FOOFOO";
var characters = [];
var gw2data;
var i = 0;
module.exports.account = function() {
request('https://api.guildwars2.com/v2/characters/?access_token=' + apiKey, function (error, response, body) {
gw2data = JSON.parse(body);
console.log('out request ' + gw2data); // {name1, name2 ...}
for (i; i < gw2data.length; i++) {
getCharacInfo(gw2data[i], i);
}
});
console.log('out request ' + characters); // undefined
return characters;
};
function getCharacInfo (name, position) {
request('https://api.guildwars2.com/v2/characters/' + name + '/?access_token=' + apiKey, function (error, response, body) {
if (!error && response.statusCode == 200) {
characters[position] = JSON.parse(body);
}
});
}
I don't understand why the gw2data variable become undefined when I go out of the request ... Someone can explain me ?
EDIT : I come to you because my problem has changed, I now need to make an API call loop in my first API call, same async problem I guess.
The previous code has evoluate with previous answers :
module.exports.account = function(cb) {
var request = require('request');
var apiKey = "FOOFOO-FOOFOO-FOO-FOO-FOOFOO-FOOFOO";
var characters = [];
var i = 0;
request('https://api.guildwars2.com/v2/characters/?access_token=' + apiKey, function(err, res, body) {
var numCompletedCalls = 1;
for (i; i < JSON.parse(body).length; i++) {
if (numCompletedCalls == JSON.parse(body).length) {
try {
console.log(characters); // {} empty array
return cb(null, characters);
} catch (e) {
return cb(e);
}
}else {
getCharacInfo(JSON.parse(body)[i]);
}
numCompletedCalls++;
}
});
};
function getCharacInfo (name) {
request('https://api.guildwars2.com/v2/characters/' + name + '/?access_token=' + apiKey, function (err, res, body) {
if (!err) {
console.log(characters); // {data ...}
characters.push(JSON.parse(body));
}
});
}
And the call of this function :
var express = require('express');
var router = express.Router();
router.get('/', function(req, res, next) {
var charactersInfos = require('../models/account');
charactersInfos.account(function(err, gw2data) {
if (err) {
console.error('request failed:', err.message);
} else {
res.render('index', {
persos : gw2data
});
}
});
});
The problem is, when I return the characters array, it's always an empty array, but when I check in my function getCharacInfo, the array contains the data ...

The reason gw2data is undefined in the second console.log is because your logging it too early. request is an asynchronous operation therefore it will return immediately...however, that doesn't mean the callback will.
So basically what your doing is logging gw2data before it's actually been set. When dealing with asynchronous operations the best approach is to generally make your own method asynchronous as well which can be accomplished in a couple of ways - the simplest being having your function accept a callback which expects the data in an asynchronous way e.g.
module.exports.account = function(cb) {
request(..., function(err, res, body) {
// return an error if `request` fails
if (err) return cb(err);
try {
return cb(null, JSON.parse(body));
} catch (e) {
// return an error if `JSON.parse` fails
return cb(e);
}
});
}
...
var myModule = require('mymodule');
myModule.account(function(err, g2wdata) {
if (err) {
console.error('request failed', err.message);
} else {
console.log('out request', g2wdata);
}
});
Based on your syntax I'm assuming you aren't working with ES6, worth looking at this (particularly if your just starting to learn). With built-in promises & also async-await support coming relatively soon this sort of stuff becomes relatively straightforward (at least at the calling end) e.g.
export default class MyModule {
account() {
// return a promise to the caller
return new Promise((resolve, reject) => {
// invoke request the same way but this time resolve/reject the promise
request(..., function(err, res, body) {
if (err) return reject(err);
try {
return resolve(JSON.parse(body));
} catch (e) {
return reject(e);
}
});
});
}
}
...
import myModule from 'mymodule';
// handle using promises
myModule.account()
.then(gw2data => console.log('out request', gw2data))
.catch(e => console.error('request failed', e));
// handle using ES7 async/await
// NOTE - self-executing function wrapper required for async support if using at top level,
// if using inside another function you can just mark that function as async instead
(async () => {
try {
const gw2data = await myModule.account();
console.log('out request', gw2data);
} catch (e) {
console.log('request failed', e);
}
})();
Finally, should you do decide to go down the Promise route, there are a couple of libraries out there that can polyfill Promise support into existing libraries. One example I can think of is Bluebird, which from experience I know works with request. Combine that with ES6 you get a pretty neat developer experience.

The request API is asynchronous, as is pretty much all I/O in Node.js. Check out the Promise API which is IMO the simplest way to write async code. There are a few Promise adapters for request, or you could use something like superagent with promise support.