Mongoose bulk insert misses critical information - javascript

The picture above shows an example of a request getting sent to the following route:
and the following picture shows what have been inserted to the database in compass (notice how there are three entries):
As we know, Model.create() accepts an array of objects, or an object.
In this example, I am sending an array of objects, to insert them.
Model.create([]) will insert the documents one by one to the database, it doesn't skip the validation part, which is why I chose it.
and when it finds a document with a validation error, it skips it, and moves to the next one.
until it finishes, then it reports the errors it encounters.
That's what it should be, However it's not exactly working like that.
Note that I have two documents which holds validation errors:
However, mongoose is only reporting the first one, it's not reporting the second one, even though it passes by it, and it sees it.
Why this information is critical?
Because on the client side, I have to know which documents got inserted, and which did not.
In this case, (when I will know which are the ones got inserted and the ones that did not), I can show the client for example that the documents x, y, z has been inserted, while the documents f, g, h did not. So the user can correct his mistake and send the request again.
The current error report is useless, because it only tells "there was a validation error", but it doesn't tell you the "where"
The error report should include all the documents which refused to be written to the database in an array.
Update
I realized that
const data = await User.insertMany(req.body)
Has exactly the same behavior.
It doesn't only apply to Model.create().
Model.insertMany() has the same problem as well.
How to make mongoose report the full errors?
since we don't have a fold code option yet, I will include the code shown in the images, down here. and I hope this isn't going to polute the question.
mongoose.connect('mongodb://localhost:27017/temp', (err) => {
if (err) return log.error(log.label, 'an error occured while connecting to DB!')
log.success(log.label, 'Successfully connected to the database!')
})
app.use(express.json())
app.post('/users', async (req, res, next) => {
try {
const data = await User.collection.insertMany(req.body)
res.json({
message: 'success!',
data,
})
} catch (error) {
console.log(error)
res.json({
message: 'an error',
data: error,
})
}
})

I checked the source code of create() method. It indeed only saves the first error and not all the errors.
However, since the create() method will send one request for each item anyway, you can implement your own logic where you will wrap all the items with Promise.allSettled() and use the create() method for each item. That way, you will know exactly which item was successfully added, and which threw an error:
app.post('/users', async (req, res, next) => {
try {
const items = req.body;
const results = await Promise.allSettled(
items.map((item) => User.create(item))
);
constole.log(results.map((result) => result.status);
// Items that were successfully created will have the status "fulfilled",
// and items that were not successfully created will have the status "rejected".
return res.status(200).json({ message: 'success', results })
} catch (error) {
console.log(error)
return res.status(400).json({ message: 'an error', data: error })
}
})

Just want to draw your attention to insertMany issue-5337 which is similar to your question, they resolved it differently, like below:
Comment on issue: https://github.com/Automattic/mongoose/issues/5783#issuecomment-341590245
In hind sight, going to have to punt on this one until a later release because we need to return a different structure if rawResult is false. We can't just return a ValidationError like in #5698 because that would cause a promise rejection, which isn't correct with insertMany() with ordered: false because that's very inconsistent with how the driver handles it. Using rawResult: true is the way to go right now.
ordered: false should give you multiple validation errors, but if ordered is not set (true by default) we should fall back to the current behavior.
User.collection.insertMany(req.body, { rawResult: true, ordered: false })
.then(users => {
console.log(users)
})
.catch(err => {
console.log(`Error: ${err}`);
});
Console Print:
{
acknowledged: true,
insertedCount: 3,
insertedIds: {
'0': new ObjectId("63a09dcaf4f03d04b07ec1dc"),
'1': new ObjectId("63a09dcaf4f03d04b07ec1de")
'2': new ObjectId("63a0a0bdfd94d1d4433e77da")
},
mongoose: {
validationErrors: [
[
Error: WASetting validation failed: ....
at ValidationError.inspect (...)
...
errors: { itemId: [ValidatorError] },
_message: '.....'
],
[
Error: WASetting validation failed: ....
at ValidationError.inspect (...)
...
errors: { itemId: [ValidatorError] },
_message: '....'
]
]
}
}
You can see the above response, this is not giving which object failed.
Currently, I would suggest #NeNaD's solution.

Related

Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to client [duplicate]

I'm facing this weird issue in NodeJS when using with Passport.js, Express and Mongoose. Basically, I get an error saying "Cannot set headers after they are sent to the client" even though I don't send more than one header.
I've read other posts and tried them out as well, and none of them worked.
app.get - is there any difference between res.send vs return res.send
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
Cannot set headers after they are sent to the client
I've dug through github issues and I can't seem to find a solution. I get the problem that this error is triggered when I send multiple response headers, but the fact is that I am not sending multiple headers. It seems just weird.
This is my stack trace:
(node:9236) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
Server Running on port 5000
MongoDB Connected Error
[ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the
client
at validateHeader (_http_outgoing.js:503:11)
at ServerResponse.setHeader (_http_outgoing.js:510:3)
at ServerResponse.header (/Users/lourdesroashan/code/github/devlog/node_modules/express/lib/response.js:767:10)
at ServerResponse.json (/Users/lourdesroashan/code/github/devlog/node_modules/express/lib/response.js:264:10)
at Profile.findOne.then.profile (/Users/lourdesroashan/code/github/devlog/routes/api/profile.js:27:30)
at <anonymous>
This is my server code:
router.get("/userprofile", passport.authenticate('jwt', { session: false }), (req, res) => {
Profile.findOne({ user: req.user.id }).then(profile => {
if (!profile) {
return res.status(404).json({ error: "No Profile Found" });
}
else {
res.json(profile);
}
}).catch(err => {
console.log(err);
})
});
I understand what the error means, but from what I know, I don't think I am sending multiple headers, I even checked by console.log that only one of the blocks is run.
Thank you so much in advance! :)
Full Code at: https://github.com/lourdesr/devlog
EDIT:
I figured it out. It was a problem in my passport.js while trying to get the authenticated user. I forgot to use 'return' on the 'done' method, which had caused it. Just added the return statement and it worked!
That particular error occurs whenever your code attempts to send more than one response to the same request. There are a number of different coding mistakes that can lead to this:
Improperly written asynchronous code that allows multiple branches to send a response.
Not returning from the request handler to stop further code in the request handler from running after you've sent a response.
Calling next() when you've already sent a response.
Improper logic branching that allows multiple code paths to execute attempt to send a response.
The code you show in your question does not appear like it would cause that error, but I do see code in a different route here that would cause that error.
Where you have this:
if (!user) {
errors.email = "User not found";
res.status(404).json({ errors });
}
You need to change it to:
if (!user) {
errors.email = "User not found";
res.status(404).json({ errors });
// stop further execution in this callback
return;
}
You don't want the code to continue after you've done res.status(404).json({ errors }); because it will then try to send another response.
In addition, everywhere you have this:
if (err) throw err;
inside an async callback, you need to replace that with something that actually sends an error response such as:
if (err) {
console.log(err);
res.sendStatus(500);
return;
}
throwing inside an async callback just goes back into the node.js event system and isn't thrown to anywhere that you can actually catch it. Further, it doesn't send a response to the http request. In otherwords, it doesn't really do what the server is supposed to do. So, do yourself a favor and never write that code in your server. When you get an error, send an error response.
Since it looks like you may be new here, I wanted to compliment you on including a link to your full source code at https://github.com/lourdesr/devlog because it's only by looking at that that I was able to see this place where the error is occuring.
I was receiving this error because of a foolish mistake on my part. I need to be more careful when referencing my other working code. The truly embarrassing part is how long I spent trying to figure out the cause of the error. Ouf!
Bad:
return res
.send(C.Status.OK)
.json({ item });
Good:
return res
.status(C.Status.OK)
.json({ item });
Use ctrl + F hotkey and find all 'res.' keywords
then replace them with 'return res.',
change all 'res.' to 'return res.'
something like this:
res.send() change to --> return res.send()
maybe you have 'res.' in some block, like if() statement
Sorry for the Late response,
As per the mongoose documentation "Mongoose queries are not promises. They have a .then() function for co and async/await as a convenience. However, unlike promises, calling a query's .then() can execute the query multiple time"
so to use promises
mongoose.Promise = global.Promise //To use the native js promises
Then
var promise = Profile.findOne({ user: req.user.id }).exec()
promise.then(function (profile){
if (!profile) {
throw new Error("User profile not found") //reject promise with error
}
return res.status(200).json(profile) //return user profile
}).catch(function (err){
console.log(err); //User profile not found
return res.status(404).json({ err.message }) //return your error msg
})
here is an nice article about switching out callbacks with promises in Mongoose
and this answer on mongooses promise rejection handling Mongoose right promise rejection handling
There is a simple fix for the node error [ERR_HTTP_HEADERS_SET]. You need to add a return statement in front of your responses to make sure your router exits correctly on error:
router.post("/", async (req, res) => {
let user = await User.findOne({email: req.body.email});
if (!user) **return** res.status(400).send("Wrong user");
});
Because of multiple response sending in your request. if you use return key word in your else condition your code will run properly
if (!profile) {
return res.status(404).json({ error: "No Profile Found" });
}
else {
**return** res.json(profile);
}
This also happen when you tries to send the multiple response for a same request !!
So make sure you always use return keyword to send response to client inorder to stop the further processing !!
Where you have this:
if (!user) { errors.email = "User not found"; res.status(404).json({ errors }); }
You need to change it to:
if (!user) { errors.email = "User not found"; return res.status(404).json({ errors }); }
I got the same error using express and mongoose with HBS template engine. I went to Expressjs and read the docs for res.render, and it says // if a callback is specified, the rendered HTML string has to be sent explicitly. So I wasnt originally sending my html explicitly in the callback,. This is only for a contact form btw, not login info, albeit GET
//Original
let { username, email } = req.query; //My get query data easier to read
res.status(200).render('index', { username, email });
//Solution without error. Second param sending data to views, Third param callback
res.status(200).render('index', { username, email }, (err, html)=>{
res.send(html);
});
In react, if your are calling the function in useEffect hook, make sure to add a dependency to the dependency Array.
I had this error from an if statement not having an else block.
if(someCondition) {
await () => { }
}
await () => { }
I changed the above to this below and solved my issue
if(someCondition) {
await () => { }
} else {
await () => { }
}
For me, I accidentally put a res.status inside of a for loop. So my server would trigger the error the second time a res.status was returned. I needed to put the res.status outside of the for loop so it would only trigger once within the function.
First of all : make sure you didn't miss any asynchronous action without an async/await or use promises/callbacks.
Then attach any res with the return keyword : return res.status(..).json({});
And finally which was my problem: don't use return res.sendStatus if you always have some return res... inside a callback function, but you can always do a retun res.status();
in my case it was :
users.save((err,savedDoc){
if(err) return res.status(501).json({})
res.status(200).json({});
});
return res.status(500); // instead ofdoing return res.sendStatus(500)
you have to enable Promises in your programm, in my case i enabled it in my mongoose schema by using mongoose.Promise = global.Promise .
This enables using native js promises.
other alternatives to this soloution is :
var mongoose = require('mongoose');
// set Promise provider to bluebird
mongoose.Promise = require('bluebird');
and
// q
mongoose.Promise = require('q').Promise;
but you need to install these packages first.
My problem besides not returning, i was forgetting to await an asynchronous function in the handler. So handler was returning and after a bit the async function did its thing. 🤦🏻‍♀️
Before:
req.session.set('x', {...});
req.session.save();
return req.status(200).end();
When i needed to await:
req.session.set('x', {...});
await req.session.save();
return req.status(200).end();
I'm putting this here for anyone else who has the same problem as me- this happened to me because I'm using the next() function without a return preceding it. Just like a lot of the other answers state, not using return with your response will / can cause / allow other code in the function to execute. In my case, I had this:
app.get("/customerDetails", async (req, res, next) => {
// check that our custom header from the app is present
if (req.get('App-Source') !== 'A Customer Header') next();
var customerID = req.query.CustomerID
var rows = await get_customer_details(customerID);
return res.json(rows);
});
In my case, I forgot to include the header in my request, so the conditional statement failed and next() was called. Another middleware function must have then been executed. After the middleware finishes, without a return, the rest of the code in the original middleware function is then executed. So I simply added a return before my next() call:
// serve customer details payload
app.get("/customerDetails", async (req, res, next) => {
// check that our custom header from the app is present
if (req.get('App-Source') !== 'A Customer Header') return next();
var customerID = req.query.CustomerID
var rows = await get_customer_details(customerID);
return res.json(rows);
});

Duplicate object in MongoDB array using Mongoose

I have a Node.js app that is creating a 'unit' in the DB under a 'building' that is saving two 'units' inside the array included in 'building'. Both of these are identical in both data and created timestamp. Anyone have an idea why the below code would be causing this to happen? I am a bit confused at it. I don't see how it would be adding two objects to the array. The code is not being run twice, I checked that with console.log() and just looking at my API logs.
await Building.findOneAndUpdate(
{buildingID},
{"$push": {units: unitData}},
(err, doc) => {
if(err) {
logger.error(`POST unit/new save error for unit: ${unitID} - error message: `, err)
return res.json({success: false, err, message: 'Error saving new unit, please try again'})
}
logger.debug('POST unit/new save() doc: ', doc)
return res.json({success: true, message: `Successfully saved new unit with ID of: ${unitID}`, unitData})
}
)
I ended up figuring it out, when there is an await and a callback it will save the data twice. This is a documented issue on the MongoDB documentation.

How do I display my catch block message when there is no ID match?

I am getting data from an API and am displaying it on my local server.
Below is my code to get data which matches the ID from the API data:
router.get('/:id', async (req, res) => {
checkString(req.params.id)
try {
const person = await peopleData.getPersonById(req.params.id);
res.json(person);
} catch (e) {
res.status(404).json({ message: 'There is no person with that ID' });
}
If there is no match I want to display the message like in the catch block, but the code does not go there as not getting a match is not an error technically.
So I tried the below code to get this message:
router.get('/:id', async (req, res) => {
checkString(req.params.id)
try {
const person = await peopleData.getPersonById(req.params.id);
if(!person) res.json('There is no person with that ID'); // Added new line here
res.json(person);
} catch (e) {
res.status(404).json({ message: 'There is no person with that ID' });
}
This does the work but it prints the message with quotes around as a string, is there a way I can display the message in the catch block if no match is found?
You can throw an error and the catch will display it.
if(!person) throw new Error("There is no person with that ID");
....
then in the catch...
catch(e){
res.status(404).json({ message: e.message })
}
If you're sending people to a fullscreen "error stack" page, then you may not need to use res.json()! You can also use res.send()
if(!person){ res.send('<p>There is no person with that ID</p>'; return; }
// Or
if(!person){ res.send('There is no person with that ID'; return; }
You are returning Json responses, so it looks like your consumer is not a web page but another app. If so, you should return undefined or null if there is no person found, and let the web page or consumer decide what message to show. Reasons are:
It should be easier to modify web pages than code, and typically the UI or marketing people will always want to fine tune (usually many times) every message on a web page.
Your app is an API app. The place where the user not found message is to be shown can be many steps away. Or it may be inappropriate to show the message at all, for example the consuming app might want to redirect to/show a registration page instead if user is not found.
Your web site may be multi-lingual, and you don't want the back-end to be involved in this.
"User not found" in many situations is not really an error, but it all depends on your application.
The catch block in your case should be used to handle other errors, for example, your database server might be down, or the database request might have timed out, etc etc. Your current code will misleadingly show "user not found" if there is a database error!
I would also let the Express error handler take care of such real errors, instead of coding error handling for every API function you have:
router.get('/:id', async (req, res, next) => {
checkString(req.params.id);
try {
const person = await peopleData.getPersonById(req.params.id);
res.json(person); // assuming getPersonById returns null if user not found
} catch (e) {
next(e);
});
Your Express error handler, where the invocation of the above next function lands, should be something like this (asssuming router is your Express app):
router.use((err, req, res, next) => {
let statusCode = err.status || 500;
// Assuming your app need to return only json responses
res.json(err);
});

Unhandled rejection SequelizeUniqueConstraintError: Validation error

I'm getting this error:
Unhandled rejection SequelizeUniqueConstraintError: Validation error
How can I fix this?
This is my models/user.js
"use strict";
module.exports = function(sequelize, DataTypes) {
var User = sequelize.define("User", {
id: { type: DataTypes.INTEGER, autoIncrement: true, primaryKey: true},
name: DataTypes.STRING,
environment_hash: DataTypes.STRING
}, {
tableName: 'users',
underscored: false,
timestamps: false
}
);
return User;
};
And this is my routes.js:
app.post('/signup', function(request, response){
console.log(request.body.email);
console.log(request.body.password);
User
.find({ where: { name: request.body.email } })
.then(function(err, user) {
if (!user) {
console.log('No user has been found.');
User.create({ name: request.body.email }).then(function(user) {
// you can now access the newly created task via the variable task
console.log('success');
});
}
});
});
The call to User.create() is returning a Promise.reject(), but there is no .catch(err) to handle it. Without catching the error and knowing the input values it's hard to say what the validation error is - the request.body.email could be too long, etc.
Catch the Promise reject to see the error/validation details
User.create({ name: request.body.email })
.then(function(user) {
// you can now access the newly created user
console.log('success', user.toJSON());
})
.catch(function(err) {
// print the error details
console.log(err, request.body.email);
});
Update, since it's 2019 and you can use async/await
try {
const user = await User.create({ name: request.body.email });
// you can now access the newly created user
console.log('success', user.toJSON());
} catch (err) {
// print the error details
console.log(err, request.body.email);
}
Check in your database if you have an Unique Constraint created, my guess is that you put some value to unique: true and changed it, but sequelize wasn't able to delete it's constraint from your database.
I had this issue with my QA database. Sometimes a new record would save to the database, and sometimes it would fail. When performing the same process on my dev workstation it would succeed every time.
When I caught the error (per #doublesharp's good advice) and printed the full results to the console, it confirmed that a unique constraint as being violated - specifically, the primary key id column, which was set to default to an autoincremented value.
I had seeded my database with records, and even though the ids of those records were also set to autoincrement, the ids of the 200-some records were scattered between 1 and 2000, but the database's autoincrement sequence was set to start at 1. Usually the next id in sequence was unused, but occasionally it was already occupied, and the database would return this error.
I used the answer here to reset the sequence to start after the last of my seeded records, and now it works every time.
When using SQLite as a database, Sequelize (currently 5.21.5) throws SequelizeUniqueConstraintError on every constraint error, even if it has nothing to do with unique indexes. One examle would be inserting NULL into a non-nullable column. So be sure to also check for other types of errors when debugging this exception.
Building on #Ricardo Machado's answer, if you add unique:true to a model and already have values in an existing table that wouldn't be allowed under this new constraint you will get this error. To fix you can manually delete the rows or delete the table and so Sequelize builds it again.
In case you stumble upon this Validation error from Sequelize: check that the query populating (creating) the table is performed once.
My Nest.js app was not properly set up and executed the same command twice thus violating the unique constraint.

JavaScript errors not showing up in console?

fields is undefined in the following code snipped, but it is not logged to the console when the error happens. In this specific instance, why, and what is the de facto way to handle this?
"Testing" is logged to the console (Line #2), but the undefined variable fields (Line #4) is not being reported. The error is returned in an API response (Line #5) but with no relevant information such as line #, stack trace, etc.
How can I make errors like this log to the console, and why are they not?
export function post(req, res) {
console.log("Testing")
User.create( getFields(req, ["name_first", "name_last"]) )
.then(user => respondJSON (res, fields, { status: 201 }))
.catch(err => respondError (res, err))
}
Since the catch is responding with an error, I get the following API response:
{
"error": true,
"data": {
"message": "fields is not defined"
}
}
I am using Babel 6 and babel-node to run my code through NPM scripts. I am using morgan logging as well. Removing the middleware for logging does not alter the error output.
The automatic logging to console is a mechanism for unhandled exceptions. Because Promises automatically catch exceptions in the callbacks, the exceptions are no-longer unhandled, so nothing will be automatically logged.
If you want it to be logged, you could perhaps add a throw err at the end of your catch block. This will convert it into an unhandled promise rejection, which is typically handled similarly to an unhandled exception.
Because you didn't actually log the error?
export function post(req, res) {
console.log("Testing")
User.create( getFields(req, ["name_first", "name_last"]) )
.then(user => respondJSON (res, fields, { status: 201 }))
.catch(err => {
console.error(err);
respondError(res, err);
});
}
I had a similar problem caused by a 'finally' which was appended to the main async function running.
run()
.finally(()=>{process.exit(0)})
modifying it to:
run()
.catch(err => {console.log(err)})
.finally(()=>{process.exit(0)})
solved the problem

Categories