We Keep Coding

Mongodb - Handling insertMany unique constraint violation

I'm working with node and mongodb 5. I have a unique index added to the Parcel property of my collection. As I run the program while testing, I get:
MongoBulkWriteError: E11000 duplicate key error collection: mydata.pima2 index: Parcel_1 dup key: { Parcel: "AARON" }
My code:
for (let j = 0; j < lastNameRecords.length; j++) {
const ln = lastNameRecords[j].name;
const recordsObj = { 'Parcel': ln, 'recordNum': 'TBD' };
recordsArr.push(recordsObj);
}
console.log('number of records: ', recordsArr.length);
try {
--> const response = await collection.insertMany(recordsArr, { ordered: false });
const updated = await collection.updateOne(result, { recordNum: 'ERD' });
} catch (error) {
console.log(error);
}
Th error is occurring at the line with the arrow above. Obviously as I test, I am inserting multiple records , all of which have been previously inserted into the 'pima2' collection. How do I avoid causing an error or handle it so that I can move on to the next statement

Your error is related to your constraint violation, so it's prevent the scenario of repeated Parcel data.
{ Parcel: "AARON" }
There's two options to you,
Basically you can remove this constraint if you really don't need it and the error may disappear
Validate if this value already exists in your collection, after that create a find query passing this Parcel value and check if this value already exists. So you can advice to the user that this value already exists.

How to update array inside object, inside array mongoDB

So the document contains an array of objects, each object containing it's own array. So how would I go about updating one of the elements in the array that's inside the object which is inside another array. I've read some things with $. But I don't understand completely how to use it to call a position. I know the position of the element. But I can't just say $[] because the position is defined in a variable and not a string...
I've tried doing a simple
db.collection.findOne({...}, (err, data) => {...});
and then changing the arrays in the objects in the array in there with a simple:
data.arr[x].type[y] = z; data.save().catch(err => {console.log(err)});
But it doesn't save the new values I set for for the element of the array.
Sample structure after proposed solution from #Tom Slabbaert:
Data.findOne({
userID: 'CMA'
}, (err, doc) => {
if(err) {console.log(err)}
if(doc) {
for(var i = 0; i<CMA.stockMarket.length; i++) {
if(CMA.stockMarket[i].name == data.userID) {
for(var z = 0; z<CMA.stockMarket[i].userStock.length; z++) {
if(z == company) {
var updateAmount = CMA.stockMarket[i].userStock[z]+args[1]
var updateKey = `stockMarket.${i}.userStock.${z}`
Data.updateOne({userID: 'CMA'}, {'$set': {[updateKey]: updateAmount}})
}
}
}
}
}
});
-------------------------EDIT-------------------------
So I tried changing some things around in the data base to see if that would fix the problem I was having. I modified the updated code that was provided by #Tom Slabbaert. But nothing seems to work for some reason :/ Here's what I have so far, at this point I hope it's just a syntax error somewhere. Cause this is really frustrating at this point. Note that I'm still using the for loops here to find if the info exists. And if not, push that info into the database. This might only be temporary until I find a better way / if there is a better way.
for(var i = 0; i<CMA.userStocks.length; i++) {
if(CMA.userStocks[i].name == data.userID) {
for(var z = 0; z<CMA.userStocks[i].shares.length; z++) {
//console.log(CMA.userStocks[i].shares[z].companyName)
if(CMA.userStocks[i].shares[z].companyName == args[0]) {
var updateKey = `CMA.userStocks.$[elem1].shares.$[elem2].amount`
Data.updateOne(
{userID: 'CMA'},
{
"$inc": {
[updateKey]: args[1]
}
},
{
arrayFilters: [
{
"elem1.name": data.userID,
"elem2.companyName": args[0]
}
]
}
)
purchaseComplete(); return;
}
}
CMA.userStocks[i].shares.push({companyName: args[0], amount: parseInt(args[1])})
CMA.save().catch(err => {console.log(err)});
purchaseComplete(); return;
}
}
CMA.userStocks.push({name: data.userID, shares: [{companyName: args[0], amount: parseInt(args[1])}]});
CMA.save().catch(err => {console.log(err)});
purchaseComplete(); return;
The data I'm trying to find and change is structured like the following:
And what I'm trying to change in the end is the 'amount' (which is an integer)
_id: (Not relavent in this question)
userID: 'CMA'
stockMarket: [...] (Not relavent in this question)
userStocks: [
Object: (position 0 in userStocks array)
name: 'string' (equal to data.userID in the code)
shares: [
Object: (position 0 in shares array)
companyName: 'string' (this is args[0] in the code)
amount: integer
]
]

You can just prepare the "key" ahead of time. like so:
const updateKey = `arr.${x}.type.${y}`
db.collection.updateOne(
{...},
{
"$set": {
[updateKey]: z
}
})
Mongo Playground
Using Mongo's positional operators ($ and $[]) are usually required when you don't know the position in the array and want to use a condition to update the element.
------ EDIT-----
After given your sample code you just have a minor syntax error:
var updateKey = `stockMarket.${i}.userStock.${z}`
Should just be:
var updateKey = `CMA.stockMarket.${i}.userStock.${z}`
However After seeing your code I recommend you execute the following solution which uses a single update with arrayFilters, it just cleans up the code quite a bit:
const updateKey = `CMA.stockMarket.$[elem1].userStock.${company}`;
db.collection.update(
{userID: 'CMA'},
{
"$inc": {
[updateKey]: args[1]
}
},
{
arrayFilters: [
{
"elem1.name": data.userID
}
]
})
Mongo Playground

Well I found something that worked. Apparently it didn't save the db.collection.updateMany unless I made a .then() function on the end? I have no idea why, but it's the same with an aggregate I made. (It basically does the same as a Data.findOne and save it too, but it isn't limited by the parallel save error)
Solution I found with aggregation:
<collection field> = <new data for collection field>
Data.aggregate([
{
$match: { //This is used to create a filter
['<insert field>']: <insert filter>
}
}, {
$addFields: { //This is used to update existing data, or create a new field containing the data if the field isn't found
['<collection field>']: <new data for collection field>
}
}, {
$merge: { //This is used to merge the new data / document with the rest of the collection. Thus having the same effect as a standard save
into: {
db: '<insert database name>',
coll: '<insert collection name>'
}
}
}
]).then(() => {
//After it's done, do something here. Or do nothing at all it doesn't matter as long as the .then() statement remains. I found that not having this part will break the code and make it not save / work for some reason.
}); return;
Solution I found with db.collection.updateMany
db.collection.updateMany(
{<insert field>: filter}, {$set: {'<insert field>': <new data>}}
).then(() => {
//This .then() statment in my case was needed for the updateMany function to work correctly. It wouldn't save data without it for some reason, it does not need to contain any actual info in this part. As long as it's here.
});
With this new info I could simply access and change the data that I was trying to before using the previous instructions provided by #Tom Slabbaert and my new method of actually making it save the changes made into the document.

How do to retrieve all documents in a collection and see if a document exists in firebase Firestore?

Schema:
This is how my schema looks
Current Implementation:
for (let i=0; i<data.length; i++) {
try
{
var ifPresent = db.collection("Safes-Hardware").doc(data[i]['Mac address Check']);
ifPresent.get()
.then(async (doc)=>{
if (!doc.exists)
{
// Do stuff
}
else
{
//Do stuff
}
return { message: "Success is within the palm of our hands." }
}
}
Problem:
Even though this code does the job, for each data in the array I'm doing a lookup, and this results in a socket hang-up.(sometimes)
So I'm thinking I'll get all the documents in the collection in one go, store it locally and look up if a documents exists locally instead of querying the database every time.
Question:
How do I implement this?

You can just use collection("Safes-Hardware").get().then() and you can save the data locally.
let collection = []
db.collection("Safes-Hardware").get().then(function(querySnapshot) {
collection = querySnapshot.map((doc) => ({
id: doc.id,
...doc.data()
}))
});
then you can use collection to search for what you want, maybe like this
data.forEach( doc => {
let x = collection.find(v => v.id === doc['Mac address Check'])
if(x){
//it exists
}else{
// not exists
}
})
But take care you are compromising bandwidth or number of requests with o(n^2) operation in the client side

Load array of object with their id in : react native + firebase

I come from C++, C, Python space and I'm new to react native / JS / back-end world.
I have some issues loading data from firebase. Here is what I want :
My Database :
users : uid : postids[]
posts : postids : content
I want to load the postids[] array from a user and then, load content of every postids[] in this array (according to every postids in the postids[] array).
Here is my code :
_getPostsFromDatabase() {
var docRef = firebase.firestore().collection("users").doc(firebase.auth().currentUser.uid);
return docRef.get().then(function(doc) {
if (doc.exists) {
return doc.data()["posts"];
}
}).catch(function(error) {
alert("Error getting document:", error);
});
}
_loadPosts() {
var new_posts = [];
this._getPostsFromDatabase()
.then(res => {
var i;
for (i = 0; i < res.length; i++) {
firebase.firestore().collection("posts").doc(res[i])
.onSnapshot(function(doc) {
new_posts.push(doc.data());
console.log(new_posts); --> This line print correct data
});
}
})
.catch(error => console.log(error));
console.log(new_posts); ---> This line print an empty array
}
componentDidMount() {
this._loadPosts()
}
So I want this behavior :
In componentDidMount I begine the routine --> this works
loadPosts is loading the postids[] array with _getPostsFromDatabase() function --> this works
Then, I make a for loop to push every object in an array to set the state at the end --> FAIL
At step 3, everything f... up, I made some console log to debug but there is a huge real time issue because evrything is printed randomly.
How can I get my new_posts filled array at the end of the for loop to setState. Maybe I'm wrong with this method, or if I'm not, I must have some issues with Async funtion ?
Is there an expert to help me understund better what is inside this kind of use case ?
Thanks

Basically the problem is that you are trying to perform an asynchronous code in a synchronous way.
You solution might be waiting for all promises to resolve.
_loadPosts() {
this._getPostsFromDatabase()
.then(res => {
let promises = res.map(id => {
return firebase.firestore().collection("posts").doc(id)
.get().then(doc => doc.data())
})
Promise.all(promises).then(res => {console.log(res);})
}

Your console will log before the for loop, that's the reason you are getting an empty array just include your console in the response just like this:
this._getPostsFromDatabase()
.then(res => {
var i;
for (i = 0; i < res.length; i++) {
firebase.firestore().collection("posts").doc(res[i])
.onSnapshot(function(doc) {
new_posts.push(doc.data());
console.log(new_posts); --> This line print correct data
});
}
console.log(new_posts); ---->Include here
})
Hope this helps!

How to print promised data to a webpage?

I am trying to bring an array of strings from a database to a dropdown menu on a website I have created. I have everything working properly except for the final transfer of the data from the retrieval method to the website. Right now the data is in the form of a Promise, and I cannot for the life of me figure out how to get it to print out on my webpage. right now I'm just sending it to localhost:3000, I'm not at the point where I'm putting it into the dropdown yet. How would I do this ?
I've found very very little on this issue online and thus have been mainly just trying hack fixes that haven't really worked (tacking on the resolve() method, all() method). both of those resulted in syntax errors. All Var names/SQL queries have been changed btw. My latest attempt is below:
//code that sends the names to the webpage
app.get('/formfetch', function(req, res) {
const data = async() => {
let rawDat = await dbFormFetch.getNames();
return rawDat;
}
}
const hNs = data();
hNs.then((names) => {
if (names === null) {
res.end("Error: Names list came through as null.");
} else if (names.length > 0) {
resolve(names);
for (var i = 0; i < names.length; i++) {
res.end(names[i]);
}
res.status('200');
}
})
.catch((err) => {
res.status('404').json(err)
console.log("conversion of promise failed")
})
});
//the getNames() method (in a different file)
async function getNames() {
console.log("trying to get Names");
let query = `select NAME from NAMESTAB`;
console.log("query: " + query);
const binds = {};
const result = await database.simpleExecute(query, binds);
var results = [];
console.log("for loop in formfetch.js: ");
for (var i = 0; i < result.rows.length; i++) {
results[i] = i + ": " + result.rows[i].NAME+ ' \n';
}
return results;
}
The res.send method from the app.get function prints out "Made it to the Web server:" on my localhost. I checked the console, and I didn't see anything hidden in the html or something like that.
**Note: all of the data that should be in the promise is in the code (I can print it to console at any point in the code), but when I put it on the website it won't print. **

so big surprise here, I was doing it all wrong. Lesson of the day: read up on Promises and how they work before running and gunning your way through some async code. It's not as intuitive as you would hope.
// I only had made changes to the first of the two methods.
app.get('/formfetch', function(req, res) {
async function data() {
let rawDat = await dbFormFetch.getNames();
return rawDat;
}
data().then((Names) => {
if (Names === undefined) {
res.end("Error: Names list came through as null.");
} else if (Names.length > 0) {
res.setHeader('Content-Type', 'application/json');
res.status(200).json({ "names": Names });
}
})
.catch((err) => {
res.status('404').send("name retrieval failed in server.js module")
console.log(err)
console.log("conversion of promise failed")
})
});
when you use res.end() , it sets the header status and renders it immutable after calling this method, so it was the wrong thing to use. Instead of this, I used the setHeader() method to tell the website what kind of information I'm sending it, and then filled in the content by chaining the .json() method to the status() response I sent. I've never worked with promises before and I'm fairly new to NodeJS so this was a bit of a learning curve, but hopefully this helps people who are where I was yesterday. if you're new to promises, see this article and this article before you try to use this coding tool. you'll save yourself hours of debugging and error tracing.