We Keep Coding

How to check update and insert while using bulkCreate in node.js

I am using node.js to upload an excel file into the database, in my service i am using bulkCreate to upload the data into the mysql db.Let me post the table structure
table name : customer_details
columns:
customer_org_id INT,
customer_name VARCHAR,
customer_type char,
active boolean,
customer_slot VARCHAR,
service_start_time DATE,
service_end_time DATE
I have one additional requirement that is, while i will upload the excel and try to push into the db then it must check in the database that customer_org_id &customer_name exists in the database or not.If the combination exists then the existing record will be updated and active column will be false.And a new row will be inserted with customer_org_id & customer_name and the active will be set to true.I am able to do the individual operations like create , update ,delete etc but i don't understand where to put these operations together while doing a bulkCreate. I am posting my code
const upload = async(req,res) => {
try{
if(req.file == undefined){
return res.status(400).send("Please upload an excel file");
}
let path=
__basedir + "/resources/static/assets/uploads/" + req.file.filename;
readXlsxFile(path).then((rows)=>{
rows.shift();
let custdetail = [];
row.forEach((row)=>{
let custdetails ={
customer_org_id: row[0],
customer_name :row[1],
customer_type :row[2],
active :row[3],
customer_slot: row[4],
};
custdetails.push(custdetail);
});
CustomerDetails.bulkCreate(custdetails)
.then(()=>{
res.status(200).send({
message: "Uploaded the file successfully :" + req.file.originalname,
});
})
.catch((error) =>{
res.status(500).send({
message : "Fail to import data into DB",
error : error.message,
});
});
});
}catch(error){
console.log(error);
res.status(500).send({
message : "Could not upload the file :" +req.file.originalname,
});
}
}
Can anyone let me know how i can do the operations before adding data to the Db ? I am new to node js

If I understood it correctly, it seems that bulkCreate is not best solution for your problem, because, you will need to do a validation and create/update for each line of your array.
I didn't understand all your requirements, but the code would be something close to this:
const upload = async (rows) => {
const promises = rows.map(async (sindleRow) => {
const customer = await CustomerDetails.find({ where: { customer_org_id: row[0], customer_name: row[1] }})
if (customer !== undefined) { // CUSTOMER FOUND
await CustomerDetails.update({ where: { id: customer.id }, data: { active: false }})
await CustomerDetails.create({ data: { customer_org_id: row[0], customer_name: row[1], active: false }})
}
return;
});
return await Promise.all(promises);
}
Important: This code is only an example.
Your scenario does not seem to benefit from the use of a bulkCreate, because the data needs to be individually verified.

How to Insert items into multiple Arrays in Node.js & MongoDB

This might be a weird question but I believe nothing is completely impossible.
I have a List of Users in MongoDB, each user has among other things, properties array which is currently empty.
In Excel sheet, I have a data that represents each user's properties which I want to programmatically insert in each user's properties array.
Importing excel sheet is fast and easy to populating each user's properties is what gives me the problem.
I have added userId, and PropeId, from the users and the properties they bought, so Identify them as seen below
router.put('/importdata', async (req, res)=>{
// upload queries
const imported = req.files.importdata;
const uploadpath = path.resolve(`public/excel_maneger/uploads/ ${imported.name}`);
if (imported.truncated) {
throw new Error("Uploaded File is too big, should not be morethan 20 MB");
}
await imported.mv(uploadpath);
const file = render.readFile(uploadpath);
const sheets = file.SheetNames;
const data = [];
for (let i = 0; i < sheets.length; i++) {
const sheetname = sheets[i];
const sheetData = render.utils.sheet_to_json(file.Sheets[sheetname]);
sheetData.forEach((item) => {
data.push(item);
});
}
try {
const users = await User.find({role: 'Customer'})
for(let i = 0; i < users.length; i++ ){
data.forEach((d) => {
if(users[i].id == d.id){
User.updateMany(
{},
{
$set: {
properties: {
propeId: d.propeId,
},
},
},
(err, d) => {
if (err) console.log(err);
}
);
}
});
}
} catch (err) {
console.log(err)
}
})
The Problem is that this code updates everyone on the Database (including non specified users) with the same information, Please I need help, I am trying to import 11 thousand users information from excel to database

When you are updating your User.updateMany(), You are not passing in the Id.
What it does is it when if statement is true, it updates all the user, You can use findByIdAndUpdate
Also you should be using async/await. Since that is what you are using to find the user
await User.findByIdAndUpdate( users[i]._id,{ $set: { properties: { propeId: d.propeId }}})

I have finally figured where I made the mistake, I am supposed to use $in: operator to access multiple Ids as desired.
Here's the solution:
data.forEach((d) => {
User.updateMany(
{ _id: { $in: [d.id] } },
{
$push: {
properties: d.propeId,
},
},
(err, d) => {
if (err) return false;
}
);
});
Above solved the problem amazingly

Mongoose Update Field With the Item of the Another Field That is Array

I am building an Expense Tracker Application, There is a User Collection that has fields like Name, Amount, Expenses Array, Incomes Array, and So On.
My Database Is Mongo Db with Mongoose and the Server Is Written In Express.
Here Is A Screenshot Of the Database after The Values are Filled
I am trying to Implement a Route in which The User Can Delete an Expense and After Deleting the Expense I want to update the Balance and make the Balance = Balance + Expense. I am able to Delete and Expense But Not able to Update the Balance as I do not know how to retrieve the Balance from the Deleted Expense
Here is the Delete Route :
router.delete("/", (req, res) => {
const { UserID, ExpenseID } = req.query;
const query = {
$pull: {
Expenses: { _id: ExpenseID },
},
};
User.findByIdAndUpdate(UserID, query)
});
I want to add a Mongoose Method which Will Fetch the Expense Amount from the Received Expense Id and Store it in a variable and After the Expense Gets Deleted, I want to call a method to update the balance of the User in the Promise.
Here is an example of what I intend to do
// Deletes An Expense
router.delete("/", (req, res) => {
const { UserID, ExpenseID } = req.query;
const query = {
$pull: {
Expenses: { _id: ExpenseID },
},
};
User.find({/*Some Query Which Will Return me The Provided Expense ID's Amount*/})
User.findByIdAndUpdate(UserID, query)
.then(() => {
// Once I find and Remove the Expense, Now I would Like to Update the Users Balance
// NewBalance = OldBalance + Expense ID's Amount Returned by the Find Method
// Here I would Call another User.find method to update the Users Balance
})
});
Let's Say From the Above Database Snap, I want to delete the Expenses (Starting From 0) 1st Object Element, Name: Uber Fare, I will send the Object Id which is 6091f725403c2e1b8c18dda3 to the Server and should Expect my Balance to Increase from 48495 to 49695

What you can do is:
Fetch user document with UserID
Find the expense with ExpenseID
Update the Balance with the expense amount
Remove expense from Expenses array
Save user document
router.put("/", async (req, res) => {
try {
const { UserID, ExpenseID } = req.query;
let user = await User.find({ _id: UserID });
index = user.Expenses.findIndex((expense) => expense._id === ExpenseID);
if (index != -1) {
user.Balance += user.Expenses[index].Amount;
user.Expenses.splice(index, 1);
}
await user.save();
return res.status(200).json({ success: true, user: user });
} catch (error) {
return res.status(400).json({ success: false, error: error })
}
});
NOTE: Since this is updating the document, I configured put method instead of delete on the router.

Create a static method for the schema like so
userSchema.statics.deleteExpenseUpdateBalance = async function (userID, expenseID) {
const user = await this.findOne({ _id: userID });
let userExpenses = user.Expenses;
userExpenses.forEach((expense) => {
if (expense._id === expenseID) {
let amount = expense.Amount;
let index = userExpenses.indexOf(expense);
userExpenses.splice(index, 1);
user.Balance += amount;
return;
}
});
user.save((err) => {
if (err) {
console.log(err)
}
});
};
Access this in your route handler
router.delete("/", async (req, res) => {
const { UserID, ExpenseID } = req.query;
let result = await User.deleteExpenseUpdateBalance(UserID, ExpenseID)
});
This static method will remove the expense and update the balance.
Explanation: The static method just finds the user and iterates through their Expenses and removes the required expense and the amount is just added to the balance. Finally the user.save() method is called to save all documents.
NOTE: Use this only if you are sure that the expenseID exists in the Expense array, otherwise you'll just have to add an extra check for that in the static method
Also put the static method in the file where you create your schema and model in between both of their creations i.e after you've created the userSchema and before you finalise it into a model.

Firebase functions: sync with Algolia doesn't work

I am currently trying to sync my firestore documents with algolia upon a new document creation or the update of a document. The path to the collection in firestore is videos/video. The function seems to be triggering fine, however after triggering, the firebase function does not seem to relay any of the information to algolia (no records are being created). I am not getting any errors in the log. (I also double checked the rules and made sure the node could be read by default, and yes I am on the blaze plan). Does anyone know how to sync a firestore node and algolia? Thanks for all your help!
"use strict";
const functions = require("firebase-functions");
const admin = require("firebase-admin");
const algoliasearch_1 = require("algoliasearch");
// Set up Firestore.
admin.initializeApp();
const env = functions.config();
// Set up Algolia.
// The app id and API key are coming from the cloud functions environment, as we set up in Part 1,
const algoliaClient = algoliasearch_1.default(env.algolia.appid, env.algolia.apikey);
// Since I'm using develop and production environments, I'm automatically defining
// the index name according to which environment is running. functions.config().projectId is a default property set by Cloud Functions.
const collectionindexvideo = algoliaClient.initIndex('videos');
exports.collectionvideoOnCreate = functions.firestore.document('videos/{uid}').onCreate(async(snapshot, context) => {
await savevideo(snapshot);
});
exports.collectionvideoOnUpdate = functions.firestore.document('videos/{uid}').onUpdate(async(change, context) => {
await updatevideo(change);
});
exports.collectionvideoOnDelete = functions.firestore.document('videos/{uid}').onDelete(async(snapshot, context) => {
await deletevideo(snapshot);
});
async function savevideo(snapshot) {
if (snapshot.exists) {
const document = snapshot.data();
// Essentially, you want your records to contain any information that facilitates search,
// display, filtering, or relevance. Otherwise, you can leave it out.
const record = {
objectID: snapshot.id,
uid: document.uid,
title: document.title,
thumbnailurl: document.thumbnailurl,
date: document.date,
description: document.description,
genre: document.genre,
recipe: document.recipe
};
if (record) { // Removes the possibility of snapshot.data() being undefined.
if (document.isIncomplete === false) {
// In this example, we are including all properties of the Firestore document
// in the Algolia record, but do remember to evaluate if they are all necessary.
// More on that in Part 2, Step 2 above.
await collectionindexvideo.saveObject(record); // Adds or replaces a specific object.
}
}
}
}
async function updatevideo(change) {
const docBeforeChange = change.before.data();
const docAfterChange = change.after.data();
if (docBeforeChange && docAfterChange) {
if (docAfterChange.isIncomplete && !docBeforeChange.isIncomplete) {
// If the doc was COMPLETE and is now INCOMPLETE, it was
// previously indexed in algolia and must now be removed.
await deletevideo(change.after);
} else if (docAfterChange.isIncomplete === false) {
await savevideo(change.after);
}
}
}
async function deletevideo(snapshot) {
if (snapshot.exists) {
const objectID = snapshot.id;
await collectionindexvideo.deleteObject(objectID);
}
}

Still don't know what I did wrong, however if anyone else is stuck in this situation, this repository is a great resource: https://github.com/nayfin/algolia-firestore-sync. I used it and was able to properly sync firebase and algolia. Cheers!
// Takes an the Algolia index and key of document to be deleted
const removeObject = (index, key) => {
// then it deletes the document
return index.deleteObject(key, (err) => {
if (err) throw err
console.log('Key Removed from Algolia Index', key)
})
}
// Takes an the Algolia index and data to be added or updated to
const upsertObject = (index, data) => {
// then it adds or updates it
return index.saveObject(data, (err, content) => {
if (err) throw err
console.log(`Document ${data.objectID} Updated in Algolia Index `)
})
}
exports.syncAlgoliaWithFirestore = (index, change, context) => {
const data = change.after.exists ? change.after.data() : null;
const key = context.params.id; // gets the id of the document changed
// If no data then it was a delete event
if (!data) {
// so delete the document from Algolia index
return removeObject(index, key);
}
data['objectID'] = key;
// upsert the data to the Algolia index
return upsertObject(index, data);
};

select collection and get the last item of each in other collection in MongoDB

I subscribed to the publication with the code below. So in my server side file,
Meteor.publish('people', function() {
cursor = EmailDB.find({category:{$ne:"Spammer"}},{sort:{"lastMessage.date": -1},limit:20});
let transformData = (fields) => {
// this is the person we're processing now
let eadd = fields.eadd;
// get the latest message for this person
key.eadd = eadd;
let lastMessage = MessageDB.findOne(key,
{
sort: {date: -1}
}
);
// add his last message to the returned data
fields.lastMessage = lastMessage;
return fields;
};
let handle = cursor.observeChanges({
added: (id, fields) => {
fields = transformData(fields);
this.added('emails', id, fields);
},
changed: (id, fields) => {
fields = transformData(fields);
this.changed('emails', id, fields);
},
removed: (id) => {
this.removed('emails', id);
}
});
this.ready();
this.onStop(() => {
handle.stop();
});
return cursor;
}
It get all the records of Person.
I want to get the latest message of each person.
So I used the code above but it is not working, It does not add additional fields to my query.
I need to just subscribe and fetch all records, and in each record there must be a last message.

whenever you want to join data from 2 published collections like this, you have a choice of doing it on the client or on the server. if you do it on the client, you've created a race condition for yourself. it can be solved, but i find it's cleaner and easier to do it on the server. and it makes it dead simple on the client, since you need to subscribe to only one collection.
so to do it on the server, you can take care of it in the publish. there, you can transform a returned collection (PersonDB) and add data to it (MessagesDB).
i don't know what your collections look like, or how they're named, so i'm obviously making some assumptions here.
Meteor.publish('people', function() {
let cursor = PersonDB.find({});
let transformData = (fields) => {
// this is the person we're processing now
let personId = fields.personId;
// get the latest message for this person
let lastMessage = MessagesDB.findOne({personId: personId},
{
sort: {createdAt: -1}
}
);
// add his last message to the returned data
fields.lastMessage = lastMessage;
return fields;
};
let handle = cursor.observeChanges({
added: (id, fields) => {
fields = transformData(fields);
this.added('personsdb', id, fields);
},
changed: (id, fields) => {
fields = transformData(fields);
this.changed('personsdb', id, fields);
},
removed: (id) => {
this.removed('personsdb', id);
}
});
this.ready();
this.onStop(() => {
handle.stop();
});
});
on the client, you can subscribe to PersonDB like normal, and do the find like normal, and on each record you'll now have a "lastMessage" field appended.

You need to add the PersonDB Id in MessagesDB (if it is not the case)
MessagesDB.find({PersonDB:PersonDB._id},{limit:1}).fetch()[0]