How to use mongoose updateMany middleware to increase performance? - javascript

SOLVED: SOLUTION AT THE BOTTOM
I have the following Code where I am updating element by element:
//registerCustomers.js
const CustomerRegistrationCode = require("../models/CustomerRegistrationCode");
const setRegCodesToUsed = async (regCodes) => {
for (let regCode of regCodes) {
await setRegCodeToUsed(regCode._id);
}
};
const setRegCodeToUsed = async (id) => {
await CustomerRegistrationCode.findByIdAndUpdate(id, { used: true });
};
The Code works fine but is to slow and i want to update many (1000) CustomerRegistrationCodes at once.
I had a look at the updateMany middleware function but found not much info online and on the official docs. I changed my code to the following but don't know how further.
//registerCustomers.js
const setRegCodesToUsed = async (regCodes) => {
await CustomerRegistrationCode.updateMany(regCodes);
}
//CustomerRegistrationCode.js
CustomerRegistrationCodeSchema.pre('updateMany', async function (next, a) {
console.log('amount arguments: ', arguments.length); //is 2
console.log(arguments); //both parameters are functions.
next();
});
What would be the best way to update many CustomerRegistrationCodes with 1000 different id's?
SOLUTION, thanks to Murat Colyaran
const setRegCodesToUsed = async (regCodes) => {
const ids = [];
regCodes.map(code => ids.push(code._id));
await setRegCodeToUsed(ids);
};
const setRegCodeToUsed = async (ids) => {
await CustomerRegistrationCode.updateMany(
{ _id: { $in: ids } },
{ used: true }
);
};

This should work:
//registerCustomers.js
const CustomerRegistrationCode = require("../models/CustomerRegistrationCode");
const setRegCodesToUsed = async (regCodes) => {
let ids = [];
regCodes.map((code) => ids.push(code._id.toString()));
await setRegCodeToUsed(ids);
};
const setRegCodeToUsed = async (ids) => {
await CustomerRegistrationCode.updateMany(
{
id : { $in: ids }
},
{
used: true
}
);
};
Instead of sending a query for every records, we just parse the id and send a bulk request with $in

Related

Get firestore data as array

Im tryng to display firestore data but I just get one value. I have try forEach and map. Nothing is working. Heres my code:
React.useEffect(() => {
retrieveNetwork();
}, []);
const retrieveNetwork = async () => {
try {
const q = query(collection(db, "cities", uidx, "adress"));
const querySnapshot = await getDocs(q);
let result = [];
//querySnapshot.docs.map((doc) => setGas(doc.data().home));
querySnapshot.docs.map((doc) => {
result.push(doc.data().home);
setGas(result);
});
} catch (e) {
alert(e);
}
};```
The .map method returns an array (official docs here), so you could do something like this:
The map() method creates a new array populated with the results of calling a provided function on every element in the calling array.
React.useEffect(() => {
retrieveNetwork();
}, []);
const retrieveNetwork = async () => {
try {
const q = query(collection(db, "cities", uidx, "adress"));
const querySnapshot = await getDocs(q);
// an array from the docs filled only with "home" data
const results = querySnapshot.docs.map((doc) => {
return doc.data().home;
});
// only update the state once per invokation
setGas(results)
} catch (e) {
alert(e);
}
};

Including the results of an async function in my 'return' while mapping an array

Problem
I am trying to call a database function while mapping an array. I created some sample code to demonstrate my issue on a smaller scale. I am using the 'mysql2' and 'sequelize' dependencies, as well as MySQL Workbench.
Goal in Sample Code
I have two tables in my database - one called 'boxes' and one called 'items'. Each item will be in a box (it will have a property of 'box_id'). Each box has a location. I want to get an array of objects that simply shows the name of the item and the location of the box it's in (not just the ID). I want this to be run in the cleanest way possible. I am aware there's a way to relate the two databases together, and an answer on how to do that would be appreciated, but I feel I would learn more important concepts using a different method - so ideally, I would like an alternative solution.
Code
mainFunction() is the starting point
// Gets all items currently in my database and returning the dataValues
const getAllItems = async () => {
const findAllItems = await Item.findAll()
const items = findAllItems.map(item => item.dataValues)
return items
}
// Gets a box from my database, by ID, and returning its dataValues
const getOneBox = async (id) => {
const findOneBox = await Box.findOne({ where: {id}})
return findOneBox.dataValues
}
// Starting point
const mainFunction = async () => {
// Get all items in database
const items = await getAllItems()
// Go through each item, and everytime, get the box that corresponds to the item's box_id
const myArray = items.map(async item => {
const getBox = await getOneBox(item.box_id)
// Return an object with my custom formatting, and including the box's location
return {
itemID: item.id,
itemName: item.name,
itemLocation: getBox.location
}
})
// The function will only work if I manually give my function enough time to finish
console.log('myArray before delay => ', myArray)
await new Promise(response => setTimeout(response, 500))
console.log('myArray after delay => ', myArray)
}
Here is the result in my terminal:
Setup
Here is my setup if it matters. I populated my tables manually to simplify things:
items =>
boxes =>
// Populating the existing 'test' schema with relevant tables and running main function after connecting
const Sequelize = require('sequelize')
const database = new Sequelize ('test', 'root', [REDACTED], {
host: 'localhost',
dialect: 'mysql',
define: {
timestamps: false
}
})
const connect = async () => {
await database.authenticate()
.then(async () => {
await database.sync()
console.log('Connected...')
mainFunction()
}).catch(error => {
console.log('Failed to connect => ', error)
})
}
connect()
// Defining my models
const Box = database.define('box', {
name: Sequelize.STRING,
location: Sequelize.STRING
})
const Item = database.define('item', {
name: Sequelize.STRING,
box_id: Sequelize.INTEGER
})
Turns out the issue was in my approach; it's actually very easy with for loops. I'll leave my solution in case it helps anyone.
This one just adds another property to my items array
const mainFunction = async () => {
const items = await getAllItems()
for(i = 0; i < items.length; i++) {
const getBox = await getOneBox(items[i].box_id)
items[i]['location'] = getBox.location
}
console.log(items)
}
This one is for if I wanted to format it in my own way
const mainFunction = async () => {
const items = await getAllItems()
const itemsFormatted = []
for(i = 0; i < items.length; i++) {
const getBox = await getOneBox(items[i].box_id)
itemsFormatted.push({
itemID: items[i].id,
itemName: items[i].name,
itemLocation: getBox.location
})
}
console.log(itemsFormatted)
}

How to await admin.auth().getUser() method within a forEach loop?

I am trying to iterate over an array of comments and need to grab the commenter's uid for each comment. I am a beginner to JavaScript and need a little bit of help with the following use case:
I need to grab the uid for each comment and then run a .getUser() method which will return the user's email address that is associated with the user's uid. Since .getUser() returns a promise (method reference link), I need to await somewhere in this loop. How to do so? Is this even a good approach?
(Note: My end goal is to eventually attach the email addresses to a to property in a msg object where I will then send out email notifications.)
Example data for comments:
[
{
id: 1,
uid: 'RGaBbiui'
},
{
id: 2,
uid: 'ladfasdflal'
},
{
id: 3,
uid: 'RGaBbiui'
},
{
id: 4,
uid: 'RGaBbiui'
},
{
id: 5,
uid: 'ladfasdflal'
},
{
id: 6,
uid: 'ladfasdflal'
}
]
Cloud function example:
export const sendCommentNotification = functions.firestore
.document('users/{uid}/posts/{postId}/comments/{commentId}')
.onCreate(async (snapshot, context) => {
try {
const commentsQuery = await admin
.firestore()
.collection(
`users/${context.params.uid}/posts/${context.params.postId}/comments`
)
.get()
const commentsArr = []
commentsQuery.forEach((documentSnapshot) =>
commentsArr.push(documentSnapshot.data())
)
const commentsArrUids = new Set(commentsArr.map((c) => c.uid))
console.log(commentsArrUids)
const emailAddresses = []
commentsArrUids.forEach((uid) =>
emailAddresses.push(admin.auth().getUser(uid)) // how to use await here?
)
...
const msg = {
to: //TO DO..put email addresses here..
...
You cannot use await in a forEach loop. You could use await in a for-loop but they won't run simultaneously as in Promise.all().
You can just await all the promises at once using Promise.all():
Returned values will be in order of the Promises passed, regardless of completion order.
const emailAddresses = []
commentsArrUids.forEach((uid) => {
emailAddresses.push(admin.auth().getUser(uid))
})
const data = await Promise.all(emailAddresses)
Data will be an array of UserRecord.
Then you can use the .map() method to get an array of all the emails.
const emails = data.map((user) => user.email)
The code can be written like this to make it easier:
const commentsDocs = await admin.firestore().collection(`users/${context.params.uid}/posts/${context.params.postId}/comments`).get()
const userIds = commentsDocs.docs.map(comment => comment.userId)
const usersReq = userIds.map(u => admin.auth().getUser(u.uid))
const emails = (await Promise.all(usersReq))).map((user) => user.email)
Use for loop instead.
for (let i = 0; i < commentsArrUids.length; i++) {
let user = await new Promise((resolve, reject) => {
admin.auth().getUser(commentsArrUids[i]).then((user) => {
resolve(user);
});
};
emailAddresses.push(user);
}
I will replace forEach with for of and the promises is in series. Also, I rewrite some of your codes as they are redundant.
export const sendCommentNotification = functions.firestore
.document("users/{uid}/posts/{postId}/comments/{commentId}")
.onCreate(async (snapshot, context) => {
try {
const comments = await admin
.firestore()
.collection(
`users/${context.params.uid}/posts/${context.params.postId}/comments`
)
.get();
const uids = new Set();
for (const comment of comments) {
uids.add(comment.data().uid);
}
const emailAddresses = [];
for (const uid of uids) {
const res = await admin.auth().getUser(uid);
emailAddresses.push(res);
}
} catch (err) {
console.log(err);
}
});

Node JS with axios async/await: write response to local file

I'm developing a Node CLI app to use locally. It takes a CSV file as input, and based on the values in its userId column, it makes a GET request to an API using one of those values at a time as input. I've created a dummy example of this below.
Here is the axios request wrapped in an async function, which returns a Promise:
const axios = require("axios");
const utils = require("./utils");
const fs = require("fs").promises;
async function getTitleGivenId(id) {
try {
return await axios.get(`https://jsonplaceholder.typicode.com/posts/${id}`);
} catch (error) {
console.error(error);
}
}
// This works fine
getTitleGivenId(1).then(res => {
console.log(res.data.title);
});
I've come up with this in order to write a CSV, but the allData string doesn't get updated inside the map function:
async function saveTitles(inCsv, outCsv) {
try {
const arrOfObj = utils.readCsv(inCsv);
// [
// { userId: '1', color: 'green' },
// { userId: '2', color: 'blue' },
// { userId: '3', color: 'red' }
// ]
let allData = "color,title\n";
arrOfObj.map(o => {
let title;
getTitleGivenId(o["userId"]).then(res => {
title = res.data.title;
allData += `${o["color"]},${title}\n`;
});
});
await fs.writeFile(outCsv, allData);
} catch (err) {
console.error(err);
}
}
// This writes only "color,title" to "outCsv.csv"
saveTitles("./inputCsv.csv", "./outCsv.csv");
Any suggestions/alternative ways to proceed would be much appreciated.
It gets updated. You're just not waiting for it to be finished. The map() function is executed, but it will not wait for the promises inside to be finished. So one option is to make the map function async as well and just wait for all iterations to be finished:
let allData = "color,title\n";
await Promise.all( arrOfObj.map( async (o) => {
const res = await getTitleGivenId(o["userId"])
const title = res.data.title;
allData += `${o["color"]},${title}\n`;
}) );
await fs.writeFile(outCsv, allData);

How to merge asynchronous function's callback result

Here is the code to get some information about server.
cpu.usage()
.then(info => {
console.log(info);
});
cpu.free()
.then(info => {
console.log(info)
});
mem.info()
.then(info => {
console.log(info)
});
I want to get every result in a function.
get_resource () {
...
console.log(cpu_usage, cpu_free, mem_info);
};
How can I design it?
Thank you.
You can try to use async/await to do that:
var get_resource = async function () {
var cpu_usage = await cpu.usage();
var cpu_free = await cpu.free();
var mem_info = await mem.info();
console.log(cpu_usage, cpu_free, mem_info);
};
Or
Promise.all([cpu.usage(), cpu.free(), mem.info()]).then(function (info) {
console.log('cpu_usage:', info[0]);
console.log('cpu_free:', info[1]);
console.log('mem_info:', info[2]);
})
You can use Promise.all() as following:
let cpuUsage = cpu.usage()
let cpuFree = cpu.free()
let memInfo = mem.info()
Promise.all([cpuUsage, cpuFree, memInfo]).then((values) => {
console.log(values);
});
If you can use ES6, then you can use array destructuring while getting results:
Promise.all([cpuUsage, cpuFree, memInfo]).then(([cpuUsageResult, cpuFreeResult, memInfoResult]) => {
console.log(cpuUsageResult);
console.log(cpuFreeResult);
console.log(memInfoResult);
});
Callback Hell
This is the old scenario where you would have to call things inside one another
cpu.usage()
.then(cpu_usage => {
cpu.free()
.then(cpu_free => {
mem.info()
.then(mem_info => {
console.log(cpu_usage, cpu_free, mem_info);
});
});
});
Async Await
in this scenario you make a function that is asynchronous.
async function get_resource () {
const cpu_usage = await cpu.usage();
const cpu_free = await cpu.free();
const mem_info = await mem.info();
console.log(cpu_usage, cpu_free, mem_info);
};
The value assigned to each const in the async function is the same value that you get as an argument in the callback of the then.

Categories