On the site I am creating, users can enter different tags and separate them with commas. ExpressJS should then search through whether they exist or not. If they do not exist, then it should create an object for each of them. I have an array and am iterating through it with a for function, however, only one object is created thanks to the callback... Is there any possible way to create multiple objects at once depending on the array's length?
for (i=0;i<postTopics.length;i++) {
var postTopic = postTopics[i],
postTopicUrl = postTopic.toString().toLowerCase().replace(' ', '-');
Topic.findOne({ "title": postTopics[i] }, function (err, topic) {
if (err) throw err;
if (!topic) {
Topic.create({
title: postTopic,
url: postTopicUrl
}, function (err, topic) {
if (err) throw err;
res.redirect('/');
});
}
});
}
Try out async.parallel.
$ npm install async
// Get the async module so we can do our parallel asynchronous queries much easier.
var async = require('async');
// Create a hash to store your query functions on.
var topicQueries = {};
// Loop through your postTopics once to create a query function for each one.
postTopics.forEach(function (postTopic) {
// Use postTopic as the key for the query function so we can grab it later.
topicQueries[postTopic] = function (cb) {
// cb is the callback function passed in by async.parallel. It accepts err as the first argument and the result as the second.
Topic.findOne({ title: postTopic }, cb);
};
});
// Call async.parallel and pass in our topicQueries object.
// If any of the queries passed an error to cb then the rest of the queries will be aborted and this result function will be called with an err argument.
async.parallel(topicQueries, function (err, results) {
if (err) throw err;
// Create an array to store our Topic.create query functions. We don't need a hash because we don't need to tie the results back to anything else like we had to do with postTopics in order to check if a topic existed or not.
var createQueries = [];
// All our parallel queries have completed.
// Loop through postTopics again, using postTopic to retrieve the resulting document from the results object, which has postTopic as the key.
postTopics.forEach(function (postTopic) {
// If there is no document at results[postTopic] then none was returned from the DB.
if (results[postTopic]) return;
// I changed .replace to use a regular expression. Passing a string only replaces the first space in the string whereas my regex searches the whole string.
var postTopicUrl = postTopic.toString().toLowerCase().replace(\ \g, '-');
// Since this code is executing, we know there is no topic in the DB with the title you searched for, so create a new query to create a new topic and add it to the createQueries array.
createQueries.push(function (cb) {
Topic.create({
title: postTopic,
url: postTopicUrl
}, cb);
});
});
// Pass our createQueries array to async.parallel so it can run them all simultaneously (so to speak).
async.parallel(createQueries, function (err, results) {
// If any one of the parallel create queries passes an error to the callback, this function will be immediately invoked with that err argument.
if (err) throw err;
// If we made it this far, no errors were made during topic creation, so redirect.
res.redirect('/');
});
});
First we create an object called topicQueries and we attach a query function to it for each postTopic title in your postTopics array. Then we pass the completed topicQueries object to async.parallel which will run each query and gather the results in a results object.
The results object ends up being a simple object hash with each of your postTopic titles as the key, and the value being the result from the DB. The if (results[postTopic]) return; line returns if results has no document under that postTopic key. Meaning, the code below it only runs if there was no topic returned from the DB with that title. If there was no matching topic then we add a query function to our createQueries array.
We don't want your page to redirect after just one of those new topics finishes saving. We want to wait until all your create queries have finished, so we use async.parallel yet again, but this time we use an array instead of an object hash because we don't need to tie the results to anything. When you pass an array to async.parallel the results argument will also be an array containing the results of each query, though we don't really care about the results in this example, only that no errors were thrown. If the parallel function finishes and there is no err argument then all the topics finished creating successfully and we can finally redirect the user to the new page.
PS - If you ever run into a similar situation, except each subsequent query requires data from the query before it, then checkout async.waterfall :)
If you really want to see if things exist already and avoid getting errors on duplicates then the .create() method already accepts a list. You don't seem to care about getting the document created in response so just check for the documents that are there and send in the new ones.
So with "finding first", run the tasks in succession. async.waterfall just to pretty the indent creep:
// Just a placeholder for your input
var topics = ["A Topic","B Topic","C Topic","D Topic"];
async.waterfall(
[
function(callback) {
Topic.find(
{ "title": { "$in": topics } },
function(err,found) {
// assume ["Topic B", "Topic D"] are found
found = found.map(function(x) {
return x.title;
});
var newList = topics.filter(function(x) {
return found.indexOf(x) == -1;
});
callback(err,newList);
}
);
},
function(newList,callback) {
Topic.create(
newList.map(function(x) {
return {
"title": x,
"url": x.toString().toLowerCase().replace(' ','-')
};
}),
function(err) {
if (err) throw err;
console.log("done");
callback();
}
);
}
]
);
You could move the "url" generation to a "pre" save schema hook. But again if you really don't need the validation rules, go for "bulk API" operations provided your target MongoDB and mongoose version is new enough to support this, which really means getting a handle to the underlying driver:
// Just a placeholder for your input
var topics = ["A Topic","B Topic","C Topic","D Topic"];
async.waterfall(
[
function(callback) {
Topic.find(
{ "title": { "$in": topics } },
function(err,found) {
// assume ["Topic B", "Topic D"] are found
found = found.map(function(x) {
return x.title;
});
var newList = topics.filter(function(x) {
return found.indexOf(x) == -1;
});
callback(err,newList);
}
);
},
function(newList,callback) {
var bulk = Topic.collection.initializeOrderedBulkOp();
newList.forEach(function(x) {
bullk.insert({
"title": x,
"url": x.toString().toLowerCase().replace(' ','-')
});
});
bulk.execute(function(err,results) {
console.log("done");
callback();
});
}
]
);
That is a single write operation to the server, though of course all inserts are actually done in order and checked for errors.
Otherwise just hang the errors from duplicates and insert as an "unordered Op", check for "non duplicate" errors after if you want:
// Just a placeholder for your input
var topics = ["A Topic","B Topic","C Topic","D Topic"];
var bulk = Topic.collection.initializeUnorderedBulkOp();
topics.forEach(function(x) {
bullk.insert({
"title": x,
"url": x.toString().toLowerCase().replace(' ','-')
});
});
bulk.execute(function(err,results) {
if (err) throw err;
console.log(JSON.stringify(results,undefined,4));
});
Output in results looks something like the following indicating the "duplicate" errors, but does not "throw" the error as this is not set in this case:
{
"ok": 1,
"writeErrors": [
{
"code": 11000,
"index": 1,
"errmsg": "insertDocument :: caused by :: 11000 E11000 duplicate key error index: test.topic.$title_1 dup key: { : \"B Topic\" }",
"op": {
"title": "B Topic",
"url": "b-topic",
"_id": "53b396d70fd421057200e610"
}
},
{
"code": 11000,
"index": 3,
"errmsg": "insertDocument :: caused by :: 11000 E11000 duplicate key error index: test.topic.$title_1 dup key: { : \"D Topic\" }",
"op": {
"title": "D Topic",
"url": "d-topic",
"_id": "53b396d70fd421057200e612"
}
}
],
"writeConcernErrors": [],
"nInserted": 2,
"nUpserted": 0,
"nMatched": 0,
"nModified": 0,
"nRemoved": 0,
"upserted": []
}
Note that when using the native collection methods, you need to take care that a connection is already established. The mongoose methods will "queue" up until the connection is made, but these will not. More of a testing issue unless there is a chance this would be the first code executed.
Hopefully versions of those bulk operations will be exposed in the mongoose API soon, but the general back end functionality does depend on having MongoDB 2.6 or greater on the server. Generally it is going to be the best way to process.
Of course, in all but the last sample which does not need this, you can go absolutely "async nuts" by calling versions of "filter", "map" and "forEach" that exist under that library. Likely not to be a real issue unless you are providing really long lists for input though.
The .initializeOrderedBulkOP() and .initializeUnorderedBulkOP() methods are covered in the node native driver manual. Also see the main manual for general descriptions of Bulk operations.
Related
I have a Documents in a Collection that have a field that is an Array (foo). This is an Array of other subdocuments. I want to set the same field (bar) for each subdocument in each document to the same value. This value comes from a checkbox.
So..my client-side code is something like
'click #checkAll'(e, template) {
const target = e.target;
const checked = $(target).prop('checked');
//Call Server Method to update list of Docs
const docIds = getIds();
Meteor.call('updateAllSubDocs', docIds, checked);
}
I tried using https://docs.mongodb.com/manual/reference/operator/update/positional-all/#positional-update-all
And came up with the following for my Server helper method.
'updateAllSubDocs'(ids, checked) {
Items.update({ _id: { $in: ids } }, { $set: { "foo.$[].bar": bar } },
{ multi: true }, function (err, result) {
if (err) {
throw new Meteor.Error('error updating');
}
});
}
But that throws an error 'foo.$[].bar is not allowed by the Schema'. Any ideas?
I'm using SimpleSchema for both the parent and subdocument
Thanks!
Try passing an option to bypass Simple Schema. It might be lacking support for this (somewhat) newer Mongo feature.
bypassCollection2
Example:
Items.update({ _id: { $in: ids } }, { $set: { "foo.$[].bar": bar } },
{ multi: true, bypassCollection2: true }, function (err, result) {
if (err) {
throw new Meteor.Error('error updating');
}
});
Old answer:
Since you say you need to make a unique update for each document it sounds like bulk updating is the way to go in this case. Here's an example of how to do this in Meteor.
if (docsToUpdate.length < 1) return
const bulk = MyCollection.rawCollection().initializeUnorderedBulkOp()
for (const myDoc of docsToUpdate) {
bulk.find({ _id: myDoc._id }).updateOne({ $set: update })
}
Promise.await(bulk.execute()) // or use regular await if you want...
Note we exit the function early if there's no docs because bulk.execute() throws an exception if there's no operations to process.
If your data have different data in the $set for each entry on array, I think you need a loop in server side.
Mongo has Bulk operations, but I don't know if you can call them using Collection.rawCollection().XXXXX
I've used rawCollection() to access aggregate and it works fine to me. Maybe work with bulk operations.
What I am trying to do
I am creating a social media app with react native and firebase. I am trying to call a function, and have that function return a list of posts from off of my server.
Problem
Using the return method on a firebase query gives me a hard to use object array:
Array [
Object {
"-L2mDBZ6gqY6ANJD6rg1": Object {
//...
},
},
]
I don't like how there is an object inside of an object, and the whole thing is very hard to work with. I created a list inside my app and named it items, and when pushing all of the values to that, I got a much easier to work with object:
Array [
Object {
//...
"key": "-L2mDBZ6gqY6ANJD6rg1",
},
]
This object is also a lot nicer to use because the key is not the name of the object, but inside of it.
I would just return the array I made, but that returns as undefined.
My question
In a function, how can I return an array I created using a firebase query? (to get the objects of an array)
My Code
runQ(group){
var items = [];
//I am returning the entire firebase query...
return firebase.database().ref('posts/'+group).orderByKey().once ('value', (snap) => {
snap.forEach ( (child) => {
items.push({
//post contents
});
});
console.log(items)
//... but all I want to return is the items array. This returns undefined though.
})
}
Please let me know if I'm getting your question correctly. So, the posts table in database looks like this right now:
And you want to return these posts in this manner:
[
{
"key": "-L1ELDwqJqm17iBI4UZu",
"message": "post 1"
},
{
"key": "-L1ELOuuf9hOdydnI3HU",
"message": "post 2"
},
{
"key": "-L1ELqFi7X9lm6ssOd5d",
"message": "post 3"
},
{
"key": "-L1EMH-Co64-RAQ1-AvU",
"message": "post 4"
}
...
]
Is this correct? If so, here's what you're suppose to do:
var items = [];
firebase.database().ref('posts').orderByKey().once('value', (snapshot) => {
snapshot.forEach((child) => {
// 'key' might not be a part of the post, if you do want to
// include the key as well, then use this code instead
//
// const post = child.val();
// const key = child.key;
// items.push({ ...post, key });
//
// Otherwise, the following line is enough
items.push(child.val());
});
// Then, do something with the 'items' array here
})
.catch(() => { });
Off the topics here: I see that you're using firebase.database().... to fetch posts from the database, are you using cloud functions or you're fetching those posts in your App, using users' devices to do so? If it's the latter, you probably would rather use cloud functions and pagination to fetch posts, mainly because of 2 reasons:
There might be too many posts to fetch at one time
This causes security issues, because you're allowing every device to connect to your database (you'd have to come up with real good security rules to keep your database safe)
I'm building a Thesaurus app, and for this question, the key note is that i'm adding a list of synonyms(words that have the same meaning) for a particular word(eg - "feline", "tomcat", "puss" are synonyms of "cat")
I have a Word object, with a property - "synonyms" - which is an array.
I'm going to add an array of synonyms to the Word synonyms property.
According to the MongoDb documentation see here, the only way to append all the indexes of an array to a document's array property at once is to try the following:
db.students.update(
{ _id: 5 },
{
$push: {
quizzes: {
$each: [ { wk: 5, score: 8 }, { wk: 6, score: 7 }, { wk: 7, score: 6 } ],
}
}
}
)
Let's re-write that solution to suit my data, before we venture further.
db.words.update(
{ baseWord: 'cat' },
{
$push: {
synonyms: {
$each: [ { _id: 'someValue', synonym: 'feline' }, { _id: 'someValue', synonym: 'puss' }, { _id: 'someValue', synonym: 'tomcat' } ],
}
}
}
)
Nice and concise, but not what i'm trying to do.
What if you don't know your data beforehand and have a dynamic array which you'd like to feed in?
My current solution is to split up the array and run a forEach() loop, resulting in an array being appended to the Word object's synonyms array property like so:
//req.body.synonym = 'feline,tomcat,puss';
var individualSynonyms = req.body.synonym.split(',');
individualSynonyms.forEach(function(synonym) {
db.words.update(
{ "_id": 5 },
{ $push: //this is the Word.synonyms
{ synonyms:
{
$each:[{ //pushing each synonym as a Synonym object
uuid : uuid.v4(),
synonym:synonym,
}]
}
}
},{ upsert : true },
function(err, result) {
if (err){
res.json({ success:false, message:'Error adding base word and synonym, try again or come back later.' });
console.log("Error updating word and synonym document");
}
//using an 'else' clause here will flag a "multiple header" error due to multiple json messages being returned
//because of the forEach loop
/*
else{
res.json({ success:true, message:'Word and synonyms added!' });
console.log("Update of Word document successful, check document list");
}
*/
});
//if each insert happen, we reach here
if (!err){
res.json({ success:true, message:'Word and synonyms added!.' });
console.log("Update of Word document successful, check document list");
}
});
}
This works as intended, but you may notice and issue at the bottom, where there's a commented out ELSE clause, and a check for 'if(!err)'.
If the ELSE clause is executed, we get a "multiple headers" error because the loop causes multiple JSON results for a single request.
As well as that, 'if(!err)' will throw an error, because it doesn't have scope to the 'err' parameter in the callback from the .update() function.
- If there was a way to avoid using a forEach loop, and directly feed the array of synonyms into a single update() call, then I can make use of if(!err) inside the callback.
You might be thinking: "Just remove the 'if(!err)' clause", but it seems unclean to just send a JSON response without some sort of final error check beforehand, whether an if, else, else if etc..
I could not find this particular approach in the documentation or on this site, and to me it seems like best practice if it can be done, as it allows you to perform a final error check before sending the response.
I'm curious about whether this can actually be done.
I'm not using the console, but I included a namespace prefix before calling each object for easier reading.
There is not need to "iterate" since $each takes an "array" as the argument. Simply .map() the produced array from .split() with the additional data:
db.words.update(
{ "_id": 5 },
{ $push: {
synonyms: {
$each: req.body.synonym.split(',').map(synonym =>
({ uuid: uuid.v4, synonym })
)
}
}},
{ upsert : true },
function(err,result) {
if (!err){
res.json({ success:true, message:'Word and synonyms added!.' });
console.log("Update of Word document successful, check document list");
}
}
);
So .split() produces an "array" from the string, which you "transform" using .map() into an array of the uuid value and the "synonym" from the elements of .split(). This is then a direct "array" to be applied with $each to the $push operation.
One request.
I have the following field within my collection:
"intent" : [
"Find a job",
"Advertise a job"
],
What I'm trying to do is basically filter results on that field.
So if the end user passes in
[
"Find a job",
]
I would then return all documents that have the intent Find a job this is my query so far:
var _intent = req.body.intent;
Reason.find({ intent: { $in: _intent }}, function (err, reason) {
if (err) {
res.send(err);
}
res.json(reason);
});
Now this works when I specify a value for intent as I'm using the $in-Logical Query Operator however when I do not specify a value for intent it fails to return any documents, so what I'm asking is, is there a way I can say if intent has a value then filter on it, otherwise if it doesn't then do not apply the filter?
Why not set the query object dynamically?
var query = {};
if ( req.body.hasOwnProperty('intent') )
query.intent = { "$in": req.body.intent };
Reason.find(query, function(err, reason) {
});
Or even
if ( req.body.intent.length > 0 )
Which ever case of logic suits. It's all just JavaScript after-all.
I'm adding ObjectId to an array from another array that I receive as the body.
exports.updateBasket = function (req, res) {
Basket.findOne({ _id: req.params.id }, function (err, basket) {
for(var i=0, len=req.body.length; i < len; i++) {
basket.update({$addToSet: { "items": req.body[i] } }, { upsert: true, safe: true });
}
if (err) {
res.send(err);
}
else {
res.json({ message: 'Successfully added' });
}
});
};
I have 2 questions concerning this :
Is there any upside to do the loop in angular and have multiple PUT?
What is the way to update this same array but when removing ObjectId?
One way that I thought of was to loop ObjectId that have to be removed and look if they are in the array of the object, if yes, delete them.
Another way would be to clear the array when PUT is called and update with the new ObjectId list (which would be the ones that were there minus the one user removed).
Both doesn't feel right ...
thanks
You code looks a bit odd. You are fetching asynchronously on the req.params._id but you are queuing up req.body.length potential worth of updates, but you send 'success' before you even get a response back from the updated results.
If you wanted to filter on arrays, look at lodash, if you want to process multiple updates asynchronously and get those response use async modules.