When to use Knex transacting() vs chaining off the trx object - javascript

Knex's documentation for transactions has code that looks like this:
knex.transaction(function(trx) {
var books = [
{title: 'Canterbury Tales'},
{title: 'Moby Dick'},
{title: 'Hamlet'}
];
return trx
.insert({name: 'Old Books'}, 'id')
.into('catalogues')
.then(function(ids) {
return Promise.map(books, function(book) {
book.catalogue_id = ids[0];
// Some validation could take place here.
return trx.insert(info).into('books');
});
});
})
Here on SO I've seen extensive use of a function transacting() with examples that look like this:
knex.transaction(function(trx) {
knex('foo')
.transacting(trx)
.insert({id:"bar", username:"bar"})
// etc
})
Knex describes transacting() with examples similar to above:
Used by knex.transaction, the transacting method may be chained to any query and passed the object you wish to join the query as part of the transaction for.
My question is:
What is the difference between trx.insert().into('foo') and knex('foo').transacting(trx).insert() and why would you use one instead of the other?

It is convenient to use .transacting(trx) when you want to perform multiple operations in the same transaction:
knex.transaction(function (trx) {
return Promise.all([
knex('foo').insert({ name: 'My Name' }).transacting(trx),
knex('bar').insert({ field: 'Value' }).transacting(trx)
])
// ---- or something like ----
return Promise.all(SOME_INPUT_VALUES.map(function (value) {
return knex('foo_bar').update('lul', value.lul).where('id', value.id).transacting(trx)
}))
})
Don't know really of a particular usage of the other method. It might be just a matter of style. You got two interfaces and you can pick one whatever you like most. As for me, I'm used to .transacing(trx)

Related

Mongoose search for items on array containing _id with aggregate [duplicate]

If I have this schema...
person = {
name : String,
favoriteFoods : Array
}
... where the favoriteFoods array is populated with strings. How can I find all persons that have "sushi" as their favorite food using mongoose?
I was hoping for something along the lines of:
PersonModel.find({ favoriteFoods : { $contains : "sushi" }, function(...) {...});
(I know that there is no $contains in mongodb, just explaining what I was expecting to find before knowing the solution)
As favouriteFoods is a simple array of strings, you can just query that field directly:
PersonModel.find({ favouriteFoods: "sushi" }, ...); // favouriteFoods contains "sushi"
But I'd also recommend making the string array explicit in your schema:
person = {
name : String,
favouriteFoods : [String]
}
The relevant documentation can be found here: https://docs.mongodb.com/manual/tutorial/query-arrays/
There is no $contains operator in mongodb.
You can use the answer from JohnnyHK as that works. The closest analogy to contains that mongo has is $in, using this your query would look like:
PersonModel.find({ favouriteFoods: { "$in" : ["sushi"]} }, ...);
I feel like $all would be more appropriate in this situation. If you are looking for person that is into sushi you do :
PersonModel.find({ favoriteFood : { $all : ["sushi"] }, ...})
As you might want to filter more your search, like so :
PersonModel.find({ favoriteFood : { $all : ["sushi", "bananas"] }, ...})
$in is like OR and $all like AND. Check this : https://docs.mongodb.com/manual/reference/operator/query/all/
In case that the array contains objects for example if favouriteFoods is an array of objects of the following:
{
name: 'Sushi',
type: 'Japanese'
}
you can use the following query:
PersonModel.find({"favouriteFoods.name": "Sushi"});
In case you need to find documents which contain NULL elements inside an array of sub-documents, I've found this query which works pretty well:
db.collection.find({"keyWithArray":{$elemMatch:{"$in":[null], "$exists":true}}})
This query is taken from this post: MongoDb query array with null values
It was a great find and it works much better than my own initial and wrong version (which turned out to work fine only for arrays with one element):
.find({
'MyArrayOfSubDocuments': { $not: { $size: 0 } },
'MyArrayOfSubDocuments._id': { $exists: false }
})
Incase of lookup_food_array is array.
match_stage["favoriteFoods"] = {'$elemMatch': {'$in': lookup_food_array}}
Incase of lookup_food_array is string.
match_stage["favoriteFoods"] = {'$elemMatch': lookup_food_string}
Though agree with find() is most effective in your usecase. Still there is $match of aggregation framework, to ease the query of a big number of entries and generate a low number of results that hold value to you especially for grouping and creating new files.
PersonModel.aggregate([
{
"$match": {
$and : [{ 'favouriteFoods' : { $exists: true, $in: [ 'sushi']}}, ........ ] }
},
{ $project : {"_id": 0, "name" : 1} }
]);
There are some ways to achieve this. First one is by $elemMatch operator:
const docs = await Documents.find({category: { $elemMatch: {$eq: 'yourCategory'} }});
// you may need to convert 'yourCategory' to ObjectId
Second one is by $in or $all operators:
const docs = await Documents.find({category: { $in: [yourCategory] }});
or
const docs = await Documents.find({category: { $all: [yourCategory] }});
// you can give more categories with these two approaches
//and again you may need to convert yourCategory to ObjectId
$in is like OR and $all like AND. For further details check this link : https://docs.mongodb.com/manual/reference/operator/query/all/
Third one is by aggregate() function:
const docs = await Documents.aggregate([
{ $unwind: '$category' },
{ $match: { 'category': mongoose.Types.ObjectId(yourCategory) } }
]};
with aggregate() you get only one category id in your category array.
I get this code snippets from my projects where I had to find docs with specific category/categories, so you can easily customize it according to your needs.
For Loopback3 all the examples given did not work for me, or as fast as using REST API anyway. But it helped me to figure out the exact answer I needed.
{"where":{"arrayAttribute":{ "all" :[String]}}}
In case You are searching in an Array of objects, you can use $elemMatch. For example:
PersonModel.find({ favoriteFoods : { $elemMatch: { name: "sushiOrAnytthing" }}});
With populate & $in this code will be useful.
ServiceCategory.find().populate({
path: "services",
match: { zipCodes: {$in: "10400"}},
populate: [
{
path: "offers",
},
],
});
If you'd want to use something like a "contains" operator through javascript, you can always use a Regular expression for that...
eg.
Say you want to retrieve a customer having "Bartolomew" as name
async function getBartolomew() {
const custStartWith_Bart = await Customers.find({name: /^Bart/ }); // Starts with Bart
const custEndWith_lomew = await Customers.find({name: /lomew$/ }); // Ends with lomew
const custContains_rtol = await Customers.find({name: /.*rtol.*/ }); // Contains rtol
console.log(custStartWith_Bart);
console.log(custEndWith_lomew);
console.log(custContains_rtol);
}
I know this topic is old, but for future people who could wonder the same question, another incredibly inefficient solution could be to do:
PersonModel.find({$where : 'this.favouriteFoods.indexOf("sushi") != -1'});
This avoids all optimisations by MongoDB so do not use in production code.

Firebase adding item to array [duplicate]

I'm currently trying Firestore, and I'm stuck at something very simple: "updating an array (aka a subdocument)".
My DB structure is super simple. For example:
proprietary: "John Doe",
sharedWith:
[
{who: "first#test.com", when:timestamp},
{who: "another#test.com", when:timestamp},
],
I'm trying (without success) to push new records into shareWith array of objects.
I've tried:
// With SET
firebase.firestore()
.collection('proprietary')
.doc(docID)
.set(
{ sharedWith: [{ who: "third#test.com", when: new Date() }] },
{ merge: true }
)
// With UPDATE
firebase.firestore()
.collection('proprietary')
.doc(docID)
.update({ sharedWith: [{ who: "third#test.com", when: new Date() }] })
None works. These queries overwrite my array.
The answer might be simple, but I could'nt find it...
Firestore now has two functions that allow you to update an array without re-writing the entire thing.
Link: https://firebase.google.com/docs/firestore/manage-data/add-data, specifically https://firebase.google.com/docs/firestore/manage-data/add-data#update_elements_in_an_array
Update elements in an array
If your document contains an array field, you can use arrayUnion() and
arrayRemove() to add and remove elements. arrayUnion() adds elements
to an array but only elements not already present. arrayRemove()
removes all instances of each given element.
Edit 08/13/2018: There is now support for native array operations in Cloud Firestore. See Doug's answer below.
There is currently no way to update a single array element (or add/remove a single element) in Cloud Firestore.
This code here:
firebase.firestore()
.collection('proprietary')
.doc(docID)
.set(
{ sharedWith: [{ who: "third#test.com", when: new Date() }] },
{ merge: true }
)
This says to set the document at proprietary/docID such that sharedWith = [{ who: "third#test.com", when: new Date() } but to not affect any existing document properties. It's very similar to the update() call you provided however the set() call with create the document if it does not exist while the update() call will fail.
So you have two options to achieve what you want.
Option 1 - Set the whole array
Call set() with the entire contents of the array, which will require reading the current data from the DB first. If you're concerned about concurrent updates you can do all of this in a transaction.
Option 2 - Use a subcollection
You could make sharedWith a subcollection of the main document. Then
adding a single item would look like this:
firebase.firestore()
.collection('proprietary')
.doc(docID)
.collection('sharedWith')
.add({ who: "third#test.com", when: new Date() })
Of course this comes with new limitations. You would not be able to query
documents based on who they are shared with, nor would you be able to
get the doc and all of the sharedWith data in a single operation.
Here is the latest example from the Firestore documentation:
firebase.firestore.FieldValue.ArrayUnion
var washingtonRef = db.collection("cities").doc("DC");
// Atomically add a new region to the "regions" array field.
washingtonRef.update({
regions: firebase.firestore.FieldValue.arrayUnion("greater_virginia")
});
// Atomically remove a region from the "regions" array field.
washingtonRef.update({
regions: firebase.firestore.FieldValue.arrayRemove("east_coast")
});
You can use a transaction (https://firebase.google.com/docs/firestore/manage-data/transactions) to get the array, push onto it and then update the document:
const booking = { some: "data" };
const userRef = this.db.collection("users").doc(userId);
this.db.runTransaction(transaction => {
// This code may get re-run multiple times if there are conflicts.
return transaction.get(userRef).then(doc => {
if (!doc.data().bookings) {
transaction.set({
bookings: [booking]
});
} else {
const bookings = doc.data().bookings;
bookings.push(booking);
transaction.update(userRef, { bookings: bookings });
}
});
}).then(function () {
console.log("Transaction successfully committed!");
}).catch(function (error) {
console.log("Transaction failed: ", error);
});
Sorry Late to party but Firestore solved it way back in aug 2018 so If you still looking for that here it is all issues solved with regards to arrays.
https://firebase.googleblog.com/2018/08/better-arrays-in-cloud-firestore.htmlOfficial blog post
array-contains, arrayRemove, arrayUnion for checking, removing and updating arrays. Hope it helps.
To build on Sam Stern's answer, there is also a 3rd option which made things easier for me and that is using what Google call a Map, which is essentially a dictionary.
I think a dictionary is far better for the use case you're describing. I usually use arrays for stuff that isn't really updated too much, so they are more or less static. But for stuff that gets written a lot, specifically values that need to be updated for fields that are linked to something else in the database, dictionaries prove to be much easier to maintain and work with.
So for your specific case, the DB structure would look like this:
proprietary: "John Doe"
sharedWith:{
whoEmail1: {when: timestamp},
whoEmail2: {when: timestamp}
}
This will allow you to do the following:
var whoEmail = 'first#test.com';
var sharedObject = {};
sharedObject['sharedWith.' + whoEmail + '.when'] = new Date();
sharedObject['merge'] = true;
firebase.firestore()
.collection('proprietary')
.doc(docID)
.update(sharedObject);
The reason for defining the object as a variable is that using 'sharedWith.' + whoEmail + '.when' directly in the set method will result in an error, at least when using it in a Node.js cloud function.
#Edit (add explanation :) )
say you have an array you want to update your existing firestore document field with. You can use set(yourData, {merge: true} ) passing setOptions(second param in set function) with {merge: true} is must in order to merge the changes instead of overwriting. here is what the official documentation says about it
An options object that configures the behavior of set() calls in DocumentReference, WriteBatch, and Transaction. These calls can be configured to perform granular merges instead of overwriting the target documents in their entirety by providing a SetOptions with merge: true.
you can use this
const yourNewArray = [{who: "first#test.com", when:timestamp}
{who: "another#test.com", when:timestamp}]
collectionRef.doc(docId).set(
{
proprietary: "jhon",
sharedWith: firebase.firestore.FieldValue.arrayUnion(...yourNewArray),
},
{ merge: true },
);
hope this helps :)
addToCart(docId: string, prodId: string): Promise<void> {
return this.baseAngularFirestore.collection('carts').doc(docId).update({
products:
firestore.FieldValue.arrayUnion({
productId: prodId,
qty: 1
}),
});
}
i know this is really old, but to help people newbies with the issue
firebase V9 provides a solution using the arrayUnion and arrayRemove
await updateDoc(documentRef, {
proprietary: arrayUnion( { sharedWith: [{ who: "third#test.com", when: new Date() }] }
});
check this out for more explanation
Other than the answers mentioned above. This will do it.
Using Angular 5 and AngularFire2. or use firebase.firestore() instead of this.afs
// say you have have the following object and
// database structure as you mentioned in your post
data = { who: "third#test.com", when: new Date() };
...othercode
addSharedWith(data) {
const postDocRef = this.afs.collection('posts').doc('docID');
postDocRef.subscribe( post => {
// Grab the existing sharedWith Array
// If post.sharedWith doesn`t exsit initiated with empty array
const foo = { 'sharedWith' : post.sharedWith || []};
// Grab the existing sharedWith Array
foo['sharedWith'].push(data);
// pass updated to fireStore
postsDocRef.update(foo);
// using .set() will overwrite everything
// .update will only update existing values,
// so we initiated sharedWith with empty array
});
}
We can use arrayUnion({}) method to achive this.
Try this:
collectionRef.doc(ID).update({
sharedWith: admin.firestore.FieldValue.arrayUnion({
who: "first#test.com",
when: new Date()
})
});
Documentation can find here: https://firebase.google.com/docs/firestore/manage-data/add-data#update_elements_in_an_array
Consider John Doe a document rather than a collection
Give it a collection of things and thingsSharedWithOthers
Then you can map and query John Doe's shared things in that parallel thingsSharedWithOthers collection.
proprietary: "John Doe"(a document)
things(collection of John's things documents)
thingsSharedWithOthers(collection of John's things being shared with others):
[thingId]:
{who: "first#test.com", when:timestamp}
{who: "another#test.com", when:timestamp}
then set thingsSharedWithOthers
firebase.firestore()
.collection('thingsSharedWithOthers')
.set(
{ [thingId]:{ who: "third#test.com", when: new Date() } },
{ merge: true }
)
If You want to Update an array in a firebase document.
You can do this.
var documentRef = db.collection("Your collection name").doc("Your doc name")
documentRef.update({
yourArrayName: firebase.firestore.FieldValue.arrayUnion("The Value you want to enter")});
Although firebase.firestore.FieldValue.arrayUnion() provides the solution for array update in firestore, at the same time it is required to use {merge:true}. If you do not use {merge:true} it will delete all other fields in the document while updating with the new value. Here is the working code for updating array without loosing data in the reference document with .set() method:
const docRef = firebase.firestore().collection("your_collection_name").doc("your_doc_id");
docRef.set({yourArrayField: firebase.firestore.FieldValue.arrayUnion("value_to_add")}, {merge:true});
If anybody is looking for Java firestore sdk solution to add items in array field:
List<String> list = java.util.Arrays.asList("A", "B");
Object[] fieldsToUpdate = list.toArray();
DocumentReference docRef = getCollection().document("docId");
docRef.update(fieldName, FieldValue.arrayUnion(fieldsToUpdate));
To delete items from array user: FieldValue.arrayRemove()
If the document contains a nested object in the form of an array, .dot notation can be used to reference and update nested fields.
Node.js example:
const users = {
name: 'Tom',
surname: 'Smith',
favorites: {
sport: 'tennis',
color: 'red',
subject: 'math'
}
};
const update = await db.collection('users').doc('Tom').update({
'favorites.sport': 'snowboard'
});
or Android sdk example:
db.collection("users").document("Tom")
.update(
'favorites.sport': 'snowboard'
);
There is a simple hack in firestore:
use path with "." as property name:
propertyname.arraysubname.${id}:
db.collection("collection")
.doc("docId")
.update({arrayOfObj: fieldValue.arrayUnion({...item})})

Sequelize wrap column with " in where condition

I have a JSONB column in DB.
I'd like to have request to DB where I can check if some value in this JSON it true or false:
SELECT *
FROM table
WHERE ("json_column"->'data'->>'data2')::boolean = true AND id = '00000000-1111-2222-3333-456789abcdef'
LIMIT 1
So, my sequelize request:
const someVariableWithColumnName = 'data2';
Model.findOne({
where: {
[`$("json_column"->'data'->>'${someVariableWithColumnName}')::boolean$`]: true,
id: someIdVariable,
},
order: [/* some order, doesn't matter */],
})
And sequelize generate bad result like:
SELECT *
FROM table
WHERE "(json_column"."->'data'->>'data2')::boolean" = true AND id = '00000000-1111-2222-3333-456789abcdef'
LIMIT 1
Split my column by . and add " to every element.
Any idea how to get rid of adding " to the column in where condition?
Edit:
Here is my query with sequelize.literal():
const someVariableWithColumnName = 'data2';
Model.findOne({
where: {
[sequelize.literal(`$("json_column"->'data'->>'${someVariableWithColumnName}')::boolean$`)]: true,
id: someIdVariable,
},
order: [/* some order, doesn't matter */],
})
You can use Sequelize.literal() to avoid spurious quotes. IMHO, wrapping the json handling in a db function might also be helpful.
I just came across a similar use case.
I believe you can use the static sequelize.where method in combination with sequelize.literal.
Here is the corresponding documentation in sequelize API reference: https://sequelize.org/master/class/lib/sequelize.js~Sequelize.html#static-method-where
And here is an example (although I will admit hard to find) in the regular documentation:
https://sequelize.org/master/manual/model-querying-basics.html#advanced-queries-with-functions--not-just-columns-
In the end for your specific sit try something like this:
const someVariableWithColumnName = 'data2';
Model.findOne({
where: {
[Op.and]: [
// We provide the virtual column sql as the first argument of sequelize.where with sequelize.literal.
// We provide the matching condition as the second argument of sequelize.where, with the usual sequelize syntax.
sequelize.where(sequelize.literal(`$("json_column"->'data'->>'${someVariableWithColumnName}')::boolean$`), { [Op.eq]: true }),
{ id: someIdVariable }
]
})

Can't get state to update array inside of an object correctly in REACT/REDUX

I am going to break this down step by step for what I want to happen so hopefully people can understand what I am wanting.
Using React/Redux, Lodash
I have many post that are sent from a back end api as an array. Each post has an _id. When I call on the action getAllPost() it gives me back that array with all the post. This is working just fine.
I then dispatch type GET_ALL_POSTS and it triggers the reducer reducer_posts to change/update the state.
reducer:
export default function(state = {}, action) {
switch(action.type) {
case GET_ALL_POSTS:
const postsState = _.mapKeys(action.payload.data, '_id');
//const newPostsState = _.map(postsState, post => {
//const newComments = _.mapKeys(post.comments, '_id');
//});
return postsState;
break;
default:
return state;
break;
}
}
As you can see I change the array into one giant object that contains many post as objects with keys that are equal to their '_id'. This works just fine and returning this part of the state also works fine.
As I mentioned each of these posts has a comments value that is an array. I would like to change the comments array into one large object that holds each comment as an object with a key that is equal to their '_id' just like I did in the post.
Now I need to do this all at once and return the newly created state with One large object that contains all the post as objects and on each of those post there should be a comments object that contains all the comments as objects. I will try to write some example code to show what I am trying to do.
Example:
BigPostsObject {
1: SinglePostObject{},
2: SinglePostObject{},
3: SinglePostObject {
_id: '3',
author: 'Mike',
comments: BigCommentObject{1: SingleCommentObject{}, 2: SingleCommentObject{}}
}
}
I hope that the example kind of clears up what I am trying to do. If it still is confusing as to what I am doing then please ask and also please do not say things like use an array instead. I know I can use an array, but that is not helpful to this post as if others want to do it this way that is not helpful information.
Write a function that processes all the comments from the comments array for each post you have in the posts array:
function processComment(post) {
post.bigCommentsObject = _.mapKeys(post.comments, '_id');
// now the comments array is no longer needed
return _.omit(post, ['comments']);
}
Now use that function to turn each comments array into a big object with all the comments WHILE it still is in the array. Then afterwards turn the array itself in a big object:
const commentsProcessed = _.map(action.payload.data, procesComment);
const postsState = _.mapKeys(commentsProcessed, '_id');
I believe nowadays JS builtin function can do this without requiring external libraries. Anyway this should be the way to go. I will really encourage you getting back to js builtin functions.
var data = [
{
_id: '3',
title: 'Going on vaccation',
comments:[
{_id: 1, comment: 'hello'},
{_id: 2, comment: 'world'}
]
},
{
_id: '2',
title: 'Going to dinner',
comments:[
{_id: 1, comment: 'hello'},
{_id: 2, comment: 'world'}
]
}
]
//you can use JS builtin reduce for this
var transformedPost= _.reduce(data, function(posts, post) {
var newPost = Object.assign({}, post)
newPost._id=post._id
//you can use js builtin map for this
newPost.comments = _.mapKeys(post.comments, '_id')
// if you are using es6, replace the last three line with this
//return Object.assign({}, posts, {[newPost._id]: newPost})
var item = {}
item[newPost._id]=newPost
return Object.assign({}, posts, item)
},{});
console.log(transformedPost)
https://jsbin.com/suzifudiya/edit?js,console

How to reformat data only for saving to MongoDB with Mongoose to avoid "not okForStorage"?

I have Schema defined in Mongoose and I just realized one attribute is being saved as object (kind of hash), but it can contain prohibited characters in it's keys. By prohibited I mean those which are not very much liked by MongoDB, causing not okForStorage errors: dots, dollar signs, etc.
As I don't want to change all my application, I want to define something on my model which reformats the object to array before passing it to MongoDB and, of course, I need also something reformatting it back when loading such data from MongoDB.
I tried getters and setters and played a while with Middleware, but could not make it working. Is there a best practise on this? What would be the best approach? I really wish I could just stick two functions somewhere on the schema and it would be pure blackbox for the rest of my app.
UPDATE: What I want to achieve (example):
toMongo = function (mapping) {
// from {'k': 'v', ...} makes [{key: 'k', value: 'v'}, ...]
return ...
}
fromMongo = function (mapping) {
// from [{key: 'k', value: 'v'}, ...] makes {'k': 'v', ...}
return ...
}
schema = mongoose.Schema({
mapping: mongoose.Schema.Types.Mixed
});
var Foo = mongoose.model('Foo', schema);
var foo = new Foo({ mapping: {'tricky.key': 'yes', 'another$key': 'no'} });
foo.mapping // results in {'tricky.key': 'yes', 'another$key': 'no'}
foo.save(function(err, doc) {
// mapping is actually saved as
// [{key: 'tricky.key', value: 'yes'}, {key: 'another$key', value: 'no'}] in mongo!
doc.mapping // results in {'tricky.key': 'yes', 'another$key': 'no'}
});
Foo.find(function (err, foos) {
foos[0].mapping // results in {'tricky.key': 'yes', 'another$key': 'no'}
});
The question is: Where should I hook my two magic functions toMongo and fromMongo so the interface works exactly as I shown in the example?
(Disclaimer: At the time of this question is asked, I am Mongoose & Node.js noob, so even silly details could be helpful to me)
I think I found the answer myself. It can be solved with Middlewares, this way:
schema.post('init', function (doc) {
doc.mapping = fromMongo(doc.mapping);
}
schema.pre('save', function (next) {
this.mapping = toMongo(this.mapping);
next();
}
This way it's pretty isolated from the rest of the app and so far I didn't have any problems with this solution. I'll try to keep updating this answer in case any problems rise up.

Categories