We Keep Coding

How do I query an index properly with Dynamoose

I'm using Dynamoose to simplify my interactions with DynamoDB in a node.js application. I'm trying to write a query using Dynamoose's Model.query function that will search a table using an index, but it seems like Dynamoose is not including all of the info required to process the query and I'm not sure what I'm doing wrong.
Here's what the schema looks like:
const UserSchema = new dynamoose.Schema({
"user_id": {
"hashKey": true,
"type": String
},
"email": {
"type": String,
"index": {
"global": true,
"name": "email-index"
}
},
"first_name": {
"type": String,
"index": {
"global": true,
"name": "first_name-index"
}
},
"last_name": {
"type": String,
"index": {
"global": true,
"name": "last_name-index"
}
}
)
module.exports = dynamoose.model(config.usersTable, UserSchema)
I'd like to be able to search for users by their email address, so I'm writing a query that looks like this:
Users.query("email").contains(query.email)
.using("email-index")
.all()
.exec()
.then( results => {
res.status(200).json(results)
}).catch( err => {
res.status(500).send("Error searching for users: " + err)
})
I have a global secondary index defined for the email field:
When I try to execute this query, I'm getting the following error:
Error searching for users: ValidationException: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request.
Using the Dynamoose debugging output, I can see that the query winds up looking like this:
aws:dynamodb:query:request - {
"FilterExpression": "contains (#a0, :v0)",
"ExpressionAttributeNames": {
"#a0": "email"
},
"ExpressionAttributeValues": {
":v0": {
"S": "mel"
}
},
"TableName": "user_qa",
"IndexName": "email-index"
}
I note that the actual query sent to DynamoDB does not contain KeyConditions or KeyConditionExpression, as the error message indicates. What am I doing wrong that prevents this query from being written correctly such that it executes the query against the global secondary index I've added for this table?

As it turns out, calls like .contains(text) are used as filters, not query parameters. DynamoDB can't figure out if the text in the index contains the text I'm searching for without looking at every single record, which is a scan, not a query. So it doesn't make sense to try to use .contains(text) in this context, even though it's possible to call it in a chain like the one I constructed. What I ultimately needed to do to make this work is turn my call into a table scan with the .contains(text) filter:
Users.scan({ email: { contains: query.email }}).all().exec().then( ... )

I am not familiar with Dynamoose too much but the following code below will do an update on a record using node.JS and DynamoDB. See the key parameter I have below; by the error message you got it seems you are missing this.
To my knowledge, you must specify a key for an UPDATE request. You can checks the AWS DynamoDB docs to confirm.
var params = {
TableName: table,
Key: {
"id": customerID,
},
UpdateExpression: "set customer_name= :s, customer_address= :p, customer_phone= :u, end_date = :u",
ExpressionAttributeValues: {
":s": customer_name,
":p": customer_address,
":u": customer_phone
},
ReturnValues: "UPDATED_NEW"
};
await docClient.update(params).promise();

Push "programmatic" array to an Object's array property

I'm building a Thesaurus app, and for this question, the key note is that i'm adding a list of synonyms(words that have the same meaning) for a particular word(eg - "feline", "tomcat", "puss" are synonyms of "cat")
I have a Word object, with a property - "synonyms" - which is an array.
I'm going to add an array of synonyms to the Word synonyms property.
According to the MongoDb documentation see here, the only way to append all the indexes of an array to a document's array property at once is to try the following:
db.students.update(
{ _id: 5 },
{
$push: {
quizzes: {
$each: [ { wk: 5, score: 8 }, { wk: 6, score: 7 }, { wk: 7, score: 6 } ],
}
}
}
)
Let's re-write that solution to suit my data, before we venture further.
db.words.update(
{ baseWord: 'cat' },
{
$push: {
synonyms: {
$each: [ { _id: 'someValue', synonym: 'feline' }, { _id: 'someValue', synonym: 'puss' }, { _id: 'someValue', synonym: 'tomcat' } ],
}
}
}
)
Nice and concise, but not what i'm trying to do.
What if you don't know your data beforehand and have a dynamic array which you'd like to feed in?
My current solution is to split up the array and run a forEach() loop, resulting in an array being appended to the Word object's synonyms array property like so:
//req.body.synonym = 'feline,tomcat,puss';
var individualSynonyms = req.body.synonym.split(',');
individualSynonyms.forEach(function(synonym) {
db.words.update(
{ "_id": 5 },
{ $push: //this is the Word.synonyms
{ synonyms:
{
$each:[{ //pushing each synonym as a Synonym object
uuid : uuid.v4(),
synonym:synonym,
}]
}
}
},{ upsert : true },
function(err, result) {
if (err){
res.json({ success:false, message:'Error adding base word and synonym, try again or come back later.' });
console.log("Error updating word and synonym document");
}
//using an 'else' clause here will flag a "multiple header" error due to multiple json messages being returned
//because of the forEach loop
/*
else{
res.json({ success:true, message:'Word and synonyms added!' });
console.log("Update of Word document successful, check document list");
}
*/
});
//if each insert happen, we reach here
if (!err){
res.json({ success:true, message:'Word and synonyms added!.' });
console.log("Update of Word document successful, check document list");
}
});
}
This works as intended, but you may notice and issue at the bottom, where there's a commented out ELSE clause, and a check for 'if(!err)'.
If the ELSE clause is executed, we get a "multiple headers" error because the loop causes multiple JSON results for a single request.
As well as that, 'if(!err)' will throw an error, because it doesn't have scope to the 'err' parameter in the callback from the .update() function.
- If there was a way to avoid using a forEach loop, and directly feed the array of synonyms into a single update() call, then I can make use of if(!err) inside the callback.
You might be thinking: "Just remove the 'if(!err)' clause", but it seems unclean to just send a JSON response without some sort of final error check beforehand, whether an if, else, else if etc..
I could not find this particular approach in the documentation or on this site, and to me it seems like best practice if it can be done, as it allows you to perform a final error check before sending the response.
I'm curious about whether this can actually be done.
I'm not using the console, but I included a namespace prefix before calling each object for easier reading.

There is not need to "iterate" since $each takes an "array" as the argument. Simply .map() the produced array from .split() with the additional data:
db.words.update(
{ "_id": 5 },
{ $push: {
synonyms: {
$each: req.body.synonym.split(',').map(synonym =>
({ uuid: uuid.v4, synonym })
)
}
}},
{ upsert : true },
function(err,result) {
if (!err){
res.json({ success:true, message:'Word and synonyms added!.' });
console.log("Update of Word document successful, check document list");
}
}
);
So .split() produces an "array" from the string, which you "transform" using .map() into an array of the uuid value and the "synonym" from the elements of .split(). This is then a direct "array" to be applied with $each to the $push operation.
One request.

Invalid callback() argument error with Mongoose

I have a collection like this (very simplified)...
var parentSchema = new mongoose.Schema({
firstName: String,
mobile: String
});
var familySchema = new mongoose.Schema({
groupId: { type: mongoose.Schema.Types.ObjectId, index: true },
parents: [parentSchema]
});
For a given group, I'd like to find all of the parents of families in that group who have a mobile value set (exists), and unset those mobile values.
I've been able to piece this much together by looking at other examples...
Family.update(
{ groupId: someGroupId, "parents.mobile": {"$exists":"true"} },
{ $unset : { "parents.$.mobile" : 1 } }, false, true
).then(function() {
// do other stuff
});
Running generates the error:
Trace: [Error: Invalid callback() argument.]
I've tried several variations, but this one seems the most correct to me.

The .update() signature for mongoose models is:
Model.update(<{query}>,<{update}>,[<{options}>],[callback])
So when using promises, it's just the first two with the optional "third" options. The "fourth" would be a callback function, and hence the error:
Family.update(
{ "groupId": someGroupId, "parents.mobile": {"$exists": true } },
{ "$unset": { "parents.$.mobile" : "" } },
{ "multi": true }
).then(function() {
Too many people read the "shell" signature, even though the usage of:
.update(<{query}>,<{update}>,<upsert>,<multi>)
Has been deprecated in favour of the standard "options" arrangement for some time.
Always refer to the method that actually applies to your language API.

Cannot infer query fields to set error on insert

I'm trying to achieve a "getOrCreate" behavior using "findAndModify".
I'm working in nodejs using the native driver.
I have:
var matches = db.collection("matches");
matches.findAndModify({
//query
users: {
$all: [ userA_id, userB_id ]
},
lang: lang,
category_id: category_id
},
[[ "_id", "asc"]], //order
{
$setOnInsert: {
users: [userA_id, userB_id],
category_id: category_id,
lang: lang,
status: 0
}
},
{
new:true,
upsert:true
}, function(err, doc){
//Do something with doc
});
What i was trying to do is:
"Find specific match with specified users, lang and category... if not found, insert a new one with specified data"
Mongo is throwing this error:
Error getting/creating match { [MongoError: exception: cannot infer query fields to set, path 'users' is matched twice]
name: 'MongoError',
message: 'exception: cannot infer query fields to set, path \'users\' is matched twice',
errmsg: 'exception: cannot infer query fields to set, path \'users\' is matched twice',
code: 54,
ok: 0 }
Is there a way to make it work?
It's impossible?
Thank you :)

It's not the "prettiest" way to handle this, but current restrictions on the selection operators mean you would need to use a JavaScript expression with $where.
Substituting your vars for values for ease of example:
matches.findAndModify(
{
"$where": function() {
var match = [1,2];
return this.users.filter(function(el) {
return match.indexOf(el) != -1;
}).length >= 2;
},
"lang": "en",
"category_id": 1
},
[],
{
"$setOnInsert": {
"users": [1,2],
"lang": "en",
"category_id": 1
}
},
{
"new": true,
"upsert": true
},
function(err,doc) {
// do something useful here
}
);
As you might suspect, the "culprit" here is the positional $ operator, even though your operation does not make use of it.
And the problem specifically is because of $all which is looking for the possible match at "two" positions in the array. In the event that a "positional" operator was required, the engine cannot work out ( presently ) which position to present. The position should arguably be the "first" match being consistent with other operations, but it is not currently working like that.
Replacing the logic with a JavaScript expression circumvents this as the JavaScript logic cannot return a matched position anyway. That makes the expression valid, and you can then either "create" and array with the two elements in a new document or retrieve the document that contains "both" those elements as well as the other query conditions.
P.S Little bit worried about your "sort" here. You may have added it because it is "mandatory" to the method, however if you do possibly expect this to match "more than one" document and need "sort" to work out which one to get then your logic is slightly flawed.
When doing this to "find or create" then you really need to specifiy "all" of the "unique" key constraints in your query. If you don't then you are likely to run into duplicate key errors down the track.
Sort can in fact be an empty array if you do not actually need to "pick" a result from many.
Just something to keep in mind.

Mongoose: Incrementing my documents version number doesn't work, and I'm getting a Version Error when I try to save

When I try to save my document, I'm getting a VersionError: No matching document found error, similar to this SO question.
After reading this blog post, it seems that the problem is with the versioning of my document. That I'm messing with an array and so I need to update the version.
However, calling document.save() doesn't work for me. When I log out the document before and after the call to save(), document._v is the same thing.
I also tried doing document._v = document._v++ which also didn't work.
Code
exports.update = function(req, res) {
if (req.body._id) { delete req.body._id; }
User.findById(req.params.id, function(err, user) {
if (err) return handleError(res, err);
if (!user) return res.send(404);
var updated = _.extend(user, req.body); // doesn't increment the version number. causes problems with saving. see http://aaronheckmann.blogspot.com/2012/06/mongoose-v3-part-1-versioning.html
console.log('pre increment: ', updated);
updated.increment();
// updated._v = updated._v++;
console.log('post increment: ', updated);
updated.save(function(err) {
if (err) return handleError(res, err);
return res.json(200, user);
});
});
};
Output
pre increment: { _id: 5550baae1b571aafa52f070c,
provider: 'local',
name: 'Adam',
email: 'azerner3#gmail.com',
hashedPassword: '/vahOqXwCwKQKtcV3KBQeFge/YB0xtqOj+YDyck7gzyALA/IP7u7BfqQhlVHBQT26//XfBTkaOCK2bQXg65OzA==',
salt: 'MvzXW7D4xuyGQBJNeFRoUg==',
__v: 32,
drafts: [],
starredSkims: [],
skimsCreated: [ 5550cfdab8dcacd1a7892aa4 ],
role: 'user' }
post increment: { _id: 5550baae1b571aafa52f070c,
provider: 'local',
name: 'Adam',
email: 'azerner3#gmail.com',
hashedPassword: '/vahOqXwCwKQKtcV3KBQeFge/YB0xtqOj+YDyck7gzyALA/IP7u7BfqQhlVHBQT26//XfBTkaOCK2bQXg65OzA==',
salt: 'MvzXW7D4xuyGQBJNeFRoUg==',
__v: 32,
drafts: [],
starredSkims: [],
skimsCreated: [ 5550cfdab8dcacd1a7892aa4 ],
role: 'user' }

The issue here has to do with using __v and trying to update it manually. .increment does not actually perform an increment immediately, but it does set an internal flag for the model to handle incrementing. I can't find any documentation on .increment, so I assume it is probably for use internally. The problem stems from trying to combine .extend with an object that already has __v (there are two underscores by the way, not that document.__v++ affects the model internally anyway) in addition to using .increment.
When you use _.extend it copies the __v property directly onto the object which seems to cause problems because Mongoose cannot find the old version internally. I didn't dig deep enough to find why this is specifically, but you can get around it by also adding delete req.body.__v.
Rather than finding and saving as two steps, you can also use .findByIdAndUpdate. Note that this does not use __v or increment it internally. As the other answer and linked bug indicate, if you want to increment the version during an update you have to do so manually.

Versioning was implemented to mitigate the doc.save() by design (not Model.update etc). But if you want you can try the following instead:
{$set: {dummy: [2]}, $inc: { __v: 1 }}
However this was a confirmed-bug according to the link
Please validate your mongoose version from the milestone of the above issue.
Thanks :)