We Keep Coding

How do I query an index properly with Dynamoose

I'm using Dynamoose to simplify my interactions with DynamoDB in a node.js application. I'm trying to write a query using Dynamoose's Model.query function that will search a table using an index, but it seems like Dynamoose is not including all of the info required to process the query and I'm not sure what I'm doing wrong.
Here's what the schema looks like:
const UserSchema = new dynamoose.Schema({
"user_id": {
"hashKey": true,
"type": String
},
"email": {
"type": String,
"index": {
"global": true,
"name": "email-index"
}
},
"first_name": {
"type": String,
"index": {
"global": true,
"name": "first_name-index"
}
},
"last_name": {
"type": String,
"index": {
"global": true,
"name": "last_name-index"
}
}
)
module.exports = dynamoose.model(config.usersTable, UserSchema)
I'd like to be able to search for users by their email address, so I'm writing a query that looks like this:
Users.query("email").contains(query.email)
.using("email-index")
.all()
.exec()
.then( results => {
res.status(200).json(results)
}).catch( err => {
res.status(500).send("Error searching for users: " + err)
})
I have a global secondary index defined for the email field:
When I try to execute this query, I'm getting the following error:
Error searching for users: ValidationException: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request.
Using the Dynamoose debugging output, I can see that the query winds up looking like this:
aws:dynamodb:query:request - {
"FilterExpression": "contains (#a0, :v0)",
"ExpressionAttributeNames": {
"#a0": "email"
},
"ExpressionAttributeValues": {
":v0": {
"S": "mel"
}
},
"TableName": "user_qa",
"IndexName": "email-index"
}
I note that the actual query sent to DynamoDB does not contain KeyConditions or KeyConditionExpression, as the error message indicates. What am I doing wrong that prevents this query from being written correctly such that it executes the query against the global secondary index I've added for this table?

As it turns out, calls like .contains(text) are used as filters, not query parameters. DynamoDB can't figure out if the text in the index contains the text I'm searching for without looking at every single record, which is a scan, not a query. So it doesn't make sense to try to use .contains(text) in this context, even though it's possible to call it in a chain like the one I constructed. What I ultimately needed to do to make this work is turn my call into a table scan with the .contains(text) filter:
Users.scan({ email: { contains: query.email }}).all().exec().then( ... )

I am not familiar with Dynamoose too much but the following code below will do an update on a record using node.JS and DynamoDB. See the key parameter I have below; by the error message you got it seems you are missing this.
To my knowledge, you must specify a key for an UPDATE request. You can checks the AWS DynamoDB docs to confirm.
var params = {
TableName: table,
Key: {
"id": customerID,
},
UpdateExpression: "set customer_name= :s, customer_address= :p, customer_phone= :u, end_date = :u",
ExpressionAttributeValues: {
":s": customer_name,
":p": customer_address,
":u": customer_phone
},
ReturnValues: "UPDATED_NEW"
};
await docClient.update(params).promise();

Array inside object | database insertion

I have the following array of objects. Currently it have one object containing several inside of it.
let arr =
[
{
"data": {
"Score": {
"score": [
"87",
"21"
],
"Player": [
"Wiki",
"Tim"
]
},
"Designation": {
"By": [
"0",
"0",
"1",
"0",
"0"
],
"Position": [
"31/07/17",
"31/07/17",
"31/07/17",
"31/07/17",
"31/07/17"
]
},
"Address": {
"Location": "London",
"House_No": "43-B",
}
}
}
]
The above data will go in one table.
I have tried looping it and inserting but did't got any way out. The above data is not constant means will change like Position have 5 elements, It can be 6 next time, So i cannot simply insert it via its indexes.
I have tried things but no success.

Mysql can store json data, and you can parse after you fetch, even you can parse json data from mysql queries but thats little complex if data changes, so its better to store it, fetch and parse.
You can select feild type JSON and store json in it.
CREATE TABLE `book` (
`id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(200) NOT NULL,
`tags` json DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
you can insert json data in it by using php function:-
$json_data = json_encode($json_data);
insertion command will be:-
INSERT INTO `book` (`title`, `tags`)
VALUES (
'ECMAScript 2015: A SitePoint Anthology',
'$json_data'
);
For manipulating json data via mysql queries there are several function like JSON_ARRAY(), JSON_OBJECT() and so on, which you can prefer to use. Please refer to below article for details:-
https://www.sitepoint.com/use-json-data-fields-mysql-databases/

In Mongodb if you have dynamic data you can use mixed type schema like below:-
details:Schema.Types.Mixed,
Sample schema:-
// grab the things we need
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var mongoosePaginate = require('mongoose-paginate');
// create a schema
var casesSchema = new Schema({
sku: {type:String, unique:true},
details:Schema.Types.Mixed,
created_at: {type:Date, default: Date.now},
images: Schema.Types.Mixed,
ebay_hosted_images: Schema.Types.Mixed,
cloudinary_hosted_images: Schema.Types.Mixed,
dropbox_hosted_images: Schema.Types.Mixed,
is_listed:Boolean,
user_id : {type: String, ref:'User'}
});
casesSchema.plugin(mongoosePaginate);
var cases = mongoose.model('cases', casesSchema);
module.exports = cases;

In ES6 you can simply get all keys and value of an object like this:Object.keys(myObj).forEach(key => {
console.log(key); // the name of the current key.
console.log(myObj[key]); // the value of the current key.
});
If it is an array for each use to get the all values
arr.forEach(function(element) {
console.log(element);
});
arr.push(element) is used to push the element to as last index of array.

Cannot change _id field in Mongo DB from int to objectId using foreach [duplicate]

I want update an _id field of one document. I know it's not really good practice. But for some technical reason, I need to update it.
If I try to update it I get:
db.clients.update({ _id: ObjectId("123")}, { $set: { _id: ObjectId("456")}})
Performing an update on the path '_id' would modify the immutable field '_id'
And the update is rejected. How I can update it?

You cannot update it. You'll have to save the document using a new _id, and then remove the old document.
// store the document in a variable
doc = db.clients.findOne({_id: ObjectId("4cc45467c55f4d2d2a000002")})
// set a new _id on the document
doc._id = ObjectId("4c8a331bda76c559ef000004")
// insert the document, using the new _id
db.clients.insert(doc)
// remove the document with the old _id
db.clients.remove({_id: ObjectId("4cc45467c55f4d2d2a000002")})

To do it for your whole collection you can also use a loop (based on Niels example):
db.status.find().forEach(function(doc){
doc._id=doc.UserId; db.status_new.insert(doc);
});
db.status_new.renameCollection("status", true);
In this case UserId was the new ID I wanted to use

In case, you want to rename _id in same collection (for instance, if you want to prefix some _ids):
db.someCollection.find().snapshot().forEach(function(doc) {
if (doc._id.indexOf("2019:") != 0) {
print("Processing: " + doc._id);
var oldDocId = doc._id;
doc._id = "2019:" + doc._id;
db.someCollection.insert(doc);
db.someCollection.remove({_id: oldDocId});
}
});
if (doc._id.indexOf("2019:") != 0) {... needed to prevent infinite loop, since forEach picks the inserted docs, even throught .snapshot() method used.

Here I have a solution that avoid multiple requests, for loops and old document removal.
You can easily create a new idea manually using something like:_id:ObjectId()
But knowing Mongo will automatically assign an _id if missing, you can use aggregate to create a $project containing all the fields of your document, but omit the field _id. You can then save it with $out
So if your document is:
{
"_id":ObjectId("5b5ed345cfbce6787588e480"),
"title": "foo",
"description": "bar"
}
Then your query will be:
db.getCollection('myCollection').aggregate([
{$match:
{_id: ObjectId("5b5ed345cfbce6787588e480")}
}
{$project:
{
title: '$title',
description: '$description'
}
},
{$out: 'myCollection'}
])

You can also create a new document from MongoDB compass or using command and set the specific _id value that you want.

As a very small improvement to the above answers i would suggest using
let doc1 = {... doc};
then
db.dyn_user_metricFormulaDefinitions.deleteOne({_id: doc._id});
This way we don't need to create extra variable to hold old _id.

Slightly modified example of #Florent Arlandis above where we insert _id from a different field in a document:
> db.coll.insertOne({ "_id": 1, "item": { "product": { "id": 11 } }, "source": "Good Store" })
{ "acknowledged" : true, "insertedId" : 1 }
> db.coll.aggregate( [ { $set: { _id : "$item.product.id" }}, { $out: "coll" } ]) // inserting _id you want for the current collection
> db.coll.find() // check that _id is changed
{ "_id" : 11, "item" : { "product" : { "id" : 11 } }, "source" : "Good Store" }
Do not use $match filter + $out as in #Florent Arlandis's answer since $out fully remove data in collection before inserting aggregate result, so effectively you will loose all data that don't match to $match filter

Is there a way to find a document matching two different populates and get his document in findOne()?

I'm using mongoose with the combo mongoDb/nodejs. I would like to findOne() a doc with some conditions.
There is my Schema :
var prognosticSchema = new Schema({
userRef : { type : Schema.Types.ObjectId, ref : 'users'},
matchRef : { type : Schema.Types.ObjectId, ref : 'match'},
...
});
Model schema 'users' contain a String 'email' and model 'match' contain a Number 'id_match' like this:
var userSchema = new Schema({
email: String,
...
});
then
var matchSchema = new Schema({
id_match: {type: Number, min: 1, max: 51},
...
});
My goal is to findOne() one doc which contains an id_match = id_match and an email = req.headers['x-key'].
I tried this:
var prognoSchema = require('../db_schema/prognostic'); // require prognostics
require('../db_schema/match'); // require match to be able to populate
var prognoQuery = prognoSchema.find()
.populate({path: 'userRef', // populate userRef
match : {
'email' : req.headers['x-key'] // populate where email match with email in headers of request (I'm using Express as node module)
},
select : 'email pseudo'
});
prognoQuery.findOne() // search for only one doc
.populate({path: 'matchRef', // populate match
match: {
'id_match': id_match // populate match where id_match is correct
}})
.exec(function(err, data) {
... // Return of value as response ...
}
When I run this code and try to get the right document knowing that there much of other prognosticSchema with such others users and match in my dataBase, i'll get userRef at null and correct matchRef in my data document.
In my dataBase, there is others users and others id_match but I would like to get the right document in findOne() helped by this two objectId in my Schema.
Is there a way to findOne() a document matching two different populates and get his document in findOne() ?

Well you can include "both" populate expressions in the same query, but of course since you actually want to "match" on the properties contained in "referenced" collections this does mean that the actual data returned from the "parent" would need to look at "all parents" first in order to populate the data:
prognoSchema.find()
.populate([
{
"path": "userRef",
"match": { "email": req.headers['x-key'] }
},
{
"path": "matchRef",
"match": { "id_match": id_match }
}
]).exec(function(err,data) {
/*
data contains the whole collection since there was no
condition there. But populated references that did not
match are now null. So .filter() them:
*/
data = data.filter(function(doc) {
return ( doc.userRef != null && doc.matchRef != null );
});
// data now contains only those item(s) that matched
})
That is not ideal, but it's just how using "referenced" data works.
A better approach would be to search the other collections "indiviually" for there single match, and then supply the found _id values to the "parent" collection. A little help from async.parallel here to facilitate waiting on the results of the other queries before executing on the parent with the matched values. Can be done in various ways, but this looks relatively clean:
async.parallel(
{
"userRef": function(callback) {
User.findOne({ "email": req.headers['x-key'] },callback);
},
"id_match": function(callback) {
Match.findOne({ "id_match": id_match },callback);
}
},
function(err,result) {
prognoSchema.findOne({
"userRef": result.userRef._id,
"matchRef": result.id_match._id
}).populate([
{ "path": "userRef", "match": { "email": req.headers['x-key'] } },
{ "path": "matchRef", "match": { "id_match": id_match } }
]).exec(function(err,progno) {
// Matched and populated data only
})
}
)
As an alternate, in modern MongoDB releases from 3.2 and onwards you could use the $lookup aggregation operator instead:
prognoSchema.aggregate(
[
// $lookup the userRef data
{ "$lookup": {
"from": "users",
"localField": "userRef",
"foreignField": "_id",
"as": "userRef"
}},
// target is an array always so $unwind
{ "$unwind": "$userRef" },
// Then filter out anything that does not match
{ "$match": {
"userRef.email": req.headers['x-key']
}},
// $lookup the matchRef data
{ "$lookup": {
"from": "matches",
"localField": "matchRef",
"foreignField": "_id",
"as": "matchRef"
}},
// target is an array always so $unwind
{ "$unwind": "$matchRef" },
// Then filter out anything that does not match
{ "$match": {
"matchRef.id_match": id_match
}}
],
function(err,prognos) {
}
)
But again similarly ugly since the "source" is still selecting everything and you are only gradually filtering out results after each $lookup operation.
The basic premise here is "MongoDB does not 'really' perform joins", and neither is .populate() a "JOIN", but just additional queries on the related collections. Since this is "not" a "join" there is no way to filter out the "parent" until the actual related data is retrieved. Even if it's done on the "server" via $lookup rather than on the "client" via .populate()
So if you "must" query this way, it's generally better to query the other collections for results "first" and then match the "parent" based on the matching _id property values as references.
But the other case here is that you "should" consider "embedding" the data instead, where it is your intent to "query" on those properties. Only when that data resides in the "single collection" is is possible for MongoDB to query and match those conditions with a single query and a performant operation.

MongoDB aggregation: How to extract the field in the results

all!
I'm new to MongoDB aggregation, after aggregating, I finally get the result:
"result" : [
{
"_id" : "531d84734031c76f06b853f0"
},
{
"_id" : "5316739f4031c76f06b85399"
},
{
"_id" : "53171a7f4031c76f06b853e5"
},
{
"_id" : "531687024031c76f06b853db"
},
{
"_id" : "5321135cf5fcb31a051e911a"
},
{
"_id" : "5315b2564031c76f06b8538f"
}
],
"ok" : 1
The data is just what I'm looking for, but I just want to make it one step further, I hope my data will be displayed like this:
"result" : [
"531d84734031c76f06b853f0",
"5316739f4031c76f06b85399",
"53171a7f4031c76f06b853e5",
"531687024031c76f06b853db",
"5321135cf5fcb31a051e911a",
"5315b2564031c76f06b8538f"
],
"ok" : 1
Yes, I just want to get all the unique id in a plain string array, is there anything I could do? Any help would be appreciated!

All MongoDB queries produce "key/value" pairs in the result document. All MongoDB content is basically a BSON document in this form, which is just "translated" back to native code form by the driver to the language it is implemented in.
So the aggregation framework alone is never going to produce a bare array of just the values as you want. But you can always just transform the array of results, as after all it is only an array
var result = db.collection.aggregate(pipeline);
var response = result.result.map(function(x) { return x._id } );
Also note that the default behavior in the shell and a preferred option is that the aggregation result is actually returned as a cursor from MongoDB 2.6 and onwards. Since this is in list form rather than as a distinct document you would process differently:
var response = db.collection.aggregate(pipeline).map(function(x) {
return x._id;
})