I am trying to add an index to a certain Schema with mongoose for text searches. If I add a text index to individual fields it works fine, also with compound indexes it is okay. For example the answer provided here is great:
Full text search with weight in mongoose
However, I am trying to add an index to fields which are references to other Schemas. For example my model looks like the following:
var Book = new Schema({
"title": String,
"createdAt": Date,
"publisher": {
"type": Schema.ObjectId,
"ref": "Publisher"
},
"author": {
"type": Schema.ObjectId,
"ref": "Author"
},
"isbn": String
});
So something like the following indexing doesn't work when you perform a search query as described below:
Indexing:
Book.index({"title": "text", "publisher.name": "text", "author.firstName": "text"});
Search query:
function searchBooks(req, res) {
var query = req.query.searchQuery;
Book.find({ $text: { $search: query } })
.populate('author publisher')
.limit(25)
.exec(function(err, books) {
if (err) {
res.json(err);
} else {
res.json(books);
}
}
);
}
Does anyone have any suggestions of how to add a text index in for the "publisher" and "author" fields, I am using the "populate" mongodb method to pull in the data for these schemas.
I think, what you are looking for is the ability to join tables of data and perform a query against the sum of that data. That is something you need a relational database for, which MongoDB isn't.
So I recommend you change your approach in how you would like to preform your search, e.g. search for your keyword within both author and title instead of attempting to search the whole dataset at the same time.
Related
{
"INFO": {
"email": "test#example.com",
"password": "123"
},
"PK": "3a95eab0-57de-4e15-90ea-004082e53384",
"SK": "user"
}
Above is my dataset in dynamoDB. I am building login api with expressjs with dynamodb. I am able to scan and update data with PK & SK keys but i want to query inside my INFO set.
I am trying like this:
var params = {
TableName: "table",
FilterExpression: "contains (INFO, :sendToVal)",
ExpressionAttributeValues: {
":sendToVal": {
email: "test#example.com",
password: "123",
},
},
};
But its returning:
{ error: 'Error retrieving Event' }
{ error: 'Event not found' }
Anyone help guide me, how can i retrive the set.
The DynamoDB documentation explains that the contains() function in an expression only works for strings or sets. This isn't completely accurate - it also works for lists. But in any case, it doesn't work for maps, which is what your INFO is, so the comparison doesn't match anything.
If you intended for INFO to be a list, not a map, please make it so. Otherwise, if you really intended for it to be a map, and you wanted to test whether { email: "test#example.com", password: "123" } is in that map, then what you really need to check is whether the email and password entries in this map is equal to the desired value. So the filter condition can be something like INFO.email = :email AND INFO.password = :password. Or something like this (I'm not sure I understannd what your intention was here).
I'm using Dynamoose to simplify my interactions with DynamoDB in a node.js application. I'm trying to write a query using Dynamoose's Model.query function that will search a table using an index, but it seems like Dynamoose is not including all of the info required to process the query and I'm not sure what I'm doing wrong.
Here's what the schema looks like:
const UserSchema = new dynamoose.Schema({
"user_id": {
"hashKey": true,
"type": String
},
"email": {
"type": String,
"index": {
"global": true,
"name": "email-index"
}
},
"first_name": {
"type": String,
"index": {
"global": true,
"name": "first_name-index"
}
},
"last_name": {
"type": String,
"index": {
"global": true,
"name": "last_name-index"
}
}
)
module.exports = dynamoose.model(config.usersTable, UserSchema)
I'd like to be able to search for users by their email address, so I'm writing a query that looks like this:
Users.query("email").contains(query.email)
.using("email-index")
.all()
.exec()
.then( results => {
res.status(200).json(results)
}).catch( err => {
res.status(500).send("Error searching for users: " + err)
})
I have a global secondary index defined for the email field:
When I try to execute this query, I'm getting the following error:
Error searching for users: ValidationException: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request.
Using the Dynamoose debugging output, I can see that the query winds up looking like this:
aws:dynamodb:query:request - {
"FilterExpression": "contains (#a0, :v0)",
"ExpressionAttributeNames": {
"#a0": "email"
},
"ExpressionAttributeValues": {
":v0": {
"S": "mel"
}
},
"TableName": "user_qa",
"IndexName": "email-index"
}
I note that the actual query sent to DynamoDB does not contain KeyConditions or KeyConditionExpression, as the error message indicates. What am I doing wrong that prevents this query from being written correctly such that it executes the query against the global secondary index I've added for this table?
As it turns out, calls like .contains(text) are used as filters, not query parameters. DynamoDB can't figure out if the text in the index contains the text I'm searching for without looking at every single record, which is a scan, not a query. So it doesn't make sense to try to use .contains(text) in this context, even though it's possible to call it in a chain like the one I constructed. What I ultimately needed to do to make this work is turn my call into a table scan with the .contains(text) filter:
Users.scan({ email: { contains: query.email }}).all().exec().then( ... )
I am not familiar with Dynamoose too much but the following code below will do an update on a record using node.JS and DynamoDB. See the key parameter I have below; by the error message you got it seems you are missing this.
To my knowledge, you must specify a key for an UPDATE request. You can checks the AWS DynamoDB docs to confirm.
var params = {
TableName: table,
Key: {
"id": customerID,
},
UpdateExpression: "set customer_name= :s, customer_address= :p, customer_phone= :u, end_date = :u",
ExpressionAttributeValues: {
":s": customer_name,
":p": customer_address,
":u": customer_phone
},
ReturnValues: "UPDATED_NEW"
};
await docClient.update(params).promise();
We needed to challenge our database approach and need your help
We needed to search a word/phrase in all fields of a Mongoose schema.
Let's say the schema is like this:
var sampleSchema = new Schema({
fieldABC: String,
fieldDEF: String,
fieldGHI: String
});
We need to write a find query which will search for a word in all fields in a document of the collection:
db.sampleCollection.find({
$or: [{
fieldABC: "wordToSearch"
}, {
fieldDEF: "wordToSearch"
}, {
fieldGHI: "wordToSearch"
}]
})
It's possible for us to write the above query but it looks very inefficient - is there some better and faster approach to this?
In the year 2015, it was not supported, is there any change in this?
As suggested by #Veeram
Step 1:
Create a text index
db.sampleCollection.createIndex( { "$**": "text" } )
Step 2:
Use the text index to search the word in concern
db.sampleCollection.find( { $text: { $search: "wordToSearch" } })
I have the following array of objects. Currently it have one object containing several inside of it.
let arr =
[
{
"data": {
"Score": {
"score": [
"87",
"21"
],
"Player": [
"Wiki",
"Tim"
]
},
"Designation": {
"By": [
"0",
"0",
"1",
"0",
"0"
],
"Position": [
"31/07/17",
"31/07/17",
"31/07/17",
"31/07/17",
"31/07/17"
]
},
"Address": {
"Location": "London",
"House_No": "43-B",
}
}
}
]
The above data will go in one table.
I have tried looping it and inserting but did't got any way out. The above data is not constant means will change like Position have 5 elements, It can be 6 next time, So i cannot simply insert it via its indexes.
I have tried things but no success.
Mysql can store json data, and you can parse after you fetch, even you can parse json data from mysql queries but thats little complex if data changes, so its better to store it, fetch and parse.
You can select feild type JSON and store json in it.
CREATE TABLE `book` (
`id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(200) NOT NULL,
`tags` json DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
you can insert json data in it by using php function:-
$json_data = json_encode($json_data);
insertion command will be:-
INSERT INTO `book` (`title`, `tags`)
VALUES (
'ECMAScript 2015: A SitePoint Anthology',
'$json_data'
);
For manipulating json data via mysql queries there are several function like JSON_ARRAY(), JSON_OBJECT() and so on, which you can prefer to use. Please refer to below article for details:-
https://www.sitepoint.com/use-json-data-fields-mysql-databases/
In Mongodb if you have dynamic data you can use mixed type schema like below:-
details:Schema.Types.Mixed,
Sample schema:-
// grab the things we need
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var mongoosePaginate = require('mongoose-paginate');
// create a schema
var casesSchema = new Schema({
sku: {type:String, unique:true},
details:Schema.Types.Mixed,
created_at: {type:Date, default: Date.now},
images: Schema.Types.Mixed,
ebay_hosted_images: Schema.Types.Mixed,
cloudinary_hosted_images: Schema.Types.Mixed,
dropbox_hosted_images: Schema.Types.Mixed,
is_listed:Boolean,
user_id : {type: String, ref:'User'}
});
casesSchema.plugin(mongoosePaginate);
var cases = mongoose.model('cases', casesSchema);
module.exports = cases;
In ES6 you can simply get all keys and value of an object like this:Object.keys(myObj).forEach(key => {
console.log(key); // the name of the current key.
console.log(myObj[key]); // the value of the current key.
});
If it is an array for each use to get the all values
arr.forEach(function(element) {
console.log(element);
});
arr.push(element) is used to push the element to as last index of array.
I'm using mongoose with the combo mongoDb/nodejs. I would like to findOne() a doc with some conditions.
There is my Schema :
var prognosticSchema = new Schema({
userRef : { type : Schema.Types.ObjectId, ref : 'users'},
matchRef : { type : Schema.Types.ObjectId, ref : 'match'},
...
});
Model schema 'users' contain a String 'email' and model 'match' contain a Number 'id_match' like this:
var userSchema = new Schema({
email: String,
...
});
then
var matchSchema = new Schema({
id_match: {type: Number, min: 1, max: 51},
...
});
My goal is to findOne() one doc which contains an id_match = id_match and an email = req.headers['x-key'].
I tried this:
var prognoSchema = require('../db_schema/prognostic'); // require prognostics
require('../db_schema/match'); // require match to be able to populate
var prognoQuery = prognoSchema.find()
.populate({path: 'userRef', // populate userRef
match : {
'email' : req.headers['x-key'] // populate where email match with email in headers of request (I'm using Express as node module)
},
select : 'email pseudo'
});
prognoQuery.findOne() // search for only one doc
.populate({path: 'matchRef', // populate match
match: {
'id_match': id_match // populate match where id_match is correct
}})
.exec(function(err, data) {
... // Return of value as response ...
}
When I run this code and try to get the right document knowing that there much of other prognosticSchema with such others users and match in my dataBase, i'll get userRef at null and correct matchRef in my data document.
In my dataBase, there is others users and others id_match but I would like to get the right document in findOne() helped by this two objectId in my Schema.
Is there a way to findOne() a document matching two different populates and get his document in findOne() ?
Well you can include "both" populate expressions in the same query, but of course since you actually want to "match" on the properties contained in "referenced" collections this does mean that the actual data returned from the "parent" would need to look at "all parents" first in order to populate the data:
prognoSchema.find()
.populate([
{
"path": "userRef",
"match": { "email": req.headers['x-key'] }
},
{
"path": "matchRef",
"match": { "id_match": id_match }
}
]).exec(function(err,data) {
/*
data contains the whole collection since there was no
condition there. But populated references that did not
match are now null. So .filter() them:
*/
data = data.filter(function(doc) {
return ( doc.userRef != null && doc.matchRef != null );
});
// data now contains only those item(s) that matched
})
That is not ideal, but it's just how using "referenced" data works.
A better approach would be to search the other collections "indiviually" for there single match, and then supply the found _id values to the "parent" collection. A little help from async.parallel here to facilitate waiting on the results of the other queries before executing on the parent with the matched values. Can be done in various ways, but this looks relatively clean:
async.parallel(
{
"userRef": function(callback) {
User.findOne({ "email": req.headers['x-key'] },callback);
},
"id_match": function(callback) {
Match.findOne({ "id_match": id_match },callback);
}
},
function(err,result) {
prognoSchema.findOne({
"userRef": result.userRef._id,
"matchRef": result.id_match._id
}).populate([
{ "path": "userRef", "match": { "email": req.headers['x-key'] } },
{ "path": "matchRef", "match": { "id_match": id_match } }
]).exec(function(err,progno) {
// Matched and populated data only
})
}
)
As an alternate, in modern MongoDB releases from 3.2 and onwards you could use the $lookup aggregation operator instead:
prognoSchema.aggregate(
[
// $lookup the userRef data
{ "$lookup": {
"from": "users",
"localField": "userRef",
"foreignField": "_id",
"as": "userRef"
}},
// target is an array always so $unwind
{ "$unwind": "$userRef" },
// Then filter out anything that does not match
{ "$match": {
"userRef.email": req.headers['x-key']
}},
// $lookup the matchRef data
{ "$lookup": {
"from": "matches",
"localField": "matchRef",
"foreignField": "_id",
"as": "matchRef"
}},
// target is an array always so $unwind
{ "$unwind": "$matchRef" },
// Then filter out anything that does not match
{ "$match": {
"matchRef.id_match": id_match
}}
],
function(err,prognos) {
}
)
But again similarly ugly since the "source" is still selecting everything and you are only gradually filtering out results after each $lookup operation.
The basic premise here is "MongoDB does not 'really' perform joins", and neither is .populate() a "JOIN", but just additional queries on the related collections. Since this is "not" a "join" there is no way to filter out the "parent" until the actual related data is retrieved. Even if it's done on the "server" via $lookup rather than on the "client" via .populate()
So if you "must" query this way, it's generally better to query the other collections for results "first" and then match the "parent" based on the matching _id property values as references.
But the other case here is that you "should" consider "embedding" the data instead, where it is your intent to "query" on those properties. Only when that data resides in the "single collection" is is possible for MongoDB to query and match those conditions with a single query and a performant operation.