mongoDB - Regex search against partial field values - javascript

I've a collection of countries with country calling code in the country object. How can I find a country using calling code with a mobile number?
const countries = [
{
name: 'UAE',
callingCode: 971
},
{
name: 'USA',
callingCode: 1
},
{
name: 'UK',
callingCode: 44
}
];
const number = '971524500000'; // Input
How can I find country for the given mobile using regex in mongoose javascript;

[https://en.wikipedia.org/wiki/List_of_country_calling_codes][1]
Take a look at the link above on country calling codes, and specifically see the section "Tree List".
One solution would be to implement a schema in Mongo based on this tree in order to decode the country codes.
So, a table could be created to store Mongo documents containing a field "1x" "2x" "21x" etc (the Y axis in the Tree List table).
Each of these documents could contain an array of sub-documents from x=0 to x=9 (the x axis in the Tree List table). The sub-document can contain the country name/code you are looking for. You can use a direct index into the array in the Mongo document for an efficient lookup.
I think you'll find this to be a pretty efficient implementation and should cover all the bases.

If you can restructure your array to an object this would be the fastest
const countries =
{
971: 'UAE',
1: 'USA',
44: 'UK',
}
;
var code = 44;
console.log(countries[code]);
const countries = [
{
name: 'UAE',
callingCode: 971
},
{
name: 'USA',
callingCode: 1
},
{
name: 'UK',
callingCode: 44
}
];
var myFound =countries.filter(myFunc.bind(this,44));
function myFunc(code,element) {
if(element.callingCode == code){
return element;
}
}
console.log(myFound);

On MongoDB v 4.2 - you can use $let & $regexFind to do this :
db.collection.aggregate([
{
$match: {
$expr: {
$eq: [
{
$let: {
vars: {
value: {
$regexFind: {
input: "971524500000", // pass in as string
regex: { $toString: "$callingCode" }
}
}
},
in: "$$value.idx",
}
},
0
]
}
}
}
]);
Test : MongoDB-Playground
Explanation :
General Use-case :
In general regex searches - Will have an input which will be sub-string of actual string, Ex.:-
Actual string in DB : 'We can do it in mongo'
Input : mongo (/mongo/ - will work)
Your Use-case :
From above case as mongo exists in actual string/value of db field then you can get that document, But your case is different :
Actual string in DB : mongo
Input : 'We can do it in mongo'
This doesn't work that way, So using normal /We can do it in mongo/ isn't going help you here (Also doing few tricks with regex). So we need to make a trick to $regexFind operator. Unlike mongo documentation we need take 971524500000 into input field & regex as string value of callingCode field which is vice-versa to what's given in documentation.
So once we do that, We would get something like below :
{
"match" : "971", /** value of callingCode field */
"idx" : 0, /** Index of `971` in '971524500000' */
"captures" : []
},{
"match" : "1",
"idx" : 2,
"captures" : []
},
null /** 3rd doc no match */
As country code has to be at first of given number we need docs where "idx" : 0 - So we're using $let to get index of returned object & checking against 0 & eventually getting respective docs using $match.
Note :
There is something you need to look into, Just in case if you've docs like below :
{
"_id": ObjectId("5e8f67091aa1cc3d2158ade1"),
"name": "USA",
"callingCode": 1.0
},
{
"_id": ObjectId("5e8f67091aa1cc3d2158ade3"),
"name": "somecountry",
"callingCode": 197.0
}
& input is 1971524500000, then this query will bring both docs in result. This will be the case you need to check on. Anyhow I would suggest to try this query, rather than getting all documents for collection to the code & extract required o/p this might be better to do.

Related

How to watch only the updated field of an object in Mongo change streams?

I have setup mongo streams for a product collection below but an update in any fields of the specs object returns the whole specs object instead of the updated field. I would expect a change of the display field to only return the display field rather than of the whole specs object
product collection on mongo db
{
"productName": "Apple iPhone 5",
"specs": {
"network": "GSM / CDMA / HSPA / EVDO / LTE",
"display": "IPS LCD",
"memory": "16GB 1GB RAM"
}
}
const productPipeline = [
{
$project: {
'fullDocument.productName': 1,
'updateDescription.updatedFields.specs': 1,
},
},
];
const productChangeStream = this.productModel.watch(productPipeline, {
fullDocument: 'updateLookup',
});
productChangeStream.on('change', async (data) => {
console.log(JSON.stringify(data, null, 2));
});
I have tried to use the productPipeline below but still did not work
const productPipeline=[
{
$project: {
'fullDocument.productName': 1,
'updateDescription.updatedFields.specs.network': 1,
'updateDescription.updatedFields.specs.display': 1,
'updateDescription.updatedFields.specs.memory': 1,
},
},
];
It looks behaviour is due to the difference between an update and a replace operation. MongoDB has two forms of update; a delta-update and a full-document replacement.
Replacements look like this:
db.coll.update({query for matching document}, {full replacement document})
OR
db.collection.replaceOne({query for matching document}, {full replacement document})
Normal (delta) updates look like this:
db.coll.update({query for matching document}, {$set: {individualField: newValue}})
If your application is performing full replacements, then the complete replacement document will be recorded in the oplog. Since change streams simply reports what it sees in the oplog, this will produce a change stream event with {operationType: "replace"} and a complete copy of the new document.
If, however, you do a normal update, then only the fields that were changed will be reported; you will receive a change stream event with {operationType: "update"} and a field called updateDescription which reports the fields that were modified by the operation. Please see this page for details of the update and replace events, and the updateDescription field.
For instance, if I insert the sample document you provided above and then run the following update:
db.testing.updateOne({}, {$set: {"branding.control_panel.companyLogo.displayName": "1200px-Logo_NIKE_updated.svg"}})
... then I receive the following change event (abbreviated for clarity):
{
"_id" : ...,
"operationType" : "update",
"clusterTime" : ...,
"ns" : ...,
"documentKey" : ...,
"updateDescription" : {
"updatedFields" : {
"branding.control_panel.companyLogo.displayName" : "1200px-Logo_NIKE_updated.svg"
},
"removedFields" : [ ]
}
}
Here is the Jira issue

How to read in this MongoDb field that contains dynamic children fields, in JavaScript?

Given the following MongoDB document structure (which I cannot change), I'm not sure how to read this in JavaScript?
so loginHistory is the main field name. It contains an Object type (okay???)
it can then contain multiple children fields (this above example has only ONE child), which are arrays. These field names are dynamic, but unique.
the 'array' content is a C# DateTimeOffset , I've been told.
So in the example above, Jane is the field and the value is an Array, but really it's a DateTime.Offset.
Here's another document I've found:
4x fields
So i don't know how to read this with node/JavaScript. Oh - and the field loginHistory might not exist on some documents, also :(
So given that existing document schema/structure, I need to somehow read in each loginHistory value and then create a new document (which i'll do other stuff with later).
This is some JavaScript code I tried, but doesn't work:
users.loginHistory.forEach(loginHistory => {
// do stuff, like create a new { id = users._id, name = "Jane", createdOn = "that date/time offset" }
}
Assuming we have an input set of docs like the screen shots e.g.:
var r = [
{
loginHistory: {
"Jane": [ 1648835363929, 0 ],
"Bob": [ 1648835363929, 0 ]
}
},
{
noLoginHistory: "nope"
},
{
loginHistory: {
"Dan": [ 1648835363929, 0 ],
"Dave": [ 1648835363929, 0 ],
"Jane": [ 1648835363929, 0 ]
}
}
];
then the following pipeline will create new, "converted" docs in a collection named foo2.
db.foo.aggregate([
// OK if loginHistory does not exist. X will be set to null
// and the $unwind will not produce anything:
{$project: {X: {$objectToArray: "$$ROOT.loginHistory"} }},
{$unwind: "$X"},
// Now we have individual docs of
// X: {k: name, v: the array}
// The OP wanted to make a new doc for each one. I don't know
// what function to apply to turn X.v[0] into a MongoDB datetime
// because I don't know what that big int (637807576034080256) is
// supposed to be so I used regular ms since epoch for the example
// instead. The OP will have to get more creative with division
// and such to turn 637807576034080256 into something "toDate-able"
// in MongoDB. You *could* store the big int as is but it is always
// good to try to turn a datetime into a real datetime in mongodb.
{$project: {
_id: false,
name: '$X.k',
createdOn: {$toDate: {$arrayElemAt:['$X.v',0]}}
}}
// Now we have docs like :
// {name: "Jane", createdOn: ISODate(...) }
// By calling $out to a new collection, a new _id will be assigned
// to each:
,{$out: "foo2"}
]);

Mongoose search for items on array containing _id with aggregate [duplicate]

If I have this schema...
person = {
name : String,
favoriteFoods : Array
}
... where the favoriteFoods array is populated with strings. How can I find all persons that have "sushi" as their favorite food using mongoose?
I was hoping for something along the lines of:
PersonModel.find({ favoriteFoods : { $contains : "sushi" }, function(...) {...});
(I know that there is no $contains in mongodb, just explaining what I was expecting to find before knowing the solution)
As favouriteFoods is a simple array of strings, you can just query that field directly:
PersonModel.find({ favouriteFoods: "sushi" }, ...); // favouriteFoods contains "sushi"
But I'd also recommend making the string array explicit in your schema:
person = {
name : String,
favouriteFoods : [String]
}
The relevant documentation can be found here: https://docs.mongodb.com/manual/tutorial/query-arrays/
There is no $contains operator in mongodb.
You can use the answer from JohnnyHK as that works. The closest analogy to contains that mongo has is $in, using this your query would look like:
PersonModel.find({ favouriteFoods: { "$in" : ["sushi"]} }, ...);
I feel like $all would be more appropriate in this situation. If you are looking for person that is into sushi you do :
PersonModel.find({ favoriteFood : { $all : ["sushi"] }, ...})
As you might want to filter more your search, like so :
PersonModel.find({ favoriteFood : { $all : ["sushi", "bananas"] }, ...})
$in is like OR and $all like AND. Check this : https://docs.mongodb.com/manual/reference/operator/query/all/
In case that the array contains objects for example if favouriteFoods is an array of objects of the following:
{
name: 'Sushi',
type: 'Japanese'
}
you can use the following query:
PersonModel.find({"favouriteFoods.name": "Sushi"});
In case you need to find documents which contain NULL elements inside an array of sub-documents, I've found this query which works pretty well:
db.collection.find({"keyWithArray":{$elemMatch:{"$in":[null], "$exists":true}}})
This query is taken from this post: MongoDb query array with null values
It was a great find and it works much better than my own initial and wrong version (which turned out to work fine only for arrays with one element):
.find({
'MyArrayOfSubDocuments': { $not: { $size: 0 } },
'MyArrayOfSubDocuments._id': { $exists: false }
})
Incase of lookup_food_array is array.
match_stage["favoriteFoods"] = {'$elemMatch': {'$in': lookup_food_array}}
Incase of lookup_food_array is string.
match_stage["favoriteFoods"] = {'$elemMatch': lookup_food_string}
Though agree with find() is most effective in your usecase. Still there is $match of aggregation framework, to ease the query of a big number of entries and generate a low number of results that hold value to you especially for grouping and creating new files.
PersonModel.aggregate([
{
"$match": {
$and : [{ 'favouriteFoods' : { $exists: true, $in: [ 'sushi']}}, ........ ] }
},
{ $project : {"_id": 0, "name" : 1} }
]);
There are some ways to achieve this. First one is by $elemMatch operator:
const docs = await Documents.find({category: { $elemMatch: {$eq: 'yourCategory'} }});
// you may need to convert 'yourCategory' to ObjectId
Second one is by $in or $all operators:
const docs = await Documents.find({category: { $in: [yourCategory] }});
or
const docs = await Documents.find({category: { $all: [yourCategory] }});
// you can give more categories with these two approaches
//and again you may need to convert yourCategory to ObjectId
$in is like OR and $all like AND. For further details check this link : https://docs.mongodb.com/manual/reference/operator/query/all/
Third one is by aggregate() function:
const docs = await Documents.aggregate([
{ $unwind: '$category' },
{ $match: { 'category': mongoose.Types.ObjectId(yourCategory) } }
]};
with aggregate() you get only one category id in your category array.
I get this code snippets from my projects where I had to find docs with specific category/categories, so you can easily customize it according to your needs.
For Loopback3 all the examples given did not work for me, or as fast as using REST API anyway. But it helped me to figure out the exact answer I needed.
{"where":{"arrayAttribute":{ "all" :[String]}}}
In case You are searching in an Array of objects, you can use $elemMatch. For example:
PersonModel.find({ favoriteFoods : { $elemMatch: { name: "sushiOrAnytthing" }}});
With populate & $in this code will be useful.
ServiceCategory.find().populate({
path: "services",
match: { zipCodes: {$in: "10400"}},
populate: [
{
path: "offers",
},
],
});
If you'd want to use something like a "contains" operator through javascript, you can always use a Regular expression for that...
eg.
Say you want to retrieve a customer having "Bartolomew" as name
async function getBartolomew() {
const custStartWith_Bart = await Customers.find({name: /^Bart/ }); // Starts with Bart
const custEndWith_lomew = await Customers.find({name: /lomew$/ }); // Ends with lomew
const custContains_rtol = await Customers.find({name: /.*rtol.*/ }); // Contains rtol
console.log(custStartWith_Bart);
console.log(custEndWith_lomew);
console.log(custContains_rtol);
}
I know this topic is old, but for future people who could wonder the same question, another incredibly inefficient solution could be to do:
PersonModel.find({$where : 'this.favouriteFoods.indexOf("sushi") != -1'});
This avoids all optimisations by MongoDB so do not use in production code.

DynamoDB Update ExpressionAttributeValues

Sample test data:
"suppliers" : [
{
"supplierName": "xxx supplier"
},
{
"supplierName": "zzz supplier"
}
]
Excerpt of UpdateItem Params:
ExpressionAttributeValues:{
":sA" : {L: [event.suppliers]}
// ":sA" : {L: event.suppliers}
}
I encounter "UnexpectedParameter: Unexpected key '0' found in params". I tried changing the formatting of the AttributeValues but I get other errors like "UnexpectedParameter: Unexpected key 'supplierName' found in params".
I need help properly defining a list AttributeValue. Note that the input sample test data may contain x number of supplierName objects and hence defining the keys (ie. '0', '1', etc) in the AttributeValue is also not an option. I also prefer not to use the documentClient version of UpdateItem.
ExpressionAttributeValues is about to set a value to the parameter of the update expression, for example:
dynamoDb.update({
TableName: ...,
Key: ...,
UpdateExpression: 'set suppliers = :suppliers',
ExpressionAttributeValues: {':suppliers': event.suppliers}
}
You have to provide the key to the record and then you can change the suppliers attribute.

MongoDB Finding duplicates sharing multiple fields using MapReduce

I am trying to find duplicates in a Mongo version 2.4 database that is being used for production and therefore cannot be updated. Since aggregate does not exist in 2.4, I cannot use the aggregate pipeline to find duplicates, therefore I am trying to find a solution using MapReduce.
I have tried the following set of map, reduce, and finalize functions, through MongoVUE's Map Reduce interface, and they returned nothing after running for less than a second on a 3,000,000 record collection that definitely has duplicates on the indicated fields. Obviously something went wrong, but MongoVUE did not show any error messages or helpful indications.
function Map() {
emit(
{name: this.name, LocationId: this.LocationId,
version: this.version},
{count:1, ScrapeDate: this.ScrapeDate}
);
}
function Reduce(key, values) {
var reduced = {count:0, ScrapeDate:''2000-01-01''};
values.forEach(function(val) {
reduced.count += val.count;
if (reduced.ScrapeDate.localeCompare(val.ScrapeDate) < 0)
reduced.ScrapeDate=val.ScrapeDate;
});
return reduced;
return values[0];
}
function Finalize(key, reduced) {
if (reduced.count > 1)
return reduced;
}
I just need to find any instance of multiple records that share the same name, LocationId, and version, and ideally display the most recent ScrapeDate of such a record.
Your map-reduce code worked without any issues, though for a very small dataset. I think return values[0]; in the reduce function would be a copy paste error. You could try the same through the mongo shell.
Since aggregate does not exist in 2.4, I cannot use the aggregate pipeline to find duplicates, therefore I am trying to find a solution
using MapReduce.
You got it wrong here, db.collection.aggregate(pipeline, options) was introduced in the version 2.2.
Here is how it could be done with the aggregation framework, but it would not be preferred since your dataset is very huge, and the $sort operator has memory limit of 10% of RAM, in v2.4.
db.collection.aggregate(
[
// sort the records, based on the 'ScrapeDate' field, in descending order.
{$sort:{"ScrapeDate":-1}},
// group by the key fields, and take the 'ScrapeDate' of the first document,
// Since it is in sorted order, the first document would contain the
// highest field value.
{$group:{"_id":{"name":"$name","LocationId":"$LocationId","version":"$version"}
,"ScrapeDate":{$first:"$ScrapeDate"}
,"count":{$sum:1}}
},
// output only the group, having documents greater than 1.
{$match:{"count":{$gt:1}}}
]
);
Coming to your Map-reduce functions, it ran without issues on my test data.
db.collection.insert({"name":"c","LocationId":1,"version":1,"ScrapeDate":"2000-01-01"});
db.collection.insert({"name":"c","LocationId":1,"version":1,"ScrapeDate":"2001-01-01"});
db.collection.insert({"name":"c","LocationId":1,"version":1,"ScrapeDate":"2002-01-01"});
db.collection.insert({"name":"d","LocationId":1,"version":1,"ScrapeDate":"2002-01-01"});
running the map-reduce,
db.collection.mapReduce(Map,Reduce,{out:{"inline":1},finalize:Finalize});
o/p:
{
"results" : [
{
"_id" : {
"name" : "c",
"LocationId" : 1,
"version" : 1
},
"value" : {
"count" : 3,
"ScrapeDate" : "2002-01-01"
}
},
{
"_id" : {
"name" : "d",
"LocationId" : 1,
"version" : 1
},
"value" : null
}
],
"timeMillis" : 0,
"counts" : {
"input" : 4,
"emit" : 4,
"reduce" : 1,
"output" : 2
},
"ok" : 1,
}
Notice that the output contains value:null for a record which doesn't have any duplicates.
This is due to your finalize function:
function Finalize(key, reduced) {
if (reduced.count > 1)
return reduced; // returned null by default for keys with single value,
// i.e count=1
}
The finalize function do not filter out keys. So you can't get only the keys that are duplicates. You will get all the keys, in the map-reduce output. In your finalize functions, you can just not show their values, which is what you are doing.

Categories