We Keep Coding

Elasticsearch returns results greater than specified in range

I'm use Elasticsearch (Version: 6.8.4) with MongoDB (4.0.3)
I want find all the documents where price between 1 and 800 and sale_date is greater than Date.now(), but I have problem with my query:
{
"query": {
"bool": {
"must": [],
"filter": {
"term": {
"sold_out": false
},
"bool": {
"should": [
{
"range": {
"sale_date": {
"gt": Date.now()
}
}
},
{
"range": {
"price": {
"gt": 1,
"lte": 800
}
}
}
]
}
}
}
},
"from": 10,
"size": 200
}
It's query returns me results with products where some of them have price greater than 800
Price field stored in Elasticsearch as long
Whant I'm try:
use from: to: and got the same results
change "should" to "must" in filter and it's returns empty results
remove from query { "range": { "sale_date": { "gt": Date.now(), } }
} and it's returns right results!
What I'm doing wrong ?

First remove the uneccessary nesting ofthe two bool objects, try having all the clauses in the same "bool" field like this.
Remeber that with must a document must have the term you are making a comparison on to be included in the result, with should a match will only improve the score, if the condition is met, so in a certain sense must is like AND and should similar to an OR.
In your example changing the filter from should to must made the query return nothing maybe because you don't have any element that has both the date and price you want, or maybe it was the problem was the bool nesting.
Try this:
{
"query": {
"bool": {
"must": [{
"range": {
"sale_date": {
"gt": Date.now()
}
}
},
{
"range": {
"price": {
"gt": 1,
"lte": 800
}
}
}],
"filter": {
"term": {
"sold_out": false
},
}
}
},
"from": 10,
"size": 200
}
This will only get the elements with price lte 800 and sale_date gt Date.now().

Change Date().now() to "gt": "now/d".

ElasticSearch EJS query to fetch missing field

I have an existing ejs query as below:
let queryBody = ejs.Request()
.size(0)
.query(
ejs.BoolQuery()
.must(
ejs.RangeQuery('hour_time_stamp').gte(this.lastDeviceDate).lte(this.lastDeviceDate)
)
)
.agg(ejs.TermsAggregation('market_agg').field('market').order('sum', 'asc').size(50000)
.agg(ejs.SumAggregation('sum').field('num_devices'))
)
currently the field('market') returns the values where data for market is present. There is data in the database for missing values for market as well, which I need to access. How do I do that?
EDIT:
Values for market in ES is either null or field is missing. I wrote ES query to get all those fields but I am not able to incorporate an ejs query for the same. Any idea how this can be done?
{
"query": {
"bool": {
"should": [
{
"exists": {
"field": "market"
}
},
{
"bool": {
"must_not": [
{
"exists": {
"field": "market"
}
}
]
}
}
]
}
}
}

As per your problem you need a way to group the empty market fields too.
So for that you can use the "missing" value parameter. It defines how the values which are missing(as in your case) are grouped. So you query in json form will be modified like below :-
{
"query":
{
"must": [
"range": {
"hour_time_stamp": {
"gte": lastDeviceDate,
"lte": lastDeviceDate
}
}
]
},
"aggs": {
"market_agg" : {
"market": {
"missing": "empty_markets",
"order": { "sum": "asc" }
}
},
"sum_agg": {
"sum" : { "field" : "num_devices" }
}
}
}
Or in your code it could be done by adding missing parameter like this.
let queryBody = ejs.Request()
.size(0)
.query(
ejs.BoolQuery()
.must(
ejs.RangeQuery('hour_time_stamp').gte(this.lastDeviceDate).lte(this.lastDeviceDate)
)
)
.agg(ejs.TermsAggregation('market_agg').field('market').missing('empty_markets').order('sum', 'asc').size(50000)
.agg(ejs.SumAggregation('sum').field('num_devices'))
)

How to Remove Unwanted Fields from Output by Condition

I have a projection stage as follows,
{
'name': {$ifNull: [ '$invName', {} ]},,
'info.type': {$ifNull: [ '$invType', {} ]},
'info.qty': {$ifNull: [ '$invQty', {} ]},
'info.detailed.desc': {$ifNull: [ '$invDesc', {} ]}
}
I am projecting empty object({}) in case of a field not present, because if sorting is performed in a field and the field doesn't exist, that document is coming first in sort order(Sort Documents Without Existing Field to End of Results). Next stage is sorting and wanted non-existing fields to come last in sorting order. This is working as expected.
Now, I want to remove those fields which are having empty object as values(if info.detailed.desc is empty info.detailed should not be there in output). I could do this in node level using lodash like this(https://stackoverflow.com/a/38278831/6048928). But I am trying to do this in mongodb level. Is it possible? I tried $redact, but it is filtering out entire document. Is is possible to PRUNE or DESCEND fields of a document based on value?

Removing properties completely from documents is not a trivial thing. The basics are that the server itself has not had any way of doing this prior to MongoDB 3.4 and the introduction of $replaceRoot, which essentially allows an expression to be returned as the document context.
Even with that addition it's somewhat impractical to do so without further features of $objectToArray and $arrayToObject as introduced in MongoDB 3.4.4. But to run through the cases.
Working with a quick sample
{ "_id" : ObjectId("59adff0aad465e105d91374c"), "a" : 1 }
{ "_id" : ObjectId("59adff0aad465e105d91374d"), "a" : {} }
Conditionally return root object
db.junk.aggregate([
{ "$replaceRoot": {
"newRoot": {
"$cond": {
"if": { "$ne": [ "$a", {} ] },
"then": "$$ROOT",
"else": { "_id": "$_id" }
}
}
}}
])
That's a pretty simple principle and can in fact be applied to any nested property to remove it's sub-keys but would require various levels of nesting $cond or even $switch to apply possible conditions. The $replaceRoot of course is needed for "top level" removal since it's the only way to conditionally express top level keys to return.
So whilst you can in theory use $cond or $switch to decide what to return, it's generally cumbersome and you would want something more flexible.
Filter the Empty Objects
db.junk.aggregate([
{ "$replaceRoot": {
"newRoot": {
"$arrayToObject": {
"$filter": {
"input": { "$objectToArray": "$$ROOT" },
"cond": { "$ne": [ "$$this.v", {} ] }
}
}
}
}}
])
This is where $objectToArray and $arrayToObject come into use. Instead of writing out the conditions for every possibly key we just convert the object contents into an "array" and apply $filter on the array entries to decide what to keep.
The $objectToArray translates any object into an array of documents representing each property as "k" for the name of the key and "v" for the value from that property. Since these are now accessible as "values", then you can use methods like $filter to inspect the each array entry and discard the unwanted ones.
Finally $arrayToObject takes the "filtered" content and translates those "k" and "v" values back into property names and values as a resulting object. In this way, the "filter" conditions removes any properties from the result object that did not meet the criteria.
A Return to $cond
db.junk.aggregate([
{ "$project": {
"a": { "$cond": [{ "$eq": [ "$a", {} ] }, "$$REMOVE", "$a" ] }
}}
])
MongoDB 3.6 introduces a new player with the $$REMOVE constant. This is a new feature that can be applied with $cond in order to decide whether or not to show the property at all. So that is another approach when of course the release is available.
In all those above cases the "a" property is not returned when the value is the empty object that we wanted to test for removal.
{ "_id" : ObjectId("59adff0aad465e105d91374c"), "a" : 1 }
{ "_id" : ObjectId("59adff0aad465e105d91374d") }
More Complex Structures
Your specific ask here is for data containing nested properties. So continuing on from the outlined approaches we can work with demonstrating how that is done.
First some sample data:
{ "_id" : ObjectId("59ae03bdad465e105d913750"), "a" : 1, "info" : { "type" : 1, "qty" : 2, "detailed" : { "desc" : "this thing" } } }
{ "_id" : ObjectId("59ae03bdad465e105d913751"), "a" : 2, "info" : { "type" : 2, "qty" : 3, "detailed" : { "desc" : { } } } }
{ "_id" : ObjectId("59ae03bdad465e105d913752"), "a" : 3, "info" : { "type" : 3, "qty" : { }, "detailed" : { "desc" : { } } } }
{ "_id" : ObjectId("59ae03bdad465e105d913753"), "a" : 4, "info" : { "type" : { }, "qty" : { }, "detailed" : { "desc" : { } } } }
Applying the filter method
db.junk.aggregate([
{ "$replaceRoot": {
"newRoot": {
"$arrayToObject": {
"$filter": {
"input": {
"$concatArrays": [
{ "$filter": {
"input": { "$objectToArray": "$$ROOT" },
"cond": { "$ne": [ "$$this.k", "info" ] }
}},
[
{
"k": "info",
"v": {
"$arrayToObject": {
"$filter": {
"input": { "$objectToArray": "$info" },
"cond": {
"$not": {
"$or": [
{ "$eq": [ "$$this.v", {} ] },
{ "$eq": [ "$$this.v.desc", {} ] }
]
}
}
}
}
}
}
]
]
},
"cond": { "$ne": [ "$$this.v", {} ] }
}
}
}
}}
])
This needs more complex handling because of the nested levels. In the main case here you need to look at the "info" key here independently and remove any sub-properties that do not qualify first. Since you need to return "something", we basically then need to remove the "info" key itself when all of it's inner properties are removed. This is the reason for the nested filter operations on each set of results.
Applying $cond with $$REMOVE
Where available this would at first seem a more logical choice, so it helps to look at this from the most simplified form first:
db.junk.aggregate([
{ "$addFields": {
"info.type": {
"$cond": [
{ "$eq": [ "$info.type", {} ] },
"$$REMOVE",
"$info.type"
]
},
"info.qty": {
"$cond": [
{ "$eq": [ "$info.qty", {} ] },
"$$REMOVE",
"$info.qty"
]
},
"info.detailed.desc": {
"$cond": [
{ "$eq": [ "$info.detailed.desc", {} ] },
"$$REMOVE",
"$info.detailed.desc"
]
}
}}
])
But then you need to look at the output this actually produces:
/* 1 */
{
"_id" : ObjectId("59ae03bdad465e105d913750"),
"a" : 1.0,
"info" : {
"type" : 1.0,
"qty" : 2.0,
"detailed" : {
"desc" : "this thing"
}
}
}
/* 2 */
{
"_id" : ObjectId("59ae03bdad465e105d913751"),
"a" : 2.0,
"info" : {
"type" : 2.0,
"qty" : 3.0,
"detailed" : {}
}
}
/* 3 */
{
"_id" : ObjectId("59ae03bdad465e105d913752"),
"a" : 3.0,
"info" : {
"type" : 3.0,
"detailed" : {}
}
}
/* 4 */
{
"_id" : ObjectId("59ae03bdad465e105d913753"),
"a" : 4.0,
"info" : {
"detailed" : {}
}
}
Whilst the other keys are removed the "info.detailed" still stays around because there is nothing that actually tests at this level. In fact you simply cannot express this in simple terms, so the only way to work around this is to evaluate the object as an expression and then apply additional filtering an conditions on each level of output to see where the empty objects still reside, and remove them:
db.junk.aggregate([
{ "$addFields": {
"info": {
"$let": {
"vars": {
"info": {
"$arrayToObject": {
"$filter": {
"input": {
"$objectToArray": {
"type": { "$cond": [ { "$eq": [ "$info.type", {} ] },"$$REMOVE", "$info.type" ] },
"qty": { "$cond": [ { "$eq": [ "$info.qty", {} ] },"$$REMOVE", "$info.qty" ] },
"detailed": {
"desc": { "$cond": [ { "$eq": [ "$info.detailed.desc", {} ] },"$$REMOVE", "$info.detailed.desc" ] }
}
}
},
"cond": { "$ne": [ "$$this.v", {} ] }
}
}
}
},
"in": { "$cond": [ { "$eq": [ "$$info", {} ] }, "$$REMOVE", "$$info" ] }
}
}
}}
])
That approach as with the plain $filter method actually removes "all" empty objects from the results:
/* 1 */
{
"_id" : ObjectId("59ae03bdad465e105d913750"),
"a" : 1.0,
"info" : {
"type" : 1.0,
"qty" : 2.0,
"detailed" : {
"desc" : "this thing"
}
}
}
/* 2 */
{
"_id" : ObjectId("59ae03bdad465e105d913751"),
"a" : 2.0,
"info" : {
"type" : 2.0,
"qty" : 3.0
}
}
/* 3 */
{
"_id" : ObjectId("59ae03bdad465e105d913752"),
"a" : 3.0,
"info" : {
"type" : 3.0
}
}
/* 4 */
{
"_id" : ObjectId("59ae03bdad465e105d913753"),
"a" : 4.0
}
Doing it all in Code
So everything here really depends on latest features or indeed "coming features" to be available in the MongoDB version you are using. Where these are not available the alternate approach is to simply remove the empty objects from the results returned by the cursor.
It's often the most sane thing to do, and really is all you require unless the aggregation pipeline needs to continue past the point where the fields are being removed. Even then, you probably should be logically working around that and leave the final results to cursor processing.
As JavaScript for the shell you can use the following approach, and the principles essentially stay the same no matter which actual language implementation:
db.junk.find().map( d => {
let info = Object.keys(d.info)
.map( k => ({ k, v: d.info[k] }))
.filter(e => !(
typeof e.v === 'object' &&
( Object.keys(e.v).length === 0 || Object.keys(e.v.desc).length === 0 )
))
.reduce((acc,curr) => Object.assign(acc,{ [curr.k]: curr.v }),{});
delete d.info;
return Object.assign(d,(Object.keys(info).length !== 0) ? { info } : {})
})
Which is pretty much the native language way of stating the same as the examples above being that where one of the expected properties contains an empty object, remove that property from the output completely.

I have removed the brands object in the output JSON using $project at end of the aggregation pipeline
db.Product.aggregate([
{
$lookup: {
from: "wishlists",
let: { product: "$_id" },
pipeline: [
{
$match: {
$and: [
{ $expr: { $eq: ["$$product", "$product"] } },
{ user: userId }
]
}
}
],
as: "isLiked"
}
},
{
$lookup: {
from: "brands",
localField: "brand",
foreignField: "_id",
as: "brands"
}
},
{
$addFields: {
isLiked: { $arrayElemAt: ["$isLiked.isLiked", 0] }
}
},
{
$unwind: "$brands"
},
{
$addFields: {
"brand.name": "$brands.name" ,
"brand._id": "$brands._id"
}
},
{
$match:{ isActive: true }
},
{
$project: { "brands" : 0 }
}
]);

$group: {
_id: '$_id',
tasks: {
$addToSet: {
$cond: {
if: {
$eq: [
{
$ifNull: ['$tasks.id', ''],
},
'',
],
},
then: '$$REMOVE',
else: {
id: '$tasks.id',
description: '$tasks.description',
assignee: {
$cond: {
if: {
$eq: [
{
$ifNull: ['$tasks.assignee._id', ''],
},
'',
],
},
then: undefined,
else: {
id: '$tasks.assignee._id',
name: '$tasks.assignee.name',
thumbnail: '$tasks.assignee.thumbnail',
status: '$tasks.assignee.status',
},
},
},
},
},
},
},
}

Manipulating MongoDB response NodeJS

I made a "little" query for mongodb to join two collections and retrieve data.
The game: insert 2 or 3 params on a URL
-include can be 0,1 or 2.
0 exclusive
1 inclusive
2 return all
-netcode: is a key to filter data
-group: another optional keys, that works with the first param "include"
-My query works perfectly, returns in a way how much times a event happened in a certain group.
-The problem? I can't work with the result of mongo db, i need to parse it to JSON.
I'm not so clever at JS, so i don't know where to put it. Since i work in corporation, some of the code was already done.
Well my output is this:
{
"events": [
{
"_id": {
"group": "GFS-CAJEROS-INFINITUM-TELDAT-M1",
"event": "SNMP DOWN"
},
"incidencias": 1
},
{
"_id": {
"group": "GFS-CAJEROS-MPLS",
"event": "Proactive Interface Input Utilisation"
},
"incidencias": 1209
},
{
"_id": {
"group": "GFS-CAJEROS-MPLS",
"event": "Proactive Interface Output Utilisation"
},
"incidencias": 1209
},
{
"_id": {
"group": "GFS-CAJEROS-MPLS",
"event": "Proactive Interface Availability"
},
"incidencias": 2199
},
{
"_id": {
"group": "GFS-SUCURSALES-HIBRIDAS",
"event": "Proactive Interface Output Utilisation"
},
"incidencias": 10
},
But i want it fused in a JSON format, like this: check the int value is next for the name of the event.
[
{
"group": "GFS-CAJEROS-MPLS",
"Proactive Interface Input Utilisation" : "1209",
"Proactive Interface Output Utilisation" : "1209",
"Proactive Interface Availability" : "2199",
},
{
"group": "GFS-SUCURSALES-HIBRIDAS",
"Proactive Interface Output Utilisation" : "10",
},
I'm using Nodejs and the mongodb module, since i dont know how this function exactly works, i don't know how to manage the response, ¿there is a better way to do this? like to get the json file, using another js to generate it?
This is the code i'm using, basically is the important part:
var events = db.collection('events');
events.aggregate([
{ $match : { netcode : data.params.netcode } },
{
$lookup:
{
from: "nodes",
localField: "name",
foreignField: "name",
as: "event_joined"
}
},
{ $unwind: {path: "$event_joined"} },
{ $match : {"event_joined.group" :
{$in:
[
groups[0] ,
groups[1] ,
groups[2] ,
groups[3] ,
groups[4] ,
groups[5] ,
groups[6] ,
groups[7] ,
groups[8] ,
groups[9] ,
]
}
}
},
{ $group : { _id : {group:"$event_joined.group", event:"$event"}, incidencias: { $sum: 1} } },
])
.toArray( function(err, result) {
if (err) {
console.log(err);
} else if (result) {
data.response.events = result;
} else {
console.log("No result");
}

You should add another $group to your pipeline {_id: "$_id.group", events: {$push : {name: "$_id.event", incidencias: "$incidencias"}}}
Then change the structure of your data on the JS code with "Array.map".
data.response.events = data.response.events.map(function (eve){
var obj = {
"group": eve.group
};
eve.events.forEach(function (e){
obj[e.name] = e.incidencias
})
return obj;
})

Mongo Aggregate: how to compare with a field from another collection?

I am trying to implement a function that collects unread messages from an articles collection. Each article in the collection has a "discussions" entry with discussion comment subdocuments. An example of such a subdocument is:
{
"id": NumberLong(7534),
"user": DBRef("users", ObjectId("...")),
"dt_create": ISODate("2015-01-26T00:10:44Z"),
"content": "The discussion comment content"
}
The parent document has the following (partial) structure:
{
model: {
id: 17676,
title: "Article title",
author: DBRef("users", ObjectId(...)),
// a bunch of other fields here
},
statistics: {
// Statistics will be stored here (pageviews, etc)
},
discussions: [
// Array of discussion subdocuments, like the one above
]
}
Each user also has a last_viewed entry which is a document, an example is as follows:
{
"17676" : "2015-01-10T00:00:00.000Z",
"18038" : "2015-01-10T00:00:00.000Z",
"18242" : "2015-01-20T00:00:00.000Z",
"18325" : "2015-01-20T00:00:00.000Z"
}
This means that the user has looked at discussion comments for the last time on January 10th 2015 for articles with IDs 17676 and 18038, and on January 20th 2015 for articles with IDs 18242 and 18325.
So I want to collect discussion entries from the article documents, and for article with ID 17676, I want to collect the discussion entries that were created after 2015-01-10, and for article with ID 18242, I want to show the discussion entries created after 2015-01-20.
UPDATED
Based on Neil Lunn's reply, the function I have created so far is:
function getUnreadDiscussions(userid) {
user = db.users.findOne({ 'model.id': userid });
last_viewed = [];
for(var i in user.last_viewed) {
last_viewed.push({
'id': parseInt(i),
'dt': user.last_viewed[i]
});
}
result = db.articles.aggregate([
// For now, collect just articles the user has written
{ $match: { 'model.author': DBRef('users', user._id) } },
{ $unwind: '$discussions' },
{ $project: {
'model': '$model',
'discussions': '$discussions',
'last_viewed': {
'$let': {
'vars': { 'last_viewed': last_viewed },
'in': {
'$setDifference': [
{ '$map': {
'input': '$$last_viewed',
'as': 'last_viewed',
'in': {
'$cond': [
{ '$eq': [ '$$last_viewed.id', '$model.id' ] },
'$$last_viewed.dt',
false
]
}
} },
[ false ]
]
}
}
}
}
},
// To get a scalar instead of a 1-element array:
{ $unwind: '$last_viewed' },
// Match only those that were created after last_viewed
{ $match: { 'discussions.dt_create': { $gt: '$last_viewed' } } },
{ $project: {
'model.id': 1,
'model.title': 1,
'discussions': 1,
'last_viewed': 1
} }
]);
return result.toArray();
}
The whole $let thing, and the $unwind after that, transforms the data into the following partial projection (with the last $match commented out):
{
"_id" : ObjectId("54d9af1dca71d8054c8d0ee3"),
"model" : {
"id" : NumberLong(18325),
"title" : "Article title"
},
"discussions" : {
"id" : NumberLong(7543),
"user" : DBRef("users", ObjectId("54d9ae24ca71d8054c8b4567")),
"dt_create" : ISODate("2015-01-26T00:10:44Z"),
"content" : "Some comment here"
},
"last_viewed" : ISODate("2015-01-20T00:00:00Z")
},
{
"_id" : ObjectId("54d9af1dca71d8054c8d0ee3"),
"model" : {
"id" : NumberLong(18325),
"title" : "Article title"
},
"discussions" : {
"id" : NumberLong(7554),
"user" : DBRef("users", ObjectId("54d9ae24ca71d8054c8b4567")),
"dt_create" : ISODate("2015-01-26T02:03:22Z"),
"content" : "Another comment here"
},
"last_viewed" : ISODate("2015-01-20T00:00:00Z")
}
So far so good here. But the problem now is that the $match to select only the discussions created after the last_viewed date is not working. I am getting an empty array response. However, if I hard-code the date and put in $match: { 'discussions.dt_create': { $gt: ISODate("2015-01-20 00:00:00") } }, it works. But I want it to take it from last_viewed.

I found another SO thread where this issue has been resolved by using the $cmp operator.
The final part of the aggregation would be:
[
{ /* $match, $unwind, $project, $unwind as before */ },
{ $project: {
'model': 1,
'discussions': 1,
'last_viewed': 1,
'compare': {
$cmp: [ '$discussions.dt_create', '$last_viewed' ]
}
} },
{ $match: { 'compare': { $gt: 0 } } }
]
The aggregation framework is great, but it takes quite a different approach in problem-solving. Hope this helps anyone!
I'll keep the question unanswered in case anyone else has a better answer/method. If this answer has been upvoted enough times, I'll accept this one.