MongoDB group by date and select values - javascript

I'm relatively new to MongoDB and node.js. Here is what I'm trying to achieve:
I have a collection that contains several thousand documents in MongoDB. The documents look like this:
{
"date": "2020-02-24",
"iso_code": "USA",
"country": "USA",
"avg_temperature": "25F",
},
{
"date": "2020-02-25",
"iso_code": "USA",
"country": "USA",
"avg_temperature": "27F",
},
{
"date": "2020-02-24",
"iso_code": "CHN",
"country": "China",
"avg_temperature": "10C",
}
I want to avoid running costly calculation in node.js, so I would like to receive the temperatures grouped by date and retrieve the actual values for avg_temperature (so no $min or $max calculation). I expect the result to look something like this:
{
"date": "2020-02-24",
"avg_temperatures": {
"USA": "25F",
"China": "10C"
}
}
My ultimate goal is to create a chart that visualizes the data over a timeline for multiple countries. My idea is to simply iterate through the contents of the result and then trough the temperatures and display the data.
If someone has done something similar before, feel free to suggest a better alternative :)

apparently getting the result as you described will be a little complex. Instead, you can do it by making a simple aggregation pipeline like this
var pipeline = [
{
"$group" : {
"_id" : "$date",
"docs" : {
"$push" : {
"country" : "$country",
"temp" : "$avg_temperature"
}
}
}
}
];
For the above pipeline you will get the output like
db.stack.aggregate(pipeline).pretty()
{
"_id" : "2020-02-25",
"docs" : [
{
"country" : "USA",
"temp" : "27F"
}
]
}
{
"_id" : "2020-02-24",
"docs" : [
{
"country" : "USA",
"temp" : "25F"
},
{
"country" : "China",
"temp" : "10C"
}
]
}
I hope this will solve your problem!

You can do it with Aggregation framework:
db.collection.aggregate([
{
"$group": {
"_id": "$date",
"avg_temperatures": {
"$mergeObjects": {
"$arrayToObject": [
[
{
k: "$country",
v: "$avg_temperature"
}
]
]
}
}
}
}
])
Working example

To increase performance, first it would be good to create index for "date" field in your collection. By doing that, group by date in aggregation pipeline will be much faster. You can do that by adding this in your file where you have defined Schema for your collection. "Schema" is name of schema for your collection.
Schema.index({
date: 1,
});
Do not forget to rebuild indexes for specific collection. You can do that by typing this in mongo shell, where "collection" is name of your collection:
db.collection.reIndex()
And you need to add group stage in aggregation pipeline.
{
"$group" : {
"_id" : "$date",
"docs": { $push : "$$ROOT" },
}
}

Related

Don't replace null with zero in MongoDB

I wrote a mongo shell script to insert in a new collection the result of an aggregation; my problem is that the null values from the source collection are replaced by zero; the aggregation part is:
db.getCollection("unhcr_pop_concern_flat").aggregate(
[
{
"$group" : {
"_id" : {
"country" : "$country",
"time" : "$time",
"origin": "$origin"
},
"value_fields": {
"$addToSet": {
k: "$type",
v: {$ifNull: ["$value", null]}
}
}
}
},
{ $addFields: { value_fields: { $arrayToObject: "$value_fields" } } }
],
{
"allowDiskUse" : true
}
Where is the problem? I have also used the $ifnull operator.
The query is working correctly; there was a problem in the source collections where data were changed by others!

How to add within an array information using findOneAndUpdate without deleting information that was previously contained [duplicate]

I am working on an express js application where I need to update a nested array.
1) Schema :
//Creating a mongoose schema
var userSchema = mongoose.Schema({
_id: {type: String, required:true},
name: String,
sensors: [{
sensor_name: {type: String, required:true},
measurements: [{time: String}]
}] });
2)
Here is the code snippet and explanation is below:
router.route('/sensors_update/:_id/:sensor_name/')
.post(function (req, res) {
User.findOneAndUpdate({_id:req.body._id}, {$push: {"sensors" :
{"sensor_name" : req.body.sensor_name , "measurements.0.time": req.body.time } } },
{new:true},function(err, newSensor) {
if (err)
res.send(err);
res.send(newSensor)
}); });
I am able to successfully update a value to the measurements array using the findOneAndUpdate with push technique but I'm failing when I try to add multiple measurements to the sensors array.
Here is current json I get if I get when I post a second measurement to the sensors array :
{
"_id": "Manasa",
"name": "Manasa Sub",
"__v": 0,
"sensors": [
{
"sensor_name": "ras",
"_id": "57da0a4bf3884d1fb2234c74",
"measurements": [
{
"time": "8:00"
}
]
},
{
"sensor_name": "ras",
"_id": "57da0a68f3884d1fb2234c75",
"measurements": [
{
"time": "9:00"
}
]
}]}
But the right format I want is posting multiple measurements with the sensors array like this :
Right JSON format would be :
{
"_id" : "Manasa",
"name" : "Manasa Sub",
"sensors" : [
{
"sensor_name" : "ras",
"_id" : ObjectId("57da0a4bf3884d1fb2234c74"),
"measurements" : [
{
"time" : "8:00"
}
],
"measurements" : [
{
"time" : "9:00"
}
]
}],
"__v" : 0 }
Please suggest some ideas regarding this. Thanks in advance.
You might want to rethink your data model. As it is currently, you cannot accomplish what you want. The sensors field refers to an array. In the ideal document format that you have provided, you have a single object inside that array. Then inside that object, you have two fields with the exact same key. In a JSON object, or mongo document in this context, you can't have duplicate keys within the same object.
It's not clear exactly what you're looking for here, but perhaps it would be best to go for something like this:
{
"_id" : "Manasa",
"name" : "Manasa Sub",
"sensors" : [
{
"sensor_name" : "ras",
"_id" : ObjectId("57da0a4bf3884d1fb2234c74"),
"measurements" : [
{
"time" : "8:00"
},
{
"time" : "9:00"
}
]
},
{
// next sensor in the sensors array with similar format
"_id": "",
"name": "",
"measurements": []
}],
}
If this is what you want, then you can try this:
User.findOneAndUpdate(
{ _id:req.body._id "sensors.sensor_name": req.body.sensor_name },
{ $push: { "sensors.0.measurements": { "time": req.body.time } } }
);
And as a side note, if you're only ever going to store a single string in each object in the measurements array, you might want to just store the actual values instead of the whole object { time: "value" }. You might find the data easier to handle this way.
Instead of hardcoding the index of the array it is possible to use identifier and positional operator $.
Example:
User.findOneAndUpdate(
{ _id: "Manasa" },
{ $push: { "sensors.$[outer].measurements": { "time": req.body.time } } }
{ "arrayFilters:" [{"outer._id": ObjectId("57da0a4bf3884d1fb2234c74")}]
);
You may notice than instead of getting a first element of the array I specified which element of the sensors array I would like to update by providing its ObjectId.
Note that arrayFilters are passed as the third argument to the update query as an option.
You could now make "outer._id" dynamic by passing the ObjectId of the sensor like so: {"outer._id": req.body.sensorId}
In general, with the use of identifier, you can get to even deeper nested array elements by following the same procedure and adding more filters.
If there was a third level nesting you could then do something like:
User.findOneAndUpdate(
{ _id: "Manasa" },
{ $push: { "sensors.$[outer].measurements.$[inner].example": { "time": req.body.time } } }
{ "arrayFilters:" [{"outer._id": ObjectId("57da0a4bf3884d1fb2234c74"), {"inner._id": ObjectId("57da0a4bf3884d1fb2234c74"}}]
);
You can find more details here in the answer written by Neil Lunn.
refer ::: positional-all
--- conditions :: { other_conditions, 'array1.array2.field_to_be_checked': 'value' }
--- updateData ::: { $push : { 'array1.$[].array2.$[].array3' : 'value_to_be_pushed' } }

Manipulating MongoDB response NodeJS

I made a "little" query for mongodb to join two collections and retrieve data.
The game: insert 2 or 3 params on a URL
-include can be 0,1 or 2.
0 exclusive
1 inclusive
2 return all
-netcode: is a key to filter data
-group: another optional keys, that works with the first param "include"
-My query works perfectly, returns in a way how much times a event happened in a certain group.
-The problem? I can't work with the result of mongo db, i need to parse it to JSON.
I'm not so clever at JS, so i don't know where to put it. Since i work in corporation, some of the code was already done.
Well my output is this:
{
"events": [
{
"_id": {
"group": "GFS-CAJEROS-INFINITUM-TELDAT-M1",
"event": "SNMP DOWN"
},
"incidencias": 1
},
{
"_id": {
"group": "GFS-CAJEROS-MPLS",
"event": "Proactive Interface Input Utilisation"
},
"incidencias": 1209
},
{
"_id": {
"group": "GFS-CAJEROS-MPLS",
"event": "Proactive Interface Output Utilisation"
},
"incidencias": 1209
},
{
"_id": {
"group": "GFS-CAJEROS-MPLS",
"event": "Proactive Interface Availability"
},
"incidencias": 2199
},
{
"_id": {
"group": "GFS-SUCURSALES-HIBRIDAS",
"event": "Proactive Interface Output Utilisation"
},
"incidencias": 10
},
But i want it fused in a JSON format, like this: check the int value is next for the name of the event.
[
{
"group": "GFS-CAJEROS-MPLS",
"Proactive Interface Input Utilisation" : "1209",
"Proactive Interface Output Utilisation" : "1209",
"Proactive Interface Availability" : "2199",
},
{
"group": "GFS-SUCURSALES-HIBRIDAS",
"Proactive Interface Output Utilisation" : "10",
},
I'm using Nodejs and the mongodb module, since i dont know how this function exactly works, i don't know how to manage the response, ¿there is a better way to do this? like to get the json file, using another js to generate it?
This is the code i'm using, basically is the important part:
var events = db.collection('events');
events.aggregate([
{ $match : { netcode : data.params.netcode } },
{
$lookup:
{
from: "nodes",
localField: "name",
foreignField: "name",
as: "event_joined"
}
},
{ $unwind: {path: "$event_joined"} },
{ $match : {"event_joined.group" :
{$in:
[
groups[0] ,
groups[1] ,
groups[2] ,
groups[3] ,
groups[4] ,
groups[5] ,
groups[6] ,
groups[7] ,
groups[8] ,
groups[9] ,
]
}
}
},
{ $group : { _id : {group:"$event_joined.group", event:"$event"}, incidencias: { $sum: 1} } },
])
.toArray( function(err, result) {
if (err) {
console.log(err);
} else if (result) {
data.response.events = result;
} else {
console.log("No result");
}
You should add another $group to your pipeline {_id: "$_id.group", events: {$push : {name: "$_id.event", incidencias: "$incidencias"}}}
Then change the structure of your data on the JS code with "Array.map".
data.response.events = data.response.events.map(function (eve){
var obj = {
"group": eve.group
};
eve.events.forEach(function (e){
obj[e.name] = e.incidencias
})
return obj;
})

Meteor Mongodb count fields in nested object

I am trying to create a dashboard where I show summaries of order data within the app. In this case I am simply wanting count the number of items in a given category in my Orders collection. My code so far looks like this:
Collection data
{
"_id" : "a6wHiXxyM5DwSAsfq",
"orderNumber" : 1234,
"createdAt" : "11/01/2016, 14:43:49",
"productsInOrder" : [
{
"category" : "ambient",
"item" : 50818,
"desc" : "Tasty Rubber Chicken",
"quantity" : "44",
"price" : "0.92",
"lineprice" : "40.48",
"_id" : "FFNxG8vujs6NGN69r"
},
{
"category" : "frozen",
"item" : 71390,
"desc" : "Generic Granite Fish",
"quantity" : "11",
"price" : "1.00",
"lineprice" : "11.00",
"_id" : "LcRtpyLxkWyh39kkB"
}
]
}
{
"_id" : "PdpywXCvfew7qojmA",
"orderNumber" : 1234,
"createdAt" : "11/01/2016, 14:44:15",
"productsInOrder" : [
{
"category" : "frozen",
"item" : 71390,
"desc" : "Generic Granite Fish",
"quantity" : "44",
"price" : "1.00",
"lineprice" : "44.00",
"_id" : "dAscx4R8pcBgbzoZs"
},
{
"category" : "frozen",
"item" : 66940,
"desc" : "Gorgeous Granite Bike",
"quantity" : "55",
"price" : "4.21",
"lineprice" : "231.55",
"_id" : "xm3mFRmPmmdPxjfP9"
},
{
"category" : "frozen",
"item" : 96029,
"desc" : "Gorgeous Plastic Fish",
"quantity" : "1234",
"price" : "4.39",
"lineprice" : "5417.26",
"_id" : "7u7SHnpTf7PWcrhGA"
}
]
}
{
"_id" : "xcHZ25qwvyDpDJtAZ",
"orderNumber" : 1234,
"createdAt" : "11/01/2016, 14:44:47",
"productsInOrder" : [
{
"category" : "frozen",
"item" : 31104,
"desc" : "Handcrafted Rubber Keyboard",
"quantity" : "11",
"price" : "4.78",
"lineprice" : "52.58",
"_id" : "LMMwbKFEgnCbgCt9c"
},
{
"category" : "frozen",
"item" : 77832,
"desc" : "Practical Rubber Shirt",
"quantity" : "21",
"price" : "0.62",
"lineprice" : "13.02",
"_id" : "63otkkXWGrTJkwEgX"
},
{
"category" : "frozen",
"item" : 66940,
"desc" : "Gorgeous Granite Bike",
"quantity" : "111",
"price" : "4.21",
"lineprice" : "467.31",
"_id" : "rbPSujey8CFeMPjza"
}
]
}
JS
So far I have tried:
Orders.find({ 'productsInOrder': ['ambient']}).count();
Orders.find({ productsInOrder: { category: 'ambient' }}).count();
Orders.find({ productsInOrder: { $all: [ 'frozen' ] }}).count();
I am having a hard time understanding Mongo queries when the data is nested in this manner. Please can you help point me in the right direction? Many thanks in advance.
* SOLUTION *
I have accomplished the desired result thanks to the contributions below. To make this work I created a method on the server as the query cannot be run on the client using an existing collection. This is done as follows:
Meteor.methods({
'byCategory': function() {
var result = Orders.aggregate([
{ "$unwind": "$productsInOrder" },
{
"$group": {
"_id": null,
"ambient_count": {
"$sum": {
"$cond": [ { "$eq": [ "$productsInOrder.category", "ambient" ] }, 1, 0 ]
}
},
"frozen_count": {
"$sum": {
"$cond": [ { "$eq": [ "$productsInOrder.category", "frozen" ] }, 1, 0 ]
}
},
"other_category_count": {
"$sum": {
"$cond": [ { "$eq": [ "$productsInOrder.category", "other_category" ] }, 1, 0 ]
}
}
}
}
]);
return result;
}
})
and then on the client:
Meteor.call('byCategory', function( error, result ) {
if( error ) {
console.log( error.reason );
} else {
console.log( result[0].ambient_count );
console.log( result[0].frozen_count );
etc....
}
});
Thanks and credit to #chridam and #Brett.
An alternative approach is to use the aggregation framework. Consider the following aggregation pipeline which as the first stage of the aggregation pipeline, the $unwind operator denormalizes the productsInOrder array to output for each input document, n documents where n is the number of array elements. The next pipeline stage has the $group operator which groups all the documents into a single document and stores the counts for each category with the help of the $sum and $cond operators.
In Meteor, you can then use meteorhacks:aggregate package to implement the aggregation:
Add to your app with
meteor add meteorhacks:aggregate
Note, this only works on server side and there is no oberserving support or reactivity built in. Then simply use .aggregate function like below.
var coll = new Mongo.Collection('orders');
var pipeline = [
{ "$unwind": "$productsInOrder" },
{
"$group": {
"_id": null,
"ambient_count": {
"$sum": {
"$cond": [ { "$eq": [ "$productsInOrder.category", "ambient" ] }, 1, 0 ]
}
},
"frozen_count": {
"$sum": {
"$cond": [ { "$eq": [ "$productsInOrder.category", "frozen" ] }, 1, 0 ]
}
},
"other_category_count": {
"$sum": {
"$cond": [ { "$eq": [ "$productsInOrder.category", "other_category" ] }, 1, 0 ]
}
}
}
}
];
var result = coll.aggregate(pipeline);
Running the same pipeline in mongo shell using the sample data will yield:
{
"result" : [
{
"_id" : null,
"ambient_count" : 1,
"frozen_count" : 7,
"other_category_count" : 0
}
],
"ok" : 1
}
You can access the native mongo collection and publish the aggregation results to the orders collection on the client side:
Meteor.publish('categoryCounts', function() {
var self = this,
db = MongoInternals.defaultRemoteCollectionDriver().mongo.db;
orders = db.collection("orders").aggregate(pipeline, // Need to wrap the callback so it gets called in a Fiber.
Meteor.bindEnvironment(
function(err, result) {
// Add each of the results to the subscription.
_.each(result, function(e) {
self.added("orders", e._id, e);
});
self.ready();
},
function(error) {
Meteor._debug( "Error doing aggregation: " + error);
}
)
);
});
If you don't want to do this within Meteor, you will need to use mongo aggregation. Minimongo doesn't include aggregation though, so you will need this package to accomplish it:
https://docs.mongodb.org/manual/core/aggregation-introduction/
I only tested this in mongo itself, so you will have to adapt it to the way that the aggregation package does it:
db.orders.aggregate([
{
$unwind: "$productsInOrder"
},
{
$match: {
"productsInOrder.category": "frozen"
}
},
{
$group: {
_id: null,
count: {
$sum: 1
}
}
}
]);
The first part is unwinding the collection. It will basically make an "order" entry for every instance of $productsInOrder. Once you have the array flattened out, we match on the category you care about; in this case, the "frozen" category. Next we group it up so we can count the number of documents returned. $group is simply constructing the final object that will be output from the query. You can modify this to be whatever you want, or you could group by productsInOrder.category and not even $match on "frozen".

MongoDB: Convert array to object

How I can convert an array to an object in MongoDB?
For example, I want to convert this document:
{
"_id" : NumberLong(279),
"userAddressList" : [
{
"street" : "Street",
"house" : "House",
"building" : "Building",
"flat" : NumberLong(0),
"entrance" : NumberLong(0),
"floor" : NumberLong(0),
"intercom" : "Intercome"
}
],
}
to this:
{
"_id" : NumberLong(279),
"userAddressList" :
{
"street" : "Street",
"house" : "House",
"building" : "Building",
"flat" : NumberLong(0),
"entrance" : NumberLong(0),
"floor" : NumberLong(0),
"intercom" : "Intercome"
},
}
So I need to convert ""userAddressList" : [{..}]" to the ""userAddressList" : {..}".
For MongoDB 4.2 and newer
You could try the following query which uses the aggregation pipeline in the update:
db.collection.updateMany(
{},
[
{ '$addFields': {
'userAddressList': {
'$arrayElemAt': ['$userAddressList', 0]
}
} }
]
)
For older MongoDB versions:
db.collection.find().forEach(function(doc){
userAddressList = doc.userAddressList[0];
doc.userAddressList = userAddressList;
db.collection.save(doc);
})
or use the aggregation framework where you run the following pipeline
db.collection.aggregate([
{ "$addFields": {
"userAddressList": {
"$arrayElemAt": ["$userAddressList", 0]
}
} },
{ "$out": "collection" }
])
Note that this does not update your collection but replaces the existing one and does not change any indexes that existed on the previous collection. If the aggregation fails, the $out operation makes no changes to the pre-existing collection.

Categories