Group by Timestamp with aggregation framework in MongoDB - javascript

I have the below array of object from my MongoDB query to get some statistics based on time:
[
{
_id: { year: 2014, dayOfYear: 128, hour: 9 },
count: 2,
avg: 0.12455,
min: 0.1245,
max: 0.1246,
gv: 7.98954654895666,
bv: 0.9950000000000001
},
{
_id: { year: 2014, dayOfYear: 134, hour: 14 },
count: 8,
avg: 0.12217854,
min: 0.1212,
max: 0.12345678,
gv: 25.869999999999997,
bv: 3.1652450614477337
},
{
_id: { year: 2014, dayOfYear:126, hour: 19 },
count: 3,
avg: 0.11234099999999998,
min: 0.112341,
max: 0.112341,
gv: 29.849999999999998,
bv: 3.3533788500000004
}
]
I want to basically convert the _id object into a unix timestamp with moment.js I just cant to find the right underscore function to iterate through the objects, get the ID data and make an array out of it again.
In the process i could just do moment().year('2014').format('Z')
result wanted:
{
unixtime: 1400599394,
count: 3,
avg: 0.11234099999999998,
min: 0.112341,
max: 0.112341,
gv: 29.849999999999998,
bv: 3.3533788500000004
}

I know that you did ask for how to do this with promises, but I'm just taking a stab here based on the naming of the fields under _id that this is actually output from the aggregation framework and therefore there is another way to do this.
The date operators that you appear to have used are great for their purposes, but as you seem to want a unix timestamp as the output then the logical thing to do is keep that field in unix timestamp format, just with the grouping boundaries you need.
This seems to be grouping "per hour" within each day, so let's apply the date math in order to alter that timestamp value. So instead of using the date operators to break up the date in your $group pipeline key, do this instead:
{ "$group": {
"_id": {
"$subtract": [
{ "$subtract": [ "$created", new Date("1970-01-01") ] },
{ "$mod": [
{ "$subtract": [ "$created", new Date("1970-01-01") ] },
1000*60*60
]}
]
}
}}
Of course include all the other things you have in your aggregation pipeline ( which are not supplied in this question ), but the result is the actual epoch timestamp value being truncated to the hour of the day from your original date field which is named "created".
Just to break down the parts of this consider this date object:
var date = new Date()
date.valueOf()
1400639001169
So there you see the epoch timestamp representation extracted from the date object. In terms of the math, retrieving that value via the .valueOf() method is exactly the same as:
date - new Date("1970-01-01")
1400639001169
Which provides you with the seconds ( or milliseconds in this case ) elapsed since "1970-01-01", which is exactly what epoch time is. To further this we can get the milliseconds elapsed in 1 hour as a portion of this date by obtaining the modulus:
date % ( 1000*60*60 )
1401169
So with 1000 milliseconds in a second, 60 seconds in a minute and 60 minutes in an hour we obtain this result. All that is left now is to subtract that value from the original date value:
date - date % ( 1000*60*60 )
1400637600000
And that value is now the boundary of the hour in the day and suitable for grouping as well as being the desired result you want in your output. Just for reference, here are the string formatted values of the initial and converted timestamp:
Initial: Wed, 21 May 2014 02:23:21 GMT
Converted: Wed, 21 May 2014 02:00:00 GMT
So basically you are doing the same thing for grouping that you are using the date operators for, but actually getting the output format you want without requiring additional conversion.
Just a cool trick that if I'm right about how you got here should be useful to you and if not, then at least should be useful to others who might have arrived here in exactly that way.

Related

Can't convert from BSON type string to Date

I have data like the following example in my mongoDB (collection: data_extraction_test):
{
"_id" : ObjectId("4f16fc97d1e2d32371003e27"),
"date" : "14 Nov 2000 08:22:00 -0800"
}
{
"_id" : ObjectId("4f16fc97d1e2d32371003e28"),
"date" : "14 Nov 2000 07:37:00 -0800"
}
{
"_id" : ObjectId("4f16fc97d1e2d32371003e29"),
"date" : "14 Nov 2000 07:25:00 -0800"
}
When running the javascript Code (extract is given below) the following error appears: Can't convert from BSON type string to Date
let cursor = col.aggregate([
{
$project:{
_id: "$_id",
year: {$year: new Date("13 Nov 2000 01:41:00 -0800 (PST)")},
// month: new Date(new String("$date")),
month: { $month: "$date" },
}
},
{
$out: "dan"
}
]).toArray((err, items)=>{
assert.equal(null, err);
console.log("daniel",items);
resolve(true);
db.close();
});
How can i convert the string into ISODate?
The issue is that you are trying to convert RFC date format as string object. And the query is trying to convert assuming its a Date object.
I took the dates in your database, replaced them with ISO 8601 format.
"14 Nov 2000 08:22:00 -0800" => ISODate("2000-11-14T16:22:00.000Z")
"14 Nov 2000 07:37:00 -0800" => ISODate("2000-11-14T15:37:00Z")
"14 Nov 2000 07:25:00 -0800" => ISODate("2000-11-14T15:25:00Z")
After which the aggregation query worked.
Please note that the according to the doc.
The dates are stored as a 64 bit integer representing the number of milliseconds since the Unix epoch (Jan 1, 1970).
It might be better to store the dates as Date object rather than string to begin with. Is there a reason that you are storing them as strings?
Update
As from the link suggested by the user Neil Lunn.
You can use following script which will convert the property to ISO date.
(Please make backup of your db. Incase something doesn't go right)
//change test to your collection name
db.test.find({}).forEach(function (doc) {
doc.date = new Date(doc.date);
db.test.save(doc);
});

Find the last item in array in mongo document and check if its field is a certain value

I got this document:
{
_id: "ZApHZeqw98uhwqaey",
borrowerId: "DmGQyqenbNt4eBMia",
isSeenByOther: 1,
lenderId: "JsJyvseqiiazGxRuq",
messages: [{
date: Sun Oct 25 2015 19:40:25 GMT+0100 (CET),
from: "JsJyvseqiiazGxRuq",
text: "Hi there"
},{
date: Sun Oct 25 2015 19:40:35 GMT+0100 (CET),
from: "DmGQyqenbNt4eBMia",
text: "Hey!"
}]
}
What I'm trying to do is to just get a boolean value stating whether or not the value of the field from of the last object item in the array: messages is the current user.
I have tried a lot of different mongodb projections such as $slice and $position which essentially inserts in the beginning of the array but not ideal for my case.
You can use underscore's _.last() function as follows:
var doc = MyCollection.findOne({ _id: "ZApHZeqw98uhwqaey" });
var lastElement = _.last(doc.messages);
if ( lastElement.from === Meteor.userId() ){
... do your thing
}

Get all objects for a week number

I need to get a list of week ranges for all records in my MongoDB. When I click on a week range, it will display only the records for that week range. Clicking on the week range sends the ID of the week (lets say 42, ie the 42nd week out of year 2015), it should get those results.
Question: How can I query for a set of records given a week number and year? This should work, right?
SCHEMA:
var orderSchema = mongoose.Schema({
date: Date, //ISO date
request: {
headers : {
...
First: Get all week IDs for all Objects:
var query = Order.aggregate(
[
{
$project:
{
week:
{
$week: '$date'
}
}
},
{
$group:
{
_id: null,
distinctDate:
{
$addToSet:
{
week: '$week'
}
}
}
}
]
);
Result:
distinctDate: Array[35]
0: Object
week: 40
1: Object
week: 37
...
Convert to week ranges using MomentJS and display:
data.forEach(function(v, k) {
$scope.weekRanges.push(getWeekRange(v.week));
});
function getWeekRange(weekNum) {
var monday = moment().day("Monday").isoWeek(weekNum).format('MM-DD-YYYY');
var sunday = moment().day("Sunday").isoWeek(weekNum).format('MM-DD-YYYY');
...
Output:
Week
10-12-2015 to 10-18-2015 //week ID 42
10-05-2015 to 10-11-2015 //week ID 41
09-28-2015 to 10-04-2015 ...
...
Second: Click on week range and get Objects Per Week ID:
var year = 2015;
var weekID = weekParamID; //42
if (!Order) {
Order = mongoose.model('Order', orderSchema());
}
var query = Order.aggregate(
{
$project:
{
cust_ID : '$request.headers.custID',
cost : '$response.body.pricing.cost',
year :
{
$year: '$date'
},
month :
{
$month: '$date'
},
week:
{
$week: '$date'
},
day:
{
$dayOfMonth: '$date'
}
}
},
{
$match:
{
year : year, //2015
week : weekID //42
}
}
);
And if I click on Week Range 10-12-2015 to 10-18-2015 (week ID 42), I get results with dates outside of the range (10-19-2015):
10-19-2015 Order info
10-18-2015 Order info
10-19-2015 Order info
Using MongoDB command line:
db.mycollection.aggregate({ $project: { week: { $week: '$date' }, day: { $dayOfMonth: '$date' } } }, { $match: { week: 42 } }
Results:
{ "_id" : "1bd482f6759b", "week" : 42, "day" : 19 } //shouldn't exceed week range
{ "_id" : "b3d38759", "week" : 42, "day" : 19 }
EDIT: Update
So there is a discrepancy with MongoDB ISO weeks (starts on Sunday) and Moment JS ISO (starts on Monday).
This SO post suggests subtracting the dates from the query so the Mongo date starts on Monday:
{
$project:
{
week: { $week: [ "$datetime" ] },
dayOfWeek:{$dayOfWeek:["$datetime"]}}
},
{
$project:
{
week:{$cond:[{$eq:["$dayOfWeek",1]},{$subtract:["$week",1]},'$week']}
}
}
I implemented this with my query, but now it's not returning two fields that I need:
cust_ID : '$request.headers.custID',
cost : '$response.body.pricing.cost'
Query:
db.mycollection.aggregate(
{
$project:
{
cust_ID : '$request.headers.custID',
cost : '$response.body.pricing.cost',
week:
{
$week: ['$date']
},
dayOfWeek:
{
$dayOfWeek: ['$date']
}
}
},
{
$project:
{
week: {
$cond: [
{
$eq: [
"$dayOfWeek", 1
]
},
{
$subtract: [
"$week", 1
]
}, '$week'
]
}
}
},
{
$match:
{
week : 42
}
}
);
Results:
{ "_id" : "387e2", "week" : 42 }
{ "_id" : "ef269f6341", "week" : 42 }
{ "_id" : "17482f6759b", "week" : 42 }
{ "_id" : "7123d38759", "week" : 42 }
{ "_id" : "ff89b1fb", "week" : 42 }
It's not returning the fieldsets I specified in $project
The MongoDB $week operator considers weeks to begin on Sunday, see docs:
Weeks begin on Sundays, and week 1 begins with the first Sunday of the
year... This behavior is the same as the “%U” operator to the strftime
standard library function.
Moment.JS's isoWeekday() uses the ISO week which considers weeks to begin on Monday. It also differs in that it considers the week 1 to be the first week with a Thursday in it.
This discrepancy could explain the behaviour you are seeing.
E.g. if I save this doc in MongoDB, which is a Monday:
db.test.save({ "date" : new ISODate("2015-10-19T10:10:10Z") })
then run your aggregation query above, I get week 42.
But then if I run the following:
console.log(moment().day("Monday").isoWeek(42))
I get the below date which is not the one I originally saved in MongoDB, even though it is Monday of the week MongoDB reported.
Mon Oct 12 2015
How to fix it I guess depends on which definition of week you need.
If you are happy with the MongoDB $week definition, it's probably easy to find/write an alternative implementation to convert the week number to the corresponding date. Here is one library that adds strftime support to Moment.js:
https://github.com/benjaminoakes/moment-strftime
If you want to use the ISO format, it's more complicated. As per your edit above you'll need to account for the week start difference. But you'll also need to account for the week number at start of year difference. This difference means that the strftime week number can have a week 0 while ISO always starts on week 1. For 2015 it looks like you need to add 1 week on to the strftime week to get the ISO week, as well as accounting for the week start day, but that won't be reliable in general.
Starting from MongoDB version 3.4 you can use the $isoWeek aggregation operator.
Returns the week number in ISO 8601 format, ranging from 1 to 53. Week numbers start at 1 with the week (Monday through Sunday) that contains the year’s first Thursday.
You can find more infos on this in the MongoDB docs.

Highstock with datetime and multiple columns

I've been using HighCharts a lot in the past, but I don't remember how to render this kind of chart, with HighStocks:
I have a JSON like this:
[
{
timestamp: 'Sun Aug 16 2015 10:00:00 GMT+1000 (AEST)',
run: 2,
rest: 3
},
{
timestamp: 'Sun Aug 16 2015 10:01:00 GMT+1000 (AEST)',
run: 4,
rest: 1
},
{
timestamp: 'Sun Aug 16 2015 10:02:00 GMT+1000 (AEST)',
run: 2,
rest: 1
},
]
I would like to have a chart with two columns (run and rest) per timestamp. With HighStocks, so I can define my own scale with the mouse.
How can I write this in the chart configuration ?
To use Highstocks, you need to pass your date in milliseconds. So you can use (supposing we are looping through your JSON array):
var d = new Date(json[i].timestamp);
Then your series will have the format:
[d.getTime(), json[i].rest]
You can see a working JSFiddle here, which takes your json array as data.

Plotting an empty interval for elasticsearch date histogram

I'm using an elasticsearch date histogram to group responses by count over time. The date histogram facet works great for this but if an interval doesn't have any responses that fall within in it it doesn't show up in the json. I figured the best way to combat this is to use javascript to fill in the gaps in a charting library. (ideally in highcharts but d3 or something else is possible). Months seem pretty easy to do but it get more complicated when I need to do it by week and day as well. Basically my problem is:
{ date: April: 5, count: 5 }, { date: June, count: 10 }
needs to be more like
{ date: April: 5, count: 5 }, {date: May, count: null }, { date: June, count: 10 }
min_doc_count=0 only creates intervals in between nonempty buckets. If you want to plot empty intervals outside your buckets (a few months ahead or behind of the start of your data), then add extended_bounds (docs).
In elasticsearch_dsl, to allow empty buckets out to two years ago, this looks like
A(
"date_histogram",
field="publishedAt",
calendar_interval="month",
format="MMM yyyy",
min_doc_count=0,
extended_bounds={"min": f"{date:%b %Y}||-2y"},
),
I had the same issue for a while after searching and reading the documentation I found out extended_bounds will fix my problem:
{
"aggs": {
"total": {
"date_histogram": {
"extended_bounds": {
"max": "2022-11-01",
"min": "2015-09-04"
},
"field": "eventDate",
"calendar_interval": "1d",
"min_doc_count": 0
}
}
}
}

Categories