I need to get a list of week ranges for all records in my MongoDB. When I click on a week range, it will display only the records for that week range. Clicking on the week range sends the ID of the week (lets say 42, ie the 42nd week out of year 2015), it should get those results.
Question: How can I query for a set of records given a week number and year? This should work, right?
SCHEMA:
var orderSchema = mongoose.Schema({
date: Date, //ISO date
request: {
headers : {
...
First: Get all week IDs for all Objects:
var query = Order.aggregate(
[
{
$project:
{
week:
{
$week: '$date'
}
}
},
{
$group:
{
_id: null,
distinctDate:
{
$addToSet:
{
week: '$week'
}
}
}
}
]
);
Result:
distinctDate: Array[35]
0: Object
week: 40
1: Object
week: 37
...
Convert to week ranges using MomentJS and display:
data.forEach(function(v, k) {
$scope.weekRanges.push(getWeekRange(v.week));
});
function getWeekRange(weekNum) {
var monday = moment().day("Monday").isoWeek(weekNum).format('MM-DD-YYYY');
var sunday = moment().day("Sunday").isoWeek(weekNum).format('MM-DD-YYYY');
...
Output:
Week
10-12-2015 to 10-18-2015 //week ID 42
10-05-2015 to 10-11-2015 //week ID 41
09-28-2015 to 10-04-2015 ...
...
Second: Click on week range and get Objects Per Week ID:
var year = 2015;
var weekID = weekParamID; //42
if (!Order) {
Order = mongoose.model('Order', orderSchema());
}
var query = Order.aggregate(
{
$project:
{
cust_ID : '$request.headers.custID',
cost : '$response.body.pricing.cost',
year :
{
$year: '$date'
},
month :
{
$month: '$date'
},
week:
{
$week: '$date'
},
day:
{
$dayOfMonth: '$date'
}
}
},
{
$match:
{
year : year, //2015
week : weekID //42
}
}
);
And if I click on Week Range 10-12-2015 to 10-18-2015 (week ID 42), I get results with dates outside of the range (10-19-2015):
10-19-2015 Order info
10-18-2015 Order info
10-19-2015 Order info
Using MongoDB command line:
db.mycollection.aggregate({ $project: { week: { $week: '$date' }, day: { $dayOfMonth: '$date' } } }, { $match: { week: 42 } }
Results:
{ "_id" : "1bd482f6759b", "week" : 42, "day" : 19 } //shouldn't exceed week range
{ "_id" : "b3d38759", "week" : 42, "day" : 19 }
EDIT: Update
So there is a discrepancy with MongoDB ISO weeks (starts on Sunday) and Moment JS ISO (starts on Monday).
This SO post suggests subtracting the dates from the query so the Mongo date starts on Monday:
{
$project:
{
week: { $week: [ "$datetime" ] },
dayOfWeek:{$dayOfWeek:["$datetime"]}}
},
{
$project:
{
week:{$cond:[{$eq:["$dayOfWeek",1]},{$subtract:["$week",1]},'$week']}
}
}
I implemented this with my query, but now it's not returning two fields that I need:
cust_ID : '$request.headers.custID',
cost : '$response.body.pricing.cost'
Query:
db.mycollection.aggregate(
{
$project:
{
cust_ID : '$request.headers.custID',
cost : '$response.body.pricing.cost',
week:
{
$week: ['$date']
},
dayOfWeek:
{
$dayOfWeek: ['$date']
}
}
},
{
$project:
{
week: {
$cond: [
{
$eq: [
"$dayOfWeek", 1
]
},
{
$subtract: [
"$week", 1
]
}, '$week'
]
}
}
},
{
$match:
{
week : 42
}
}
);
Results:
{ "_id" : "387e2", "week" : 42 }
{ "_id" : "ef269f6341", "week" : 42 }
{ "_id" : "17482f6759b", "week" : 42 }
{ "_id" : "7123d38759", "week" : 42 }
{ "_id" : "ff89b1fb", "week" : 42 }
It's not returning the fieldsets I specified in $project
The MongoDB $week operator considers weeks to begin on Sunday, see docs:
Weeks begin on Sundays, and week 1 begins with the first Sunday of the
year... This behavior is the same as the “%U” operator to the strftime
standard library function.
Moment.JS's isoWeekday() uses the ISO week which considers weeks to begin on Monday. It also differs in that it considers the week 1 to be the first week with a Thursday in it.
This discrepancy could explain the behaviour you are seeing.
E.g. if I save this doc in MongoDB, which is a Monday:
db.test.save({ "date" : new ISODate("2015-10-19T10:10:10Z") })
then run your aggregation query above, I get week 42.
But then if I run the following:
console.log(moment().day("Monday").isoWeek(42))
I get the below date which is not the one I originally saved in MongoDB, even though it is Monday of the week MongoDB reported.
Mon Oct 12 2015
How to fix it I guess depends on which definition of week you need.
If you are happy with the MongoDB $week definition, it's probably easy to find/write an alternative implementation to convert the week number to the corresponding date. Here is one library that adds strftime support to Moment.js:
https://github.com/benjaminoakes/moment-strftime
If you want to use the ISO format, it's more complicated. As per your edit above you'll need to account for the week start difference. But you'll also need to account for the week number at start of year difference. This difference means that the strftime week number can have a week 0 while ISO always starts on week 1. For 2015 it looks like you need to add 1 week on to the strftime week to get the ISO week, as well as accounting for the week start day, but that won't be reliable in general.
Starting from MongoDB version 3.4 you can use the $isoWeek aggregation operator.
Returns the week number in ISO 8601 format, ranging from 1 to 53. Week numbers start at 1 with the week (Monday through Sunday) that contains the year’s first Thursday.
You can find more infos on this in the MongoDB docs.
Related
I'm working with a function that needs to return an array with data, the data of the Dates is always the same but the definition of the dates is not the same, here the dates saved in an array, every date is different but when I try to put this data in other array always return the last date from the array:
const dates=[new Date(startTimestamp), new Date(startTimestamp+ 6.048e+8), new Date(startTimestamp+2*+ 6.048e+8), new Date(startTimestamp+3*+ 6.048e+8)]
for (let index = 0; index < 4; index++) {
const week = weeksData[weekIndex + index];//I have an array of weeks to search for some data
//if that data is not in the array then i want to add a new week with a new Date
if (week !== undefined ) {
newWeeks.push(week);
}
else{
const mockWeek={}
mockWeek.timestamp=dates[index].valueOf();
mockWeek.week= dayjs(dates[index]).format('MMM DD');
console.log(dates[index])
console.log(mockWeek)
newWeeks.push(mockWeek)
}
}
here you can see that the loop has the rights values but, in the end the array doesnt put the desiered value in the final array
What I'm doing wrong ?
The weeks array is like this
Array(17)
0: Week
id: "test-project_1652590800000"
projectId: "test-project"
data:{...}
timestamp: 1652590800000
week: "May 15"
1: Week
id: "test-project_1641196800000"
projectId: "test-project"
data:{...}
timestamp: 1641196800000
week: "Jan 03"
2: Week
id: "test-project_1641801600000"
projectId: "test-project"
data:{...}
timestamp: 1641801600000
week: "Jan 10"
I search in this array by some Date, but if the date don´t exist, thats when I want to add the mockWeek, with the same data,but with a different week and timestamp, there's when the mock week always have the last date from the array, here i don't have the may 8 and may 1 for example, then i want to add a mock week like this
3: Week
id: "test-project_1651381200000"
projectId: "test-project"
data:{...}
timestamp: 1651381200000
week: "May 1"
4: Week
id: "test-project_1651986000000"
projectId: "test-project"
data:{...}
timestamp: 1651986000000
week: "May 8"
But instead I have in both week data the "May 8" and timestamp from "May 8"
This isn't an answer, it's a long comment seeking clarification.
Creating a runnable snippet from the posted code shows it doesn't behave as described. I've added semicolons and fixed the syntax errors. It seems to do what you want (i.e. add missing week objects to the newWeeks array).
You need to post code (preferably as a runnable snippet) that demonstrates the issue. Until then, responses can only be guesses.
let startTimestamp = new Date().setHours(0,0,0,0);
let dates=[new Date(startTimestamp), new Date(startTimestamp+ 6.048e+8), new Date(startTimestamp+2*+ 6.048e+8), new Date(startTimestamp+3*+ 6.048e+8)];
let weekIndex = 5; // Ensure no weeks are found
let newWeeks = [];
let weeksData = [{ // Minimal data in array from OP
0: 'Week',
timestamp: 1652590800000,
week: "May 15"
},{
1: 'Week',
timestamp: 1641196800000,
week: "Jan 03"
},{
2: 'Week',
timestamp: 1641801600000,
week: "Jan 10"
}];
// Loop from OP
for (let index = 0; index < 4; index++) {
let week = weeksData[weekIndex + index];
if (week !== undefined ) {
newWeeks.push(week);
} else {
let mockWeek = {};
mockWeek.timestamp = dates[index].valueOf();
mockWeek.week = dayjs(dates[index]).format('MMM DD');
console.log(dates[index]);
console.log(mockWeek);
newWeeks.push(mockWeek);
}
}
console.log(newWeeks);
<script src="https://cdnjs.cloudflare.com/ajax/libs/dayjs/1.11.2/dayjs.min.js"></script>
I want to show a list of posts from the database based on likes and date, think of the basic "trending" items page.
I want to use a formula like score = likes / daysSinceCreation and then get the first 10 posts based on this score.
How can I add that sort function with mongoDB/Mongoose?
Posts.find().sort(???).limit(10).then(posts => console.log(posts));
Currently I can get top posts in last week (find if creation date larger than last week and order by score), but how can I implement a more complex sorting function without getting all the items from the DB?
eg:
Today is Friday
ID CREATION_DAY LIKES
1 Monday 4 // score is 5/5 = 0
2 Tuesday 10 // score is 10/4 = 2
3 Wednesday 3 // score is 3/3 = 1
4 Thursday 20 // score is 20/2 = 10
5 Friday 5 // score is 5/1 = 5
Sorted list of IDs is: [4 (Th), 5 (Fr), 2 (Tu), 3 (We), 1(Mo)]
This will create a new document in a "trendingposts" table:
const fiveDaysAgo = new Date(Date.now() - (5 * 24 * 60 * 60 * 1000));
const oid = new ObjectId();
const now = new Date();
Posts.aggregate([
{
$match: {
createdAt: {
$gte: fiveDaysAgo
},
score: {
$gt: 0
}
}
},
{
$project: {
_id: true,
createdAt: true,
updatedAt: true,
title: true,
description: true,
score: true,
trendScore: {
$divide: [ "$score", {$subtract: [new Date(), "$createdAt"]} ]
}
}
},
{
$sort: {
trendScore: -1
}
},
{
$limit: 10
},
{
$group: {
_id: { $min: oid },
evaluatedAt: { $min: now },
posts: { $push: "$$ROOT"}
}
},
{
$out: "trendingposts"
}
])
.then(...)
A few things to note:
If using Mongo 3.4+ the $project stage can also be written as:
{
$addFields: {
trendScore: {
$divide: [ "$score", {$subtract: [new Date(), "$createdAt"]} ]
}
}
},
{ $min: now } is just a hack to grab the minimum value of now on each document, even though it's the same value for all of them.
"$$ROOT" is the entire current document. This means your end result will be a single object with the form:
{
"_id" : ObjectId("5a0a2fe912a325eb331f2759"),
"evaluatedAt" : ISODate("2017-11-13T23:51:56.051Z"),
"posts" : [/*10 `post` documents, sorted by trendScore */]
}
You can then query with:
TrendingPosts.findOne({})
.sort({_id: -1})
.then(trendingPost => console.log(trendingPost));
If your description/title are changing frequently, instead of $pushing the entire document in, you could just push the ids and use them for an $in query on your posts in order to guarantee the latest data.
I have data like the following example in my mongoDB (collection: data_extraction_test):
{
"_id" : ObjectId("4f16fc97d1e2d32371003e27"),
"date" : "14 Nov 2000 08:22:00 -0800"
}
{
"_id" : ObjectId("4f16fc97d1e2d32371003e28"),
"date" : "14 Nov 2000 07:37:00 -0800"
}
{
"_id" : ObjectId("4f16fc97d1e2d32371003e29"),
"date" : "14 Nov 2000 07:25:00 -0800"
}
When running the javascript Code (extract is given below) the following error appears: Can't convert from BSON type string to Date
let cursor = col.aggregate([
{
$project:{
_id: "$_id",
year: {$year: new Date("13 Nov 2000 01:41:00 -0800 (PST)")},
// month: new Date(new String("$date")),
month: { $month: "$date" },
}
},
{
$out: "dan"
}
]).toArray((err, items)=>{
assert.equal(null, err);
console.log("daniel",items);
resolve(true);
db.close();
});
How can i convert the string into ISODate?
The issue is that you are trying to convert RFC date format as string object. And the query is trying to convert assuming its a Date object.
I took the dates in your database, replaced them with ISO 8601 format.
"14 Nov 2000 08:22:00 -0800" => ISODate("2000-11-14T16:22:00.000Z")
"14 Nov 2000 07:37:00 -0800" => ISODate("2000-11-14T15:37:00Z")
"14 Nov 2000 07:25:00 -0800" => ISODate("2000-11-14T15:25:00Z")
After which the aggregation query worked.
Please note that the according to the doc.
The dates are stored as a 64 bit integer representing the number of milliseconds since the Unix epoch (Jan 1, 1970).
It might be better to store the dates as Date object rather than string to begin with. Is there a reason that you are storing them as strings?
Update
As from the link suggested by the user Neil Lunn.
You can use following script which will convert the property to ISO date.
(Please make backup of your db. Incase something doesn't go right)
//change test to your collection name
db.test.find({}).forEach(function (doc) {
doc.date = new Date(doc.date);
db.test.save(doc);
});
I am new to MongoDB and I am stuck on the String to Date conversion. In the db the date item is stored in String type as "date":"2015-06-16T17:50:30.081Z"
I want to group the docs by date and calculate the sum of each day so I have to extract year, month and day from the date string and wipe off the hrs, mins and seconds. I have tried multiple way but they either return a date type of 1970-01-01 or the current date.
Moreover, I want to convert the following mongo query into python code, which get me the same problem, I am not able to call a javascript function in python, and the datetime can not parse the mongo syntax $date either.
I have tried:
new Date("2015-06-16T17:50:30.081Z")
new Date(Date.parse("2015-06-16T17:50:30.081Z"))
etc...
I am perfectly find if the string is given in Javascript or in Python, I know more than one way to parse it. However I have no idea about how to do it in MongoDB query.
db.collection.aggregate([
{
//somthing
},
{
'$group':{
'_id':{
'month': (new Date(Date.parse('$approTime'))).getMonth(),
'day': (new Date(Date.parse('$approTime'))).getDate(),
'year': (new Date(Date.parse('$approTime'))).getFullYear(),
'countries':'$countries'
},
'count': {'$sum':1}
}
}
])
If you can be assured of the format of the input date string AND you are just trying to get a count of unique YYYYMMDD, then just $project the substring and group on it:
var data = [
{ "name": "buzz", "d1": "2015-06-16T17:50:30.081Z"},
{ "name": "matt", "d1": "2018-06-16T17:50:30.081Z"},
{ "name": "bob", "d1": "2018-06-16T17:50:30.081Z"},
{ "name": "corn", "d1": "2019-06-16T17:50:30.081Z"},
];
db.foo.drop();
db.foo.insert(data);
db.foo.aggregate([
{ "$project": {
"xd": { "$substr": [ "$d1", 0, 10 ]}
}
},
{ "$group": {
"_id": "$xd",
"n": {$sum: 1}
}
}
]);
{ "_id" : "2019-06-16", "n" : 1 }
{ "_id" : "2018-06-16", "n" : 2 }
{ "_id" : "2015-06-16", "n" : 1 }
Starting in Mongo 4.0, you can use "$toDate" to convert a string to a date:
// { name: "buzz", d1: "2015-06-16T17:50:30.081Z" }
// { name: "matt", d1: "2018-06-16T17:50:30.081Z" }
// { name: "bob", d1: "2018-06-16T17:50:30.081Z" }
// { name: "corn", d1: "2019-06-16T17:50:30.081Z" }
db.collection.aggregate(
{ $group: {
_id: { $dateToString: { date: { $toDate: "$d1" }, format: "%Y-%m-%d" } },
n: { $sum: 1 }
}}
)
// { _id: "2015-06-16", n: 1 }
// { _id: "2018-06-16", n: 2 }
// { _id: "2019-06-16", n: 1 }
Within the group stage, this:
first converts strings (such as "2015-06-16T17:50:30.081Z") to date objects (ISODate("2015-06-16T17:50:30.081Z")) using the "$toDate" operator.
then converts the converted date (such as ISODate("2015-06-16T17:50:30.081Z")) back to string ("2015-06-16") but this time with this format "%Y-%m-%d", using the $dateToString operator.
I have the below array of object from my MongoDB query to get some statistics based on time:
[
{
_id: { year: 2014, dayOfYear: 128, hour: 9 },
count: 2,
avg: 0.12455,
min: 0.1245,
max: 0.1246,
gv: 7.98954654895666,
bv: 0.9950000000000001
},
{
_id: { year: 2014, dayOfYear: 134, hour: 14 },
count: 8,
avg: 0.12217854,
min: 0.1212,
max: 0.12345678,
gv: 25.869999999999997,
bv: 3.1652450614477337
},
{
_id: { year: 2014, dayOfYear:126, hour: 19 },
count: 3,
avg: 0.11234099999999998,
min: 0.112341,
max: 0.112341,
gv: 29.849999999999998,
bv: 3.3533788500000004
}
]
I want to basically convert the _id object into a unix timestamp with moment.js I just cant to find the right underscore function to iterate through the objects, get the ID data and make an array out of it again.
In the process i could just do moment().year('2014').format('Z')
result wanted:
{
unixtime: 1400599394,
count: 3,
avg: 0.11234099999999998,
min: 0.112341,
max: 0.112341,
gv: 29.849999999999998,
bv: 3.3533788500000004
}
I know that you did ask for how to do this with promises, but I'm just taking a stab here based on the naming of the fields under _id that this is actually output from the aggregation framework and therefore there is another way to do this.
The date operators that you appear to have used are great for their purposes, but as you seem to want a unix timestamp as the output then the logical thing to do is keep that field in unix timestamp format, just with the grouping boundaries you need.
This seems to be grouping "per hour" within each day, so let's apply the date math in order to alter that timestamp value. So instead of using the date operators to break up the date in your $group pipeline key, do this instead:
{ "$group": {
"_id": {
"$subtract": [
{ "$subtract": [ "$created", new Date("1970-01-01") ] },
{ "$mod": [
{ "$subtract": [ "$created", new Date("1970-01-01") ] },
1000*60*60
]}
]
}
}}
Of course include all the other things you have in your aggregation pipeline ( which are not supplied in this question ), but the result is the actual epoch timestamp value being truncated to the hour of the day from your original date field which is named "created".
Just to break down the parts of this consider this date object:
var date = new Date()
date.valueOf()
1400639001169
So there you see the epoch timestamp representation extracted from the date object. In terms of the math, retrieving that value via the .valueOf() method is exactly the same as:
date - new Date("1970-01-01")
1400639001169
Which provides you with the seconds ( or milliseconds in this case ) elapsed since "1970-01-01", which is exactly what epoch time is. To further this we can get the milliseconds elapsed in 1 hour as a portion of this date by obtaining the modulus:
date % ( 1000*60*60 )
1401169
So with 1000 milliseconds in a second, 60 seconds in a minute and 60 minutes in an hour we obtain this result. All that is left now is to subtract that value from the original date value:
date - date % ( 1000*60*60 )
1400637600000
And that value is now the boundary of the hour in the day and suitable for grouping as well as being the desired result you want in your output. Just for reference, here are the string formatted values of the initial and converted timestamp:
Initial: Wed, 21 May 2014 02:23:21 GMT
Converted: Wed, 21 May 2014 02:00:00 GMT
So basically you are doing the same thing for grouping that you are using the date operators for, but actually getting the output format you want without requiring additional conversion.
Just a cool trick that if I'm right about how you got here should be useful to you and if not, then at least should be useful to others who might have arrived here in exactly that way.