I have a collection with this data registered
{
_id: 0000120210903, iid: 00001, date: 20210903 }, {
_id: 0000220210903, iid: 00002, date: 20210903 }, {
_id: 0000120210101, iid: 00001, date: 20210101 }
I want to delete all except the document with the most recent date for each iid.
My idea is to group by the date, select the _id of the register with the max(date) and then delete all except this array of _ids. But I can't figure out how to do it.
db.getCollection('testing_data').aggregate(
{ $sort:{ _id:1 }},
{ $group:{
_id:"$iid",
lastId:{ "$last":"$_id" },
}},
{ $project:{ _id: 0, lastId: 1 } }
)
But I don't know where to go from here. Any help is greatly appreciated.
[Solution]
To fix the problem I used an aggregation to recover the combination of the field iid (the identifier shared between documents) and the unique _id as an array.
Then for each element on the array it performs a deleteMany operation on the iid but letting out the most recent _id. In this case I sort by _id because it includes the date but could also sort by the field date.
Due to the high volume of data { allowDiskUse: true } had to be put in the aggregate.
var ids = db.getCollection('testing_data').aggregate([
{ $sort:{ _id:1 }},
{ $group:{
_id:"$iid",
lastId:{ "$last":"$_id" },
}},
{ $project:{ _id: 1, lastId: 1 } }
], { allowDiskUse: true } ).toArray();
ids.forEach(function(x){
db.getCollection('testing_data').deleteMany({ "iid": x._id, "_id": {$ne:x.lastId} })
});
Mine Idea is just stock all _ids at some array that you want to delete, and then use deleteMany with $or filter
db.getCollection("testing_data").find({}).toArray((err,data)=>{
let to_elim = [];
let filtering ={};
for(let el of data){
if(!filtering[el.iid]) filtering[el.iid] = el;
else {
if(filtering[el.iid].date>el.date) to_elim.push({_id:new ObjectID(el._id)})
}
}
db.getCollection("testing_data").deleteMany({$or:to_elim})
})
I hope that all is written rightly, cause wrote all that down on mobile
There is missing some checking if something more recent...
[Solution]
To fix the problem I used an aggregation to recover the combination of the field iid (the identifier shared between documents) and the unique _id as an array.
Then for each element on the array it performs a deleteMany operation on the iid but letting out the most recent _id. In this case I sort by _id because it includes the date but could also sort by the field date.
Due to the high volume of data { allowDiskUse: true } had to be put in the aggregate.
var ids = db.getCollection('testing_data').aggregate([
{ $sort:{ _id:1 }},
{ $group:{
_id:"$iid",
lastId:{ "$last":"$_id" },
}},
{ $project:{ _id: 1, lastId: 1 } }
], { allowDiskUse: true } ).toArray();
ids.forEach(function(x){
db.getCollection('testing_data').deleteMany({ "iid": x._id, "_id": {$ne:x.lastId} })
});
i have my model called "Conversations", and my model "Messages", right now i want to retrieve all conversations with the last Message attached (only 1 message per conversation), so i filtered the conversationids and i queried the messages, but i'm not able to get this messages (last messages) for each conversation, thanks in advance.
let conversations = await ConversationModel.find({});
const conversationIds = conversations.map(conversation => conversation._id)
// ConversationIds is basically ["conversation1", "conversation2", "conversation3"]
// Te problem is here, i want to attach the las message for each conversation, if i put limit(1)
// i will get 1 record for all query, but i want the last message record for each conversation.
MessageModel.find({ _id: { "$in" : conversationIds} }, ...);
From information gathered in comments; This is possible to achieve in a case where MessageModel documents contain a time-stamp to identify latest of them.
Idea: Filter messages based on conversationIds via $match, sort them by timestamp for the next stage where $group on conversation reference (lets say conversation_id) and pick latest of them by $first accumulator.
Aggregation Query: playground link
db.collection.aggregate([
{
$match: {
conversation_id: {
$in: conversationIds
}
}
},
{
$sort: {
timestamp: -1
}
},
{
$group: {
_id: "$conversation_id",
latest_doc: {
$first: "$$ROOT"
}
}
}
]);
I've been playing around storing tweets inside mongodb, each object looks like this:
{
"_id" : ObjectId("4c02c58de500fe1be1000005"),
"contributors" : null,
"text" : "Hello world",
"user" : {
"following" : null,
"followers_count" : 5,
"utc_offset" : null,
"location" : "",
"profile_text_color" : "000000",
"friends_count" : 11,
"profile_link_color" : "0000ff",
"verified" : false,
"protected" : false,
"url" : null,
"contributors_enabled" : false,
"created_at" : "Sun May 30 18:47:06 +0000 2010",
"geo_enabled" : false,
"profile_sidebar_border_color" : "87bc44",
"statuses_count" : 13,
"favourites_count" : 0,
"description" : "",
"notifications" : null,
"profile_background_tile" : false,
"lang" : "en",
"id" : 149978111,
"time_zone" : null,
"profile_sidebar_fill_color" : "e0ff92"
},
"geo" : null,
"coordinates" : null,
"in_reply_to_user_id" : 149183152,
"place" : null,
"created_at" : "Sun May 30 20:07:35 +0000 2010",
"source" : "web",
"in_reply_to_status_id" : {
"floatApprox" : 15061797850
},
"truncated" : false,
"favorited" : false,
"id" : {
"floatApprox" : 15061838001
}
How would I write a query which checks the created_at and finds all objects between 18:47 and 19:00? Do I need to update my documents so the dates are stored in a specific format?
Querying for a Date Range (Specific Month or Day) in the MongoDB Cookbook has a very good explanation on the matter, but below is something I tried out myself and it seems to work.
items.save({
name: "example",
created_at: ISODate("2010-04-30T00:00:00.000Z")
})
items.find({
created_at: {
$gte: ISODate("2010-04-29T00:00:00.000Z"),
$lt: ISODate("2010-05-01T00:00:00.000Z")
}
})
=> { "_id" : ObjectId("4c0791e2b9ec877893f3363b"), "name" : "example", "created_at" : "Sun May 30 2010 00:00:00 GMT+0300 (EEST)" }
Based on my experiments you will need to serialize your dates into a format that MongoDB supports, because the following gave undesired search results.
items.save({
name: "example",
created_at: "Sun May 30 18.49:00 +0000 2010"
})
items.find({
created_at: {
$gte:"Mon May 30 18:47:00 +0000 2015",
$lt: "Sun May 30 20:40:36 +0000 2010"
}
})
=> { "_id" : ObjectId("4c079123b9ec877893f33638"), "name" : "example", "created_at" : "Sun May 30 18.49:00 +0000 2010" }
In the second example no results were expected, but there was still one gotten. This is because a basic string comparison is done.
To clarify. What is important to know is that:
Yes, you have to pass a Javascript Date object.
Yes, it has to be ISODate friendly
Yes, from my experience getting this to work, you need to manipulate the date to ISO
Yes, working with dates is generally always a tedious process, and mongo is no exception
Here is a working snippet of code, where we do a little bit of date manipulation to ensure Mongo (here i am using mongoose module and want results for rows whose date attribute is less than (before) the date given as myDate param) can handle it correctly:
var inputDate = new Date(myDate.toISOString());
MyModel.find({
'date': { $lte: inputDate }
})
Python and pymongo
Finding objects between two dates in Python with pymongo in collection posts (based on the tutorial):
from_date = datetime.datetime(2010, 12, 31, 12, 30, 30, 125000)
to_date = datetime.datetime(2011, 12, 31, 12, 30, 30, 125000)
for post in posts.find({"date": {"$gte": from_date, "$lt": to_date}}):
print(post)
Where {"$gte": from_date, "$lt": to_date} specifies the range in terms of datetime.datetime types.
db.collection.find({"createdDate":{$gte:new ISODate("2017-04-14T23:59:59Z"),$lte:new ISODate("2017-04-15T23:59:59Z")}}).count();
Replace collection with name of collection you want to execute query
MongoDB actually stores the millis of a date as an int(64), as prescribed by http://bsonspec.org/#/specification
However, it can get pretty confusing when you retrieve dates as the client driver will instantiate a date object with its own local timezone. The JavaScript driver in the mongo console will certainly do this.
So, if you care about your timezones, then make sure you know what it's supposed to be when you get it back. This shouldn't matter so much for the queries, as it will still equate to the same int(64), regardless of what timezone your date object is in (I hope). But I'd definitely make queries with actual date objects (not strings) and let the driver do its thing.
Use this code to find the record between two dates using $gte and $lt:
db.CollectionName.find({"whenCreated": {
'$gte': ISODate("2018-03-06T13:10:40.294Z"),
'$lt': ISODate("2018-05-06T13:10:40.294Z")
}});
Using with Moment.js and Comparison Query Operators
var today = moment().startOf('day');
// "2018-12-05T00:00:00.00
var tomorrow = moment(today).endOf('day');
// ("2018-12-05T23:59:59.999
Example.find(
{
// find in today
created: { '$gte': today, '$lte': tomorrow }
// Or greater than 5 days
// created: { $lt: moment().add(-5, 'days') },
}), function (err, docs) { ... });
db.collection.find({$and:
[
{date_time:{$gt:ISODate("2020-06-01T00:00:00.000Z")}},
{date_time:{$lt:ISODate("2020-06-30T00:00:00.000Z")}}
]
})
##In case you are making the query directly from your application ##
db.collection.find({$and:
[
{date_time:{$gt:"2020-06-01T00:00:00.000Z"}},
{date_time:{$lt:"2020-06-30T00:00:00.000Z"}}
]
})
You can also check this out. If you are using this method, then use the parse function to get values from Mongo Database:
db.getCollection('user').find({
createdOn: {
$gt: ISODate("2020-01-01T00:00:00.000Z"),
$lt: ISODate("2020-03-01T00:00:00.000Z")
}
})
Save created_at date in ISO Date Format then use $gte and $lte.
db.connection.find({
created_at: {
$gte: ISODate("2010-05-30T18:47:00.000Z"),
$lte: ISODate("2010-05-30T19:00:00.000Z")
}
})
use $gte and $lte to find between date data's in mongodb
var tomorrowDate = moment(new Date()).add(1, 'days').format("YYYY-MM-DD");
db.collection.find({"plannedDeliveryDate":{ $gte: new Date(tomorrowDate +"T00:00:00.000Z"),$lte: new Date(tomorrowDate + "T23:59:59.999Z")}})
mongoose.model('ModelName').aggregate([
{
$match: {
userId: mongoose.Types.ObjectId(userId)
}
},
{
$project: {
dataList: {
$filter: {
input: "$dataList",
as: "item",
cond: {
$and: [
{
$gte: [ "$$item.dateTime", new Date(`2017-01-01T00:00:00.000Z`) ]
},
{
$lte: [ "$$item.dateTime", new Date(`2019-12-01T00:00:00.000Z`) ]
},
]
}
}
}
}
}
])
For those using Make (formerly Integromat) and MongoDB:
I was struggling to find the right way to query all records between two dates. In the end, all I had to do was to remove ISODate as suggested in some of the solutions here.
So the full code would be:
"created": {
"$gte": "2016-01-01T00:00:00.000Z",
"$lt": "2017-01-01T00:00:00.000Z"
}
This article helped me achieve my goal.
UPDATE
Another way to achieve the above code in Make (formerly Integromat) would be to use the parseDate function. So the code below will return the same result as the one above :
"created": {
"$gte": "{{parseDate("2016-01-01"; "YYYY-MM-DD")}}",
"$lt": "{{parseDate("2017-01-01"; "YYYY-MM-DD")}}"
}
⚠️ Be sure to wrap {{parseDate("2017-01-01"; "YYYY-MM-DD")}} between quotation marks.
Convert your dates to GMT timezone as you're stuffing them into Mongo. That way there's never a timezone issue. Then just do the math on the twitter/timezone field when you pull the data back out for presentation.
Why not convert the string to an integer of the form YYYYMMDDHHMMSS? Each increment of time would then create a larger integer, and you can filter on the integers instead of worrying about converting to ISO time.
Scala:
With joda DateTime and BSON syntax (reactivemongo):
val queryDateRangeForOneField = (start: DateTime, end: DateTime) =>
BSONDocument(
"created_at" -> BSONDocument(
"$gte" -> BSONDateTime(start.millisOfDay().withMinimumValue().getMillis),
"$lte" -> BSONDateTime(end.millisOfDay().withMaximumValue().getMillis)),
)
where millisOfDay().withMinimumValue() for "2021-09-08T06:42:51.697Z" will be "2021-09-08T00:00:00.000Z"
and
where millisOfDay(). withMaximumValue() for "2021-09-08T06:42:51.697Z" will be "2021-09-08T23:59:99.999Z"
i tried in this model as per my requirements i need to store a date when ever a object is created later i want to retrieve all the records (documents ) between two dates
in my html file
i was using the following format mm/dd/yyyy
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<script>
//jquery
$(document).ready(function(){
$("#select_date").click(function() {
$.ajax({
type: "post",
url: "xxx",
datatype: "html",
data: $("#period").serialize(),
success: function(data){
alert(data);
} ,//success
}); //event triggered
});//ajax
});//jquery
</script>
<title></title>
</head>
<body>
<form id="period" name='period'>
from <input id="selecteddate" name="selecteddate1" type="text"> to
<input id="select_date" type="button" value="selected">
</form>
</body>
</html>
in my py (python) file i converted it into "iso fomate"
in following way
date_str1 = request.POST["SelectedDate1"]
SelectedDate1 = datetime.datetime.strptime(date_str1, '%m/%d/%Y').isoformat()
and saved in my dbmongo collection with "SelectedDate" as field in my collection
to retrieve data or documents between to 2 dates i used following query
db.collection.find( "SelectedDate": {'$gte': SelectedDate1,'$lt': SelectedDate2}})
In mongoose we are deeply searching inside a nested schema, without much success. Every time we run this function we always get an empty array returned.
function findAlarms(lastUpdate = new Date(0), record = Record) {
// For docs on find http://mongoosejs.com/docs/queries.html
return record
.find({
// Date due must be less than "now"
'documents.alarm.date_due': {
$lte: Date.now(),
},
// Must be greater than the last update and less than "now"
'documents.alarm.date_reminder.reminder': {
$gte: lastUpdate,
$lte: Date.now(),
},
})
.populate('documents')
.exec();
}
Our schemas, greatly summarized, look like this:
const RecordSchema = new mongoose.Schema({
documents: [
{
type: Schema.Types.ObjectId,
ref: 'Document',
},
],
});
And our documents schema, similarly summarized looks like this:
const DocumentSchema = new mongoose.Schema({
alarm: {
date_due: { type: Date },
date_reminder: [
{
reminder: { type: Date },
},
],
},
});
This search returns no matching elements, even though we know there are documents that match. If we modify our findAlarms method to use the documents schema:
function findAlarms(lastUpdate = new Date(0), document = Document) {
// For docs on find http://mongoosejs.com/docs/queries.html
return document
.find({
// Date due must be less than "now"
'alarm.date_due': {
$lte: Date.now(),
},
// Must be greater than the last update and less than "now"
'alarm.date_reminder.reminder': {
$gte: lastUpdate,
$lte: Date.now(),
},
})
.exec();
}
It will return all of our matching documents. However, having records is essential for our needs. Now, I could use a hack and then find records using the array of document._ids that return.
Nonetheless, I would love to know if there's an approach where we can find using the records directly, since adding that extra step feels really hacky, and this operation runs every 5 minutes so I'd love to be more efficient wherever posible.
I have my object as follows
-Notifications
userId
key1
datakey1
date: 10-10-2016
time: 3:20 pm
status: open
key2
datakey2
date: 11-10-2016
time: 5:00 pm
status: close
I'm having ref path till Notifications/userId
How do I retrieve data where status='open'
Code from comments:
ref.orderByChild("status")
.equalTo('open')
.on("value", function(snapshot) { console.log(snapshot.key); });
Firebase queries can only contain one dynamic key.
In your case you have two dynamic keys under that key1/key2 and datakey1/datakey2. You'll either have to put one of those at a fixed location (e.g. key1/my/status, key2/my/status) or you will have to structure your data differently.
notificationsStatusPerUser
userId: {
key1_datakey1: {
status: "open",
path: "key1/datakey1"
}
key2_datakey2: {
status: "close",
path: "key2/datakey2"
}
}
}
Now you can find all open items for a specific user with:
var query = ref.child('notificationsStatusPerUser')
.child(uid)
.orderByChild(status)
.equalTo('open');
query.on('child_added', function(snapshot) {
ref.child('notifications')
.child(uid)
.child(snapshot.val().path)
.once('value', function(linkedSnapshot) {
console.log(linkedSnapshot.val());
});
};