Inconsistent string to date conversion in Mongoose - javascript

Model:
const fooSchema = new mongoose.Schema({
fooCreationDate: {
type: Date
},
bar: [{
barCreationDate: {
type: Date
}
}]
});
const foo = mongoose.model(`foo`, fooSchema);
If we want to search for foo objects that were created between 2022-01-01 and 2022-01-02, we can use the following mongoose query:
foo.find({
fooCreationDate: {
$gte: "2022-01-01T00:00:00.000",
$lt: "2022-01-02T00:00:00.000"
}
});
Please note that I'm using strings instead of date objects. The reason is that the query is passed by the client through an AJAX call with dataType: "jsonp". Every date object that is passed like that to the backend is automatically converted to an ISO string. Despite that, the query works without any issues - the find function automatically parses dates represented as ISO strings.
We'd now like to extract every bar object that was created in the same time range, so we'll need to use an aggregation:
foo.aggregate([{
$unwind: `$bar`,
}, {
$match: {
"bar.barCreationDate": {
$gte: "2022-01-01T00:00:00.000",
$lt: "2022-01-02T00:00:00.000"
}
}
}]);
Unfortunately, nothing is found despite the fact that the database contains matching bar objects. This can be confirmed by passing Date objects instead of strings to the $match aggregation:
foo.aggregate([{
$unwind: `$bar`,
}, {
$match: {
"bar.barCreationDate": {
$gte: new Date("2022-01-01T00:00:00.000"),
$lt: new Date("2022-01-02T00:00:00.000")
}
}
}]);
This query returns some results, so the conclusion is that mongoose accepts ISO date strings in the find function, but can't handle them in the aggregate function.
Is there any known workaround? I could, for example, deep-scan every query object passed from the client and search for ISO date strings, then convert them to Date objects, but that's a bit dirty in my opinion. I'm using mongoose v5.6.4 and mongodb v4.2.2.

Related

How to get latest documents from FaunaDB, based on timestamp?

Currently I store some data in FaunaDB every week. This is done using a cronjob. In my code I'm trying to fetch the documents from only the last two weeks. I'd like to use the timestamp to do so.
One of the documents to fetch:
{
"ref": Ref(Collection("weeklyContributors"), "350395411XXXXXXXX"),
"ts": 1670421954340000,
"data": {
...allMyDataFields
}
}
My code
const now = Date.now() * 1000;
const twoWeeksAgo = (Date.now() - 12096e5) * 1000;
console.log(now); //returns 1670493608804000
console.log(twoWeeksAgo); // returns 1669284008804000
// the stored document has a timestamp of 1670421954340000, so this should be in between [now] and [twoWeeksAgo]
await client.query(
q.Paginate(
q.Range(
q.Match(q.Index("get_weekly_list_by_ts")),
twoWeeksAgo,
now
)
)
);
This is a screenshot of the index I created in Fauna
Above code should fetch all documents where the timestamp's between now and twoWeeksAgo but it returns an empty array (so no documents match the query). Above code doesn't generate any errors, it does return a statuscode 200, so syntax should be fine. Why can't I fetch the document I gave in this example?
UPDATE
Found the solution for the index. The index should filter on Values, not Terms. Enter TS and Ref returns the document. BUt now I don't know how to get the corresponding document.
This returns an error
await client.query(
q.Map(
q.Paginate(
q.Range(
q.Match(q.Index("get_weekly_list_by_ts")),
twoWeeksAgo,
now
)
),
q.Lambda((x) => q.Get(x))
)
);
Changed index screenshot here
Congratulations on figuring out most of the answer for yourself!
As you deduced, the terms definition in an index specifies the fields to search for, and the values definition specifies the field values to return for matching entries.
Since you added the document reference to the values definition, all that you need now is to fetch that document. To do that, you need to Map over the results.
The following example uses Shell syntax, and involves sample documents that I created with a createdAt field recording the creation timestamp (since ts is the last-modified timestamp):
> Map(
Paginate(
Range(
Match(Index("get_weekly_list_by_ts")),
TimeSubtract(Now(), 14, "days"),
Now()
)
),
Lambda(
["ts", "ref"],
Get(Var("ref"))
)
)
{
data: [
{
ref: Ref(Collection("weeklyContributors"), "350498857823502848"),
ts: 1670520608640000,
data: { createdAt: Time("2022-12-01T17:30:08.633Z"), name: 'Fourth' }
},
{
ref: Ref(Collection("weeklyContributors"), "350498864657072640"),
ts: 1670520615160000,
data: { createdAt: Time("2022-12-07T17:30:15.152Z"), name: 'Fifth' }
}
]
}
Since your index returns ts and ref, notice that the Lambda function accepts both parameters in an array. The Lambda parameters have to match the number returned by the index. Then the Lambda calls Get to fetch the document.
In case you're wondering, here's the index definition that I used for my example:
> Get(Index("get_weekly_list_by_ts"))
{
ref: Index("get_weekly_list_by_ts"),
ts: 1670520331720000,
active: true,
serialized: true,
name: 'get_weekly_list_by_ts',
source: Collection("weeklyContributors"),
values: [ { field: [ 'data', 'createdAt' ] }, { field: [ 'ref' ] } ],
partitions: 8
}
My index is misnamed: I used the same name from your original query to help you correlate what is being used.
Note: there is no need to mask the document ID in a document that you share. It is only valid for the database containing the document.

How to get current date/time using database timezone in sequelize

I need to get the current time, according to the database timezone (not my local timezone, and not default UTC timezone), using Sequelize. Is there a Sequelize method to do this?
My database is in Eastern time, and when I query the db directly SELECT CURRENT_TIMESTAMP; it returns the date/time in Eastern time (which is correct).
But when I query via Sequelize in Node const [[{time}]] = await db.sequelize.query('SELECT CURRENT_TIMESTAMP AS time'); it returns the date/time in UTC.
Two problems:
1 - I would prefer using a Sequelize method instead of a raw query.
2 - This still doesn't get me the result I want. The time needs to be Eastern.
This is my DB setup:
const sequelize = new Sequelize(dbUrl, {
dialectOptions: {
useUTC: false // for reading from database
},
timezone: '-04:00', // for writing to database
define: {
charset: 'utf8'
}
})
As mentioned above, when I query using the above queries, the date is always returned in UTC, which I did not expect, given I said useUTC: false. How do I get it in Eastern time (the database timezone)?
I'm not aware of a sequelize method like getCurrentDate().
The UTC conversion problem seems to bite everyone (myself included). Here are some details. Not sure if dialectOptions: {useUTC: false }, has any function at all - just adding the typeCast method solved the problem for me.
dialectOptions: {
typeCast: function (field, next) { // for reading from database
if (field.type === 'DATETIME') {
return field.string()
}
return next()
},
The result can be used for a new js Date object:
const sql = 'select current_timestamp';
my_app.my_DB.query(sql, {raw: true, type: Sequelize.QueryTypes.SELECT})
.then(data => {
console.log(data[0].current_timestamp);
let d1 = new Date(data[0].current_timestamp);
});
This works fine for me - but make sure to test thoroughly!
Add useUTC property in your dialectOptions like this
dialectOptions: {
encrypt: false ,
options: {
useUTC: false, // for reading from database
},
},

How to perform date comparisons against postgres with sequelize

I want to delete all records with dates before 20 minutes ago. Postgres (or Sequelize) is not satisfied with the bare javascript Date object I provide as the comparison value.
I'm using sequelize 4.37 on top of a postgres 9.6 database.
The column in question was declared with type: Sequelize.DATE, which research suggests is equivalent to TIMESTAMP WITH TIME ZONE: a full date and time with microsecond precision and a timezone signifier. (That is also what I see when I use the psql CLI tool to describe the table.)
So, I do this:
const Sequelize = require('sequelize')
const { SomeModel } = require('../models.js')
// calculate 20 minutes ago
async function deleteStuff() {
const deletionCutoff = new Date()
deletionCutoff.setMinutes( deletionCutoff.getMinutes() - 20 )
await SomeModel.destroy({
where: {
[ Sequelize.Op.lt ]: { dateColumn: deletionCutoff }
}
})
But I get this error:
Error: Invalid value { dateColumn: 2018-11-21T21:26:16.849Z }
The docs suggest I should be able to provide either a bare javascript Date, or an ISO8601 string, but both throw the same Invalid Value error. The only difference is that, if I pass a string, the error shows single quotes around the value:
// error when dateColumn: deletionCutoff.toISOString()
Error: Invalid value { dateColumn: '2018-11-21T21:26:16.849Z' }
Well, this is pretty embarrassing. I structured the where clause incorrectly.
// BAD CODE
await SomeModel.destroy({
where: {
[ Sequelize.Op.lt ]: {
dateColumn: deletionCutoff
}
}
})
// GOOD CODE
await SomeModel.destroy({
where: {
dateColumn: {
[ Sequelize.Op.lt ]: deletionCutoff
}
}
})
Maybe I should delete the question. Maybe not -- the error I got probably could be more helpful.

Average value of a whole collection on mongoDB

I need to get the average value of a whole mongo collection. To be more specific, I have two date fields, let's call them beginning and end.
First of all, I need to do something like end - beginning time to get the elapsed time. After that, I want to sum all the elapsed time and get the average time.
I need this data to plot a chart. I've seen that mongoDB has some built in functions like subtract, sum and average as well. I don't know how to use them, and I also need the data in minutes so I may use some Javascript to convert it, I don't know yet but this is not the problem.
Mongoose schema:
module.exports = mongoose.model('atendimento', {
id: String,
id_atendimento: { type: Number, default: 0 },
id_cliente: { type: Number, default: 0 },
id_user: mongoose.Schema.Types.ObjectId,
user_nome: String,
cliente_nome: String,
id_atendente: { type: Number, default: 0 },
atendente_nome: String,
atendente_imagem: String,
setor: Number,
descricao: String,
status: String,
date: { type: Date, default: Date.now },
inicio: { type: Date, default: Date.now },
fim: { type: Date, default: Date.now },
update: { type: Date, default: Date.now }
});
The begin is the variable called inicio and the end is the variable called fim.
At the moment I need help with these mongo functions or any other suggestions will be welcome.
Thanks in advance, hope I can get some help!
Sounds like you need to run an aggregation pipeline which aggregates the whole collection using a $group pipeline.
Within the $group, you need to calculate the timestamp difference with $subtract operator and divide the result by the number of milliseconds in a minute (60 * 1000) with $divide operator.
You will then apply the $avg operator to the above expression so that you will have your collection average.
For the $group pipeline, you can specify an _id value of null to calculate accumulated values for all the input documents as a whole.
Following example shows the above:
Atendimento.aggregate([
{ '$group': {
'_id': null,
'average_duration': {
'$avg': {
'$divide': [
{ '$subtract': ['$fim', '$inicio'] },
60*1000
]
}
}
} }
]).exec((err, results) => console.log(results))
You need to user aggregate function with $subtract function.Subtracts two dates to return the difference in milliseconds
db.sales.aggregate( [ { $project: { item: 1, dateDifference: { $subtract: [ new Date(), "$date" ] } } } ] )
this will give you data like
{ "_id" : 1, "item" : "abc", "dateDifference" : NumberLong("11713985194") }
{ "_id" : 2, "item" : "jkl", "dateDifference" : NumberLong("11710385194") }
You can change the new date to another date field you have in the database.
Then you need to use $substract and $group in aggregate function to get the desired result as you haven't posted exact schema so it's hard to write your query so you can try with following
https://docs.mongodb.com/manual/reference/method/db.collection.aggregate/
https://docs.mongodb.com/manual/reference/operator/aggregation/subtract/
https://docs.mongodb.com/manual/reference/operator/aggregation/group/

Mongoose find inside nested schema

In mongoose we are deeply searching inside a nested schema, without much success. Every time we run this function we always get an empty array returned.
function findAlarms(lastUpdate = new Date(0), record = Record) {
// For docs on find http://mongoosejs.com/docs/queries.html
return record
.find({
// Date due must be less than "now"
'documents.alarm.date_due': {
$lte: Date.now(),
},
// Must be greater than the last update and less than "now"
'documents.alarm.date_reminder.reminder': {
$gte: lastUpdate,
$lte: Date.now(),
},
})
.populate('documents')
.exec();
}
Our schemas, greatly summarized, look like this:
const RecordSchema = new mongoose.Schema({
documents: [
{
type: Schema.Types.ObjectId,
ref: 'Document',
},
],
});
And our documents schema, similarly summarized looks like this:
const DocumentSchema = new mongoose.Schema({
alarm: {
date_due: { type: Date },
date_reminder: [
{
reminder: { type: Date },
},
],
},
});
This search returns no matching elements, even though we know there are documents that match. If we modify our findAlarms method to use the documents schema:
function findAlarms(lastUpdate = new Date(0), document = Document) {
// For docs on find http://mongoosejs.com/docs/queries.html
return document
.find({
// Date due must be less than "now"
'alarm.date_due': {
$lte: Date.now(),
},
// Must be greater than the last update and less than "now"
'alarm.date_reminder.reminder': {
$gte: lastUpdate,
$lte: Date.now(),
},
})
.exec();
}
It will return all of our matching documents. However, having records is essential for our needs. Now, I could use a hack and then find records using the array of document._ids that return.
Nonetheless, I would love to know if there's an approach where we can find using the records directly, since adding that extra step feels really hacky, and this operation runs every 5 minutes so I'd love to be more efficient wherever posible.

Categories