We Keep Coding

How to get latest documents from FaunaDB, based on timestamp?

Currently I store some data in FaunaDB every week. This is done using a cronjob. In my code I'm trying to fetch the documents from only the last two weeks. I'd like to use the timestamp to do so.
One of the documents to fetch:
{
"ref": Ref(Collection("weeklyContributors"), "350395411XXXXXXXX"),
"ts": 1670421954340000,
"data": {
...allMyDataFields
}
}
My code
const now = Date.now() * 1000;
const twoWeeksAgo = (Date.now() - 12096e5) * 1000;
console.log(now); //returns 1670493608804000
console.log(twoWeeksAgo); // returns 1669284008804000
// the stored document has a timestamp of 1670421954340000, so this should be in between [now] and [twoWeeksAgo]
await client.query(
q.Paginate(
q.Range(
q.Match(q.Index("get_weekly_list_by_ts")),
twoWeeksAgo,
now
)
)
);
This is a screenshot of the index I created in Fauna
Above code should fetch all documents where the timestamp's between now and twoWeeksAgo but it returns an empty array (so no documents match the query). Above code doesn't generate any errors, it does return a statuscode 200, so syntax should be fine. Why can't I fetch the document I gave in this example?
UPDATE
Found the solution for the index. The index should filter on Values, not Terms. Enter TS and Ref returns the document. BUt now I don't know how to get the corresponding document.
This returns an error
await client.query(
q.Map(
q.Paginate(
q.Range(
q.Match(q.Index("get_weekly_list_by_ts")),
twoWeeksAgo,
now
)
),
q.Lambda((x) => q.Get(x))
)
);
Changed index screenshot here

Congratulations on figuring out most of the answer for yourself!
As you deduced, the terms definition in an index specifies the fields to search for, and the values definition specifies the field values to return for matching entries.
Since you added the document reference to the values definition, all that you need now is to fetch that document. To do that, you need to Map over the results.
The following example uses Shell syntax, and involves sample documents that I created with a createdAt field recording the creation timestamp (since ts is the last-modified timestamp):
> Map(
Paginate(
Range(
Match(Index("get_weekly_list_by_ts")),
TimeSubtract(Now(), 14, "days"),
Now()
)
),
Lambda(
["ts", "ref"],
Get(Var("ref"))
)
)
{
data: [
{
ref: Ref(Collection("weeklyContributors"), "350498857823502848"),
ts: 1670520608640000,
data: { createdAt: Time("2022-12-01T17:30:08.633Z"), name: 'Fourth' }
},
{
ref: Ref(Collection("weeklyContributors"), "350498864657072640"),
ts: 1670520615160000,
data: { createdAt: Time("2022-12-07T17:30:15.152Z"), name: 'Fifth' }
}
]
}
Since your index returns ts and ref, notice that the Lambda function accepts both parameters in an array. The Lambda parameters have to match the number returned by the index. Then the Lambda calls Get to fetch the document.
In case you're wondering, here's the index definition that I used for my example:
> Get(Index("get_weekly_list_by_ts"))
{
ref: Index("get_weekly_list_by_ts"),
ts: 1670520331720000,
active: true,
serialized: true,
name: 'get_weekly_list_by_ts',
source: Collection("weeklyContributors"),
values: [ { field: [ 'data', 'createdAt' ] }, { field: [ 'ref' ] } ],
partitions: 8
}
My index is misnamed: I used the same name from your original query to help you correlate what is being used.
Note: there is no need to mask the document ID in a document that you share. It is only valid for the database containing the document.

node-redis/search query elements by geospatial position and additional indexes

I want to be able to query elements in a redis cache based on 3 different indexes. Those indexes would be:
A MAC address stored as a String.
A number.
A latitude and longitude(to be able to query spatially).
I have seen that Redis has support for multi indexing using redis search and native geospatial api.
so using nodejs and node-redis I have written the following index:
client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
Now, i would like to insert records on the database that include those 3 parameters plus an additional String that stores some payload. I have tried using
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: {latitude:0, longitude:0},
mid: '00:00:5e:00:53:af',
message: 'payload'
})
But I get the following error:
throw new TypeError('Invalid argument type');
^
TypeError: Invalid argument type
So, i can't add the latitude and longitude that way, I also tried
using the module ngeohash and computing an 11 character wide geohash like so:
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
And it does not give any error but when using redis search querys It does not find points near it.
Is it even possible what I am trying to do? If so, how would you input the data to the redis database?
Here is a minimal reproducible example (Im using "ngeohash": "^0.6.3" and "redis": "^4.5.0"):
const { createClient, SchemaFieldTypes } = require('redis')
const geohash = require('ngeohash')
const client = createClient()
async function start(client) {
await client.connect()
try {
// We only want to sort by these 3 values
await client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
} catch (e) {
if (e.message === 'Index already exists') {
console.log('Skipping index creation as it already exists.')
} else {
console.error(e)
process.exit(1)
}
}
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
await client.hSet('CITS:19123123:0.001:0.001:ff:ff:ff:ff:ff:ff', {
timestamp: 19123123,
position: geohash.encode(0.001, 0.001, 11),
mid: 'ff:ff:ff:ff:ff:ff',
message: 'payload'
})
const results = await client.ft.search(
'idx:cits',
'#position:[0 0 10000 km]'
)
console.log(results)
await client.quit()
}
start(client)
Additionally, I would like to ask if there is maybe another type of database that better suits my needs. I have chosen redis because it offers low latency, and that is the biggest constraint in my environment(I will probably do more writes than reads per second). I only want it to act as a inmediate cache, as persistent data will be stored in another database that does not need to be fast.
Thank you.

You get the Invalid argument type error because Redis does not support nested fields in hashes.
"GEO allows geographic range queries against the value in this attribute. The value of the attribute must be a string containing a longitude (first) and latitude separated by a comma" (https://redis.io/commands/ft.create/)

Inconsistent string to date conversion in Mongoose

Model:
const fooSchema = new mongoose.Schema({
fooCreationDate: {
type: Date
},
bar: [{
barCreationDate: {
type: Date
}
}]
});
const foo = mongoose.model(`foo`, fooSchema);
If we want to search for foo objects that were created between 2022-01-01 and 2022-01-02, we can use the following mongoose query:
foo.find({
fooCreationDate: {
$gte: "2022-01-01T00:00:00.000",
$lt: "2022-01-02T00:00:00.000"
}
});
Please note that I'm using strings instead of date objects. The reason is that the query is passed by the client through an AJAX call with dataType: "jsonp". Every date object that is passed like that to the backend is automatically converted to an ISO string. Despite that, the query works without any issues - the find function automatically parses dates represented as ISO strings.
We'd now like to extract every bar object that was created in the same time range, so we'll need to use an aggregation:
foo.aggregate([{
$unwind: `$bar`,
}, {
$match: {
"bar.barCreationDate": {
$gte: "2022-01-01T00:00:00.000",
$lt: "2022-01-02T00:00:00.000"
}
}
}]);
Unfortunately, nothing is found despite the fact that the database contains matching bar objects. This can be confirmed by passing Date objects instead of strings to the $match aggregation:
foo.aggregate([{
$unwind: `$bar`,
}, {
$match: {
"bar.barCreationDate": {
$gte: new Date("2022-01-01T00:00:00.000"),
$lt: new Date("2022-01-02T00:00:00.000")
}
}
}]);
This query returns some results, so the conclusion is that mongoose accepts ISO date strings in the find function, but can't handle them in the aggregate function.
Is there any known workaround? I could, for example, deep-scan every query object passed from the client and search for ISO date strings, then convert them to Date objects, but that's a bit dirty in my opinion. I'm using mongoose v5.6.4 and mongodb v4.2.2.

How to get current date/time using database timezone in sequelize

I need to get the current time, according to the database timezone (not my local timezone, and not default UTC timezone), using Sequelize. Is there a Sequelize method to do this?
My database is in Eastern time, and when I query the db directly SELECT CURRENT_TIMESTAMP; it returns the date/time in Eastern time (which is correct).
But when I query via Sequelize in Node const [[{time}]] = await db.sequelize.query('SELECT CURRENT_TIMESTAMP AS time'); it returns the date/time in UTC.
Two problems:
1 - I would prefer using a Sequelize method instead of a raw query.
2 - This still doesn't get me the result I want. The time needs to be Eastern.
This is my DB setup:
const sequelize = new Sequelize(dbUrl, {
dialectOptions: {
useUTC: false // for reading from database
},
timezone: '-04:00', // for writing to database
define: {
charset: 'utf8'
}
})
As mentioned above, when I query using the above queries, the date is always returned in UTC, which I did not expect, given I said useUTC: false. How do I get it in Eastern time (the database timezone)?

I'm not aware of a sequelize method like getCurrentDate().
The UTC conversion problem seems to bite everyone (myself included). Here are some details. Not sure if dialectOptions: {useUTC: false }, has any function at all - just adding the typeCast method solved the problem for me.
dialectOptions: {
typeCast: function (field, next) { // for reading from database
if (field.type === 'DATETIME') {
return field.string()
}
return next()
},
The result can be used for a new js Date object:
const sql = 'select current_timestamp';
my_app.my_DB.query(sql, {raw: true, type: Sequelize.QueryTypes.SELECT})
.then(data => {
console.log(data[0].current_timestamp);
let d1 = new Date(data[0].current_timestamp);
});
This works fine for me - but make sure to test thoroughly!

Add useUTC property in your dialectOptions like this
dialectOptions: {
encrypt: false ,
options: {
useUTC: false, // for reading from database
},
},

CastError: Cast to date failed for value date at path

I'm trying to make a query in my database developed in Mongo, we have the schema Examen (or Exam) and we need to find all the exams between two dates.
My system can get both dates and make the Query with this form (im using Moment as well in order to get the correct format)
This is the query that I get from the website
{ "examDate": { "$gt": { "$date": "1556686800000" }, "$lt": { "$date": "1559192400000"} } }
when I run this query in Studio3T it works normally but in the website it has an error:
CastError: Cast to date failed for value "{ '$date': '1559192400000'
}" at path "examDate" for model "Examen"
I have tried to change the '$date' format using the ISO format as well, that didn't work. And is weird that the query works normally in Studio3T but it dont works like this using the Mongo db.find() function.
//this is how I make the find function for my schema
Examen.find((query), (err, info_examen) => {
if (err) return res.status(500).send({ message: `Error: ${err}` })
if (!info_examen) return res.status(404).send({ message: `there is no data` })
res.status(200).send({ info_examen })
})
I expect to get the result (info_examen) in order to make statistics with the data.