I want to set data with timestamp priority in the past or future, but not at the current date. And then be able to make queries with endAt and StartAt for specific dates (365 days)
The push method is great to set unique IDs for data and manage the order. Is there any method to generate a "unique PushId" like push() method for timestamp in past or future?
You can attempt to create unique ids similar to what push does, but this seems like a lot of work for little gain when there are built in tools in Firebase to order data. The simplest answer is to set a priority on each record using the server timestamp.
ref.push({ ...data..., ".priority": Firebase.ServerValue.TIMESTAMP });
To set one in the future or past, specify the timestamp manually.
ref.push({ ...data..., ".priority": timeInTheFuture });
.info/serverTimeOffset may also be helpful here for handling latency.
To create push ids, you would do something similar to the following:
Get the current timestamp and pad it to a fixed length (i.e. 16 characters)
Append a random series of digits, such as a random number or hash, also padded to a fixed length
Your entry will now look something like this: 000128198239:KHFDBWYBEFIWFE
You now have a lexicographically sortable id based on a timestamp, which is unique
Here's a helpful discussion on sorting numbers lexicographically
Related
Trying to create an activation code which should be unique, but it only consists of specific characters.
So, this is solution which i build
function findByActivationId() {
return Activation
.findOne({activationId})
.lean()
.exec();
}
let activationId = buildActivationId();
while (await findByActivationId(activationId)) {
activationId = buildActivationId();
}
This makes too many db calls, is there any better way to make query to mongodb?
Well, the major problem of checking if key is unique is based on how you are creating those.
Choose the best way for you to avoid bunch of problems later.
Your own generated string as a key
Well, you can do this but it's important to understand few disclaimers
If you want to generate your own key by the code and then compare if it is unique
in the database with all other currently created it can be done. Just create key by your
algorithm then select all keys from db and check if array of selected rows contains this freshly created string
Problems of this solution
As we can see we need to select all keys from DB and then compare each one to freshly created one. Problem can appear when your database is storing big amount of data. Every time application have to "download" big amount of data and then compare it to new one so in addition this might produce some freezes.
But if you are sure that your database will store not that much amount of unique rows, it is cool to work with.
Then it is important to create those keys properly. Now we talking about complexity, more symbols key is created from, harder to get same ones.
Shall we take a look at this example?
If you are creating keys based on letters a-z and numbers 1-9
and the length of key is for example 5, the complexity of this key is 35^5
which generates more than 52 milions possibilities.
Same keys can be generated but it is like a win on a lottery, almost impossible
And then you can just check if generated key is really unique, if not. (oh cmon) Repeat.
Other ways
Use mongodb _id which is always unique
Use UNIX timestamp to create unique key
I want to retrieve the last 20 documents in my large collection in an efficient manner.
This SO post offered this performant solution - but it does not answer my question because my question is specifically dealing with _id index - :
db.collectionName.find().min(minCriteria).hint(yourIndex).limit(N)
However, my collection just contains the default index (_id). I'm just not sure what min criteria would be - I obviously don't want to hardcode an _id value, as the collection is periodically emptied.
itemsCollection.find().min(<minCriteria>).hint({_id:1}).limit(20)
Is there any way to use min with the _id index? Or is my only option creating a new index?
Yes, you can use min with the _id index, as long as your <minCriteria> only reference the _id field.
If your min criteria is on something other than _id, you will need to create an index on that criteria in order to avoid this query being a full collection scan.
The min() cursor method is for establishing a lower bound for the index scan that will service the query. This is probably not what you are looking for to retrieve the most recently added documents.
Assuming each document's _id field contains an ObjectId or some other value that sorts in the order they were inserted, then you can, as noted in the comments, do a reverse sort on _id and limit to the number of documents desired, which can be very efficient.
This query should automatically use the _id index:
db.itemsCollection.find().sort({_id:-1}).limit(20)
The date part of the ObjectId is determined by the system creating the value, which in some cases is a client/application server. This means that clock drift may affect the ordering.
If you want to get the documents that were most recently inserted into the collection, you can use natural order:
db.itemsCollection.find().sort({$natural:-1}).limit(20)
This doesn't use an index, but it should still be fairly performant because it will only scan the number of documents you want to return.
I have set up my data in my firebase realtime database as follows:
Key{
Creation date,
Popularity,
Rating,
Author}
Would it be possible to retrieve the answers to the following questions:
what where the top 50 games last month in terms of popularity?
what where the lowest ranked games this week?
what is the highest rated game today?
Answering one would answer the other one probably, but just to be sure I put them all.
All of these require that you consider multiple properties. And since the Firebase Realtime Database can only order/filter over a single property, you can't perform these without modifying the data model. For a good primer, read my answer here: Query based on multiple where clauses in Firebase
In this case, you'd need three extra properties for each node:
a property combining the month and popularity, for your first query. For example: "month_popularity": "201812_125" (if the popularity is 125).
a property combining the month and rating, for your second query. For example: "week_rating": "2018w51_4".
a property combining the day and rating, for your third query. For example: "day_rating": "20181225_4".
With these properties, you can then order on the interval you want, and filter the values for the range you want. For example to get the top 50 games for this month:
ref.orderByChild("month_popularity").startAt("201812_").endAt("201812~")
Where the ~ is just a character after _ in ASCII, ensuring that we stop returning results after the ones from this month.
So as we know firebase won't let order by multiple childs. I'm looking for a solution to filter my data so at the end I will be able to limit it to 1 only. So if I won't to get the lowest price it will be something like that:
ref.orderByChild("price").limitToFirst(1).on...
The problem is that I also need to filter it by dates (timestamp)
so for that only I will do:
.orderByChild("timestamp").startAt(startValue).endAt(endValue).on...
So for now that's my query and then I'm running on all results and checking for that one row that has the lowest price. my Data is pretty big and contains around 100,000 rows. I can changed it however I want.
for the first query that gets the lowest price but all timestamps causes that the returned row might be the lowest price but not in my dates range. However this query takes ONLY 2 seconds compared to the second one which takes 20 including my code to get the lowest price.
So, what are your suggestions on how to do it best? I know I can make another index which contains the timestamp and the price but those are different data values and it makes it impossible.
full data structure:
country
store
item
price,
timestamp
just to make it even more clear, I have 2 inner loops which runs over all countries and then over all stores. so the real query is something like that:
ref.child(country[i]).child(store[j]).orderByChild("timestamp").startAt(startValue).endAt(endValue).on...
Thanks!
I want to have an array in Redis (using Node) where I can add values to it and specify how long I want it to stay in there. After that time limit, they should be deleted, and ideally be able to call something so I know what just left. ex. I may get a request with 120s, so I want to add that value to a map for that long and then have it deleted.
Is there a better way to do this? I thought of using the EXPIRE but that seems to be just for keys, not elements in an array?
Any thoughts would be great.
This is what I am doing:
app.get('/session/:length', function(req, res) {
var length = parseInt(req.param('length'), 10);
addToArray(length, ip)
var ip = req.connection.remoteAddress;
res.json({ip: ip, length: length});
});
Basically, I when I add it to the array I want it to only keep it in the array for the time that is passed in. So if you say 30 seconds, it's in that array for 30s, and then is gone, and calls a callback. Maybe there is a better way to solve this problem?
What I do now is keep the times added and ip, time in an array and periodically loop through the array checking and deleting, but thought maybe it would be possible in redis to automatically do this.
While there isn't an automatic way to do that in Redis, the common approach to these kind of problems is to use a Redis sorted set. In your case, set the IP as the member's value and the expiry time (now + time to live) as the score using epoch representation.
Instead of looping periodically, you can just call ZREMRANGEBYSCORE every once in a while.
Since set members are unique, however, that means that you'll only be able to save each IP once. If that's OK, just update the score for an IP with every hit from it, otherwise make the member value unique by concatenating the IP with the timestamp.
Lastly, to get the IPs that haven't "expired", use ZRANGEBYSCORE to get members that have scores (expiry times) higher than now. Similarly and before deleting with ZREMRANGEBYSCORE, get the keys that expired for the callback logic that you mentioned.