I need to keep track of a counter of a collection with a huge number of documents that's constantly being updated. (Think a giant list of logs). What I don't want to do is to have the server send me a list of 250k documents. I just want to see a counter rising.
I found a very similar question here, and I've also looked into the .observeChanges() in the docs but once again, it seems that .observe() as well as .observeChanges() actually return the whole set before tracking what's been added, changed or deleted.
In the above example, the "added" function will fire once per every document returned to increment a counter.
This is unacceptable with a large set - I only want to keep track of a change in the count as I understand .count() bypasses the fetching of the entire set of documents. The former example involves counting only documents related to a room, which isn't something I want (or was able to reproduce and get working, for that matter)
I've gotta be missing something simple, I've been stumped for hours.
Would really appreciate any feedback.
You could accomplish this with the meteor-streams smart package by Arunoda. It lets you do pub/sub without needing the database, so one thing you could send over is a reactive number, for instance.
Alternatively, and this is slightly more hacky but useful if you've got a number of things you need to count or something similar, you could have a separate "Statistics" collection (name it whatever) with a document containing that count.
There is an example in the documentation about this use case. I've modified it to your particular question:
// server: publish the current size of a collection
Meteor.publish("nbLogs", function () {
var self = this;
var count = 0;
var initializing = true;
var handle = Messages.find({}).observeChanges({
added: function (id) {
count++;
if (!initializing)
self.changed("counts", roomId, {nbLogs: count});
},
removed: function (id) {
count--;
self.changed("counts", roomId, {nbLogs: count});
}
// don't care about moved or changed
});
// Observe only returns after the initial added callbacks have
// run. Now return an initial value and mark the subscription
// as ready.
initializing = false;
self.added("counts", roomId, {nbLogs: count});
self.ready();
// Stop observing the cursor when client unsubs.
// Stopping a subscription automatically takes
// care of sending the client any removed messages.
self.onStop(function () {
handle.stop();
});
});
// client: declare collection to hold count object
Counts = new Meteor.Collection("counts");
// client: subscribe to the count for the current room
Meteor.subscribe("nbLogs");
// client: use the new collection
Deps.autorun(function() {
console.log("nbLogs: " + Counts.findOne().nbLogs);
});
There might be some higher level ways to do this in the future.
Related
I'm trying to create a basic TCG style game with Node/Vue/Socket.io and can't seem to figure out how to emit to both clients when a "ready" count = 2 but with different data, I'll explain a little below...
The sequence of events is as such:
player connects -> server sends player a "deck" -> player clicks ready to start and also sends back their first 'card'.. Then the server should send out to each player the other players first card. (Note my emit events don't have the correct titles atm - they were already written up on the front end so just kept them the same)
On connection I've pushed to an array called sockets, that I was using for testing. Then in the "ready" event I created an array called "firstCards" that I'm pushing the socket event data to then adding a .socket property to it (to signify who's who), then incrementing ready.
I've had a little bit of a play around with a few different methods but I can only seem to get the last card sent to both clients as opposed to each client getting the other clients first.. I also tried just putting the "if" statement outside of the socket event (as you will see below with the comment on the brackets/curly braces) which doesn't seem to work either.
I haven't tried this kind of asymmetric data transfer before and unsure if that is even the correct term... or whether this is even the correct way to do so, any help would be much appreciated!
This is the code I'm using so far:
socket.on('ready-up', function (card)
{
console.log(`Player ${socket.id} is ready`);
ready++;
console.log(ready);
card.socket = socket.id;
firstCards.push(card);
console.log(firstCards);
});
if (ready == 2)
{
for (let i = 0; i < sockets.length; i++)
{
io.to(sockets[i]).emit('p2hand', "Both players ready");
let opp = sockets.find(element => element != socket.id);
console.log(`Socket ID is: ${socket.id}`);
console.log(`Opp ID is: ${opp}`);
let card = firstCards.find(element => element.socket == opp)
console.log(card);
io.to(opp).emit('reveal',
{
'name': card.name,
'hp': card.hp,
'mp': card.mp,
'skills': card.skills,
'icon': card.icon
});
// io.to(opp).emit('reveal', card);
ready = 0;
}
}
// });
So I figured this one out for anyone who may end up wanting to do what I was trying to do....
I decided that upon connection, both clients join a room called "game1".
The server will then emit "firstCards" to that room.
After that it was just a case of making sure the player-client know which was the opponents card... Now I could have used the .name property for this, but I decided to add an "id" property using the socket.id instead due to the possibility of the same card being drawn for both players.
I'm thinking that all server-client interactions will now have to carry this property for any other cards in the game such as items, spells, etc
I need to fetch sub-set of documents in Firestore collection modified after some moment. I tried going theses ways:
It seems that native filtering can work only with some real fields in stored document - i.e. nevertheless Firestore API internally has DocumentSnapshot.getUpdateTime() I cannot use this information in my query.
I tried adding my _lastModifiedAt 'service field' via server-side firestore cloud function, but ... that updating of _lastModifiedAt causes recursive invocation of the onWrite() function. I.e. is does also not work as needed (recursion finally stops with Error: quota exceeded (Function invocations : per 100 seconds)).
Are there other ideas how to filter collection by 'lastModifiedTime'?
Here is my 'cloud function' for reference
It would work if I could identify who is modifying the document, i.e. ignore own updates of _lastModified field, but I see no way to check for this
_lastModifiedBy is set to null because of current inability of Firestore to provide auth information (see here)
exports.updateLastModifiedBy = functions.firestore.document('/{collId}/{documentId}').onWrite(event => {
console.log(event.data.data());
var lastModified = {
_lastModifiedBy: null,
_lastModifiedAt: now
}
return event.data.ref.set(lastModified, {merge: true});
});
I've found the way to prevent recursion while updating '_lastModifiedAt'.
Note: this will not work reliably if client can also update '_lastModifiedAt'. It does not matter much in my environment, but in general case I think writing to '_lastModifiedAt' should be allowed only to service accounts.
exports.updateLastModifiedBy = functions.firestore.document('/{collId}/{documentId}').onWrite(event => {
var doc = event.data.data();
var prevDoc = event.data.previous.data();
if( doc && prevDoc && (doc._lastModifiedAt != prevDoc._lastModifiedAt) )
// this is my own change
return 0;
var lastModified = getLastModified(event);
return event.data.ref.set(lastModified, {merge: true});
});
Update: Warning - updating lastModified in onWrite() event causes infinite recursion when trying to delete all documents in Firebase console. This happens because onWrite() is also triggered for delete and writing lastModified into deleted document actually resurrects it. That document propagates back into console and is tried to be deleted once again, indefinitely (until WEB page is closed).
To fix that issue above mentioned code has to be specified individually for onCreate() and onUpdate().
How about letting the client write the timestamp with FieldValue.serverTimestamp() and then validate that the value written is equal to time in security rules?
Also see Mike's answer here for an example: Firestore Security Rules: If timestamp (FieldValue.serverTimestamp) equals now
You could try the following function, which will not update the _lastModifiedAt if it has been marked as modified within the last 5 seconds. This should ensure that this function only runs once, per update (as long as you don't update more than once in 5 seconds).
exports.updateLastModifiedBy = functions.firestore.document('/{collId}/{documentId}').onWrite(event => {
console.log(event.data.data());
if ((Date.now() - 5000) < event.data.data()._lastModifiedAt) {return null};
var lastModified = {
_lastModifiedBy: null,
_lastModifiedAt: now
}
return event.data.ref.set(lastModified, {merge: true});
});
I am trying to remove an item from $firebaseArray (boxes).
The remove funcion:
function remove(boxJson) {
return boxes.$remove(boxJson);
}
It works, however it is immediately added back:
This is the method that brings the array:
function getBoxes(screenIndex) {
var boxesRef = screens
.child("s-" + screenIndex)
.child("boxes");
return $firebaseArray(boxesRef);
}
I thought perhaps I'm holding multiple references to the firebaseArray and when one deletes, the other adds, but then I thought firebase should handle it, no?
Anyway I'm lost on this, any idea?
UPDATE
When I hack it and delete twice (with a timeout) it seems to work:
function removeForce(screenIndex, boxId) {
setTimeout(function () {
API.removeBox(screenIndex, boxId);
}, 1000);
return API.removeBox(screenIndex, boxId);
}
and the API.removeBox:
function removeBox(screenIndex, boxId) {
var boxRef = screens
.child("s-" + screenIndex)
.child("boxes")
.child(boxId);
return boxRef.remove();
}
When you remove something from firebase it is asynchronous. Per the docs the proper way to remove an item is from firebase, using AngularFire is:
var obj = $firebaseObject(ref);
obj.$remove().then(function(ref) {
// data has been deleted locally and in the database
}, function(error) {
console.log("Error:", error);
});
$remove() ... Removes the entire object locally and from the database. This method returns a promise that will be fulfilled when the data has been removed from the server. The promise will be resolved with a Firebase reference for the exterminated record.
Link to docs: https://www.firebase.com/docs/web/libraries/angular/api.html#angularfire-firebaseobject-remove
The most likely cause is that you have a security rules that disallows the deletion.
When you call boxes.$remove Firebase immediately fires the child_removed event locally, to ensure the UI is updated quickly. It then sends the command to the Firebase servers to check it and update the database.
On the server there is a security rule that disallows this deletion. The servers send a "it failed" response back to the client, which then raises a child_added event to fix the UI.
Appearantly I was saving the items again after deleting them. Clearly my mistake:
function removeSelected(boxes) {
var selectedBoxes = Selector.getSelectedBoxes(boxes);
angular.forEach(selectedBoxes, function (box) {
BoxManager.remove(box);
});
Selector.clearSelection(boxes, true);
}
In the clearSelection method I was updating a field on the boxes and saved them again.
Besides the obvious mistake this is a lesson for me on how to work with Firebase. If some part of the system keeps a copy of your deleted item, saving it won't produce a bug but revive the deleted item.
For those, who have the similar issue, but didn't solve it yet.
There are two methods for listening events: .on() and .once(). In my case that was the cause of a problem.
I was working on a migration procedure, that should run once
writeRef
.orderByChild('text_hash')
.equalTo(addItem.text_hash)
.on('value', val => { // <--
if (!val.exists()) {
writeRef.push(addItem)
}
});
So the problem was exactly because of .on method. It fires each time after a data manipulation from FB's console.
Changing to .once solved that.
I have on a website facebook style notification in top right corner. I show there up to 5 latest notifications. I do initial pulling with child_added and also after same firebaseRef child_added listening for new notifications.
Now I'd like to play a sound on new notification and a little number of new notifications.
The only thing I can't figure is how to distinguish when was a new notification and when was it already seen, a.k.a page reload? Is there any other approach than making some new property read?
I was looking around and found some old answers from 2012 with suggestions limitToLast(1) which doesn't help in my case.
EDIT:
https://stackoverflow.com/a/27693310/633154 This #Kato answers recommends to listen only to new notifications which time is more than current Firebase time Firebase.ServerValue.TIMESTAMP. This seems the way to go, but I am creating a new notification with REST API and myself setting timestamp as my server's UTC. So there may be some minor inconsistencies. Shouldn't be a big deal
EDIT 2:
With this query, I'm getting correctly up to 5 last notifications on page load and no new notifications are coming afterwards
notifRef.limitToLast(5).once("value", function(snapshot) {
snapshot.forEach(function(data) {
addNotifications(data.val());
});
});
In the above linked other SO thread #Kato's answer doesn't work, notifRef.orderBy is not a function.
I have tried multiple other versions according to doc
https://www.firebase.com/docs/web/guide/retrieving-data.html#section-queries
My structure is same
{
"messages": {
"$messageid": { // firebase generated key 'JqcEWLFJrl1eaed5naN'
"sender": "kato",
"message": "hello world"
"timestamp": 1433036536108 // Firebase.ServerValue.TIMESTAMP
}
}
}
Here is what i tried to do and errors I'm getting:
var queryRef = notifRef.orderByKey().startAt(Firebase.ServerValue.TIMESTAMP);
Error:Query: When ordering by key, the argument passed to startAt(), endAt(),or equalTo() must be a string.
var queryRef = notifRef.orderByChild('timestamp').startAt(Firebase.ServerValue.TIMESTAMP);
Error: Query: First argument passed to startAt(), endAt(), or equalTo() cannot be an object.
In the documentation I have not seen that to startAt anything but the element position is passed (integer) but not a firebase timestamp object, that's why such error.
Only below compiles, just having startAt without ordering, but it's not shooting any new notifications!
var queryRef = notifRef.startAt(Firebase.ServerValue.TIMESTAMP);
queryRef.on('child_added', function(snap) {
console.log(snap.val());
addNotifications(snap.val());
// TODO clean up if more than 5 notifications
});
Any idea where could be the problem? What is the correct way to listen only to newer notifications than current timestamp?
EDIT 3
Here is my final solution
notifRef.limitToLast(5).once("value", function(snapshot) {
var lastKey = null; // at least 1 key is always present
var count = 0; // because startAt is inclusive, we have to ignore first child_added
snapshot.forEach(function(data) {
addNotifications(data.val());
lastKey = data.key();
});
checkNotifications();
notifRef.orderByKey().startAt(lastKey).on('child_added', function(snap) {
if (count > 0) {
console.log(snap.val());
addNotifications(snap.val());
// TODO clean up if more than 5 notifications
checkNotifications();
}
count++;
});
});
I don't trust browser time, so had to go first by querying last 5 existing keys, and after that passing to startAt the last key I received. notifRef.orderByKey().startAt(lastKey) can't be outside notifRef.limitToLast(5).once("value" because according to doc, once is queried last so the lastKey js variable passed to startAt would be always null.
Also need to have the count variable, because startAt is taking inclusive, but because it was already there, I need to ignore the first one.
Also with this solution when there are more than 5 notifications, I query my backend with checkNotifications only once at the end when notifications are received with once query. Otherwise on child_added it would do up to 5 times on every page load.
If there is anything that could be optimized, please tell
One solution would be to have your local client listen for the last 5 latest notifications via ref.limitToLast(5).on('child_added', ...) and then only render them to the user if some timestamp field on each of those notifications is newer than your local timestamp on the machine.
When writing those notifications from other clients, you could include a timestamp field as specified via Firebase.ServerValue.TIMESTAMP, which will use the server's notion of the Unix timestamp. Readers of that data could then compare that timestamp to their local clock to make the aforementioned determination.
I've been trying to do Meteor's leaderboard example, and I'm stuck at the second exercise, resetting the scores. So far, the furthest I've got is this:
// On server startup, create some players if the database is empty.
if (Meteor.isServer) {
Meteor.startup(function () {
if (Players.find().count() === 0) {
var names = ["Ada Lovelace",
"Grace Hopper",
"Marie Curie",
"Carl Friedrich Gauss",
"Nikola Tesla",
"Claude Shannon"];
for (var i = 0; i < names.length; i++)
Players.insert({name: names[i]}, {score: Math.floor(Random.fraction()*10)*5});
}
});
Meteor.methods({
whymanwhy: function(){
Players.update({},{score: Math.floor(Random.fraction()*10)*5});
},
}
)};
And then to use the whymanwhy method I have a section like this in if(Meteor.isClient)
Template.leaderboard.events({
'click input#resetscore': function(){Meteor.call("whymanwhy"); }
});
The problem with this is that {} is supposed to select all the documents in MongoDB collection, but instead it creates a new blank scientist with a random score. Why? {} is supposed to select everything. I tried "_id" : { $exists : true }, but it's a kludge, I think. Plus it behaved the same as {}.
Is there a more elegant way to do this? The meteor webpage says:
Make a button that resets everyone's score to a random number. (There
is already code to do this in the server startup code. Can you factor
some of this code out and have it run on both the client and the
server?)
Well, to run this on the client first, instead of using a method to the server and having the results pushed back to the client, I would need to explicitly specify the _ids of each document in the collection, otherwise I will run into the "Error: Not permitted. Untrusted code may only update documents by ID. [403]". But how can I get that? Or should I just make it easy and use collection.allow()? Or is that the only way?
I think you are missing two things:
you need to pass the option, {multi: true}, to update or it will only ever change one record.
if you only want to change some fields of a document you need to use $set. Otherwise update assumes you are providing the complete new document you want and replaces the original.
So I think the correct function is:
Players.update({},{$set: {score: Math.floor(Random.fraction()*10)*5}}, {multi:true});
The documentation on this is pretty thorough.