I am using Firestore's onSnapshot() method to listen to a collection of ~5,000 documents. In the callback passed to onSnapshot, I am taking the snap and formatting the docs into an array and object.
db.onSnapshot(snap => {
const list = [];
const ref = {};
for (let i = 0, size = snap.size; i < size; i++) {
const doc = snap.docs[i];
const id = doc.id;
const data = doc.data();
list.push(data);
ref[id] = { ...data };
}
// Save list and ref to redux store.
}
This process has been working well with smaller collections. However, now with this larger collection, when a user submits a new document to the collection, the "success" prompt that a user sees is blocked by the for loop.
To elaborate: When a user submits a new document, db.add(newDocument) is called. In the .then() we are re-rendering the page to show the user a confirmation message. At the same time though, the snapshot listener has detected a change in the collection and is now looping over all 5,000 documents again. This only takes a second or two but it produces a noticeable "lag" for the user.
How should this process be handled?
You should probably:
Preserve the value of ref for as long as the listener stays attached - don't recreate it each time.
Iterate snap.docChanges so you will only have to process those changes that will result in your prior ref contents to change. This will be indicated in the type of the change.
This should drastically reduce the number of objects being created and destroyed every time something changes.
Related
here is the code im using to listen to changes in my firestore database:
async mounted() {
let id = [];
let orders = [];
await db.collection("orders").onSnapshot(doc => {
includeMetadataChanges: true;
doc.docs.forEach(x => {
id.push(x.id);
let z = Object.assign(x.data(), { id: x.id });
orders.push(z);
});
});
I'm using vuejs, adding this listener on the mount stage so the arrays depending on this snapshot keep refreshing. but I'm facing a problem which is when changes happen to the database my new snapshot adds the data to the array which results in duplicate keys all over, I can't find an efficient way to reset the arrays on each snapshot before inserting the new version.
id array is an array I use to extract the id then insert it inside the orders array so I can use it internally.
Edit:
async mounted() {
let id = [];
let orders = [];
await db.collection("orders").onSnapshot(doc => {
includeMetadataChanges: true;
orders = []
id = []
doc.docs.forEach(x => {
id.push(x.id);
let z = Object.assign(x.data(), { id: x.id });
orders.push(z);
});
});
console.log(orders)
when I reset orders array in callback I get an empty array.
Edit 2- i found the error:
I miscalculated where to place the save the array.
this.$store.dispatch("mystore/saveOrders", orders);
I should have placed it inside the onSnapshot function so each time it runs I do the save, at first I had in the mounted function right after the onsnapshot listener but I had to reset orders as mentioned in the answer by Frank van Puffelen.
You have two options here:
The simplest one is to clear the array every time your callback gets invoked, because doc.docs contains all relevant data to rebuild the UI anyway. So this would be calling orders = [] at the top of the callback.
If you want more granular control over updating the UI, you can iterate over doc.docChanges, which allows you to view the changes between the snapshots. So with this you can determine what documents were added to the snapshot (initially that'll be all of them), which ones were removed, and which docs were updated.
Many UI frameworks these days perform minimal updates to the UI based on the data you provide them with, so I'd definitely recommend checking whether that is the case for Vue too before taking the second approach.
I have a master page that is a list of items, and a details page where I fetch and can update an Item. I have the following hooks based upon the react-query library:
const useItems = (options) => useQuery(["item"], api.fetchItems(options)); // used by master page
const useItem = id => useQuery(["item", id], () => api.fetchItem(id)); // used by details page
const useUpdateItem = () => {
const queryClient = useQueryClient();
return useMutation(item => api.updateItem(item), {
onSuccess: ({id}) => {
queryClient.invalidateQueries(["item"]);
queryClient.invalidateQueries(["item", id]);
}
});
};
The UpdatePage component has a form component that takes a defaultValue and loads that into it's local "draft" state - so it's sort of "uncontrolled" in that respect, I don't hoist the draft state.
// UpdatePage
const query = useItem(id);
const mutation = useUpdateItem();
return (
{query.isSuccess &&
!query.isLoading &&
<ItemForm defaultValue={query.data} onSubmit={mutation.mutate} />
}
);
The problem is after I update, go to Master page, then back to Details page, the "defaultValue" gets the old item before the query completes. I do see it hitting the API in the network and the new value coming back but it's too late. How do I only show the ItemForm after the data is re-queried? Or is there a better pattern?
My updateItem API function returns the single updated item from the server.
I used setQueryData to solve this.
const useUpdateItem = () => {
const queryClient = useQueryClient();
// Note - api.updateItem is return the single updated item from the server
return useMutation(item => api.updateItem(item), {
onSuccess: data => {
const { id } = data;
// set the single item query
queryClient.setQueryData('item', id], data);
// set the item, in the all items query
queryClient.setQueryData(
['item'],
// loop through old. if this item replace, otherwise, don't
old => {
return old && old.map(d => (d.id === id ? data : d));
}
);
}
});
};
I will say, react-query is picky about the key even if it is fuzzy. Originally my id was from the url search params and a string, but the item coming back from the db an int, so it didn't match. So a little gotcha there.
Also, when I go back to the Master list page, I see the item change, which is kind of weird to me coming from redux. I would have thought it was changed as soon as I fired the synchronous setQueryData. Because I'm using react-router the "pages" are complete remounted so not sure why it would load the old query data then change it.
isLoading will only be true when the query is in a hard loading state where it has no data. Otherwise, it will give you the stale data while making a background refetch. This is on purpose for most cases (stale-while-revalidate). Your data stays in the cache for 5 minutes after your detail view unmounts because that’s the default cacheTime.
Easiest fix would just set that to 0 so that you don’t keep that data around.
You could also react to the isFetching flag, but this one will always be true when a request goes out, so also for window focus refetching for example.
Side note: invalidateQueries is fuzzy per default, so this would invalidate the list and detail view alike:
queryClient.invalidateQueries(["item"])
I had the same issue today. After scanning your code it could be the same issue.
const useItem = id => useQuery(["item", id], () => api.fetchItem(id)); // used by details page
The name of the query should be unique. But based on you details the ID changes depends on the item. By that you call the query "item" with different IDs. There for you will get the cached data back if you have done the first request.
The solution in my case was to write the query name like this:
[`item-${id}`...]
I'm building a React Native app using Firebase. I use the following method (among others) in one of my components to get data from Firebase:
loadMessagesFromFirebase(chatId){
let messageList = [];
let data = {};
let message = {};
const dataRef = firebase.database().ref('chats').child(chatId);
dataRef.on('value', datasnap=>{
data = datasnap.val()
})
for (var sayer in data){
for (var m in data[sayer]){
message = {sayer: sayer, text: m.text, time: new Date(m.created)};
messageList.push(message);
}
}
messageList.sort((a,b) => (a.time > b.time) ? 1: -1);
this.setState({messageList: messageList});
}
The problem is that occasionally, data will load as an empty dictionary (Object {}) and therefore, the message list will be empty. I assume this happens because I'm not giving Firebase enough time to load. How do I make this function asynchronous, so that I can add a "loading" state to the component and not display message information until it's finished loading?
async componentDidMount(){
firebase.initializeApp(FirebaseConfig);
this.loadMessagesFromFirebase(this.state.chatId);
//once the messages are done loading
this.setState({{loading: false}})
}
Additionally, is there a way to make sure that, if data is returned as an empty dictionary, it's because Firebase finished loading and NOT because there isn't data for this chat id?
Answering this question even though OP seems to figured out the answer, since he hasn't explained the underlying concepts.
Firebase sdk uses async programming and observer pattern to provide real time updates.
The right way to Asynchronous Firebase data get on React Native world be as follows.
Initialize firebase sdk only once during application startup. In React terms this can be done inside the constructor of the top level App component.
See firebase docs for the steps. https://firebase.google.com/docs/database/web/start
Inside component constructor or componentDidMount set up the call to firebase function to load data
componentDidMount(){ this.loadMessagesFromFirebase(this.state.chatId); }
In the load messages function at up async call to the firebase realtor database. More reading here https://firebase.google.com/docs/database/web/read-and-write
The main thing to remember here is that all code that has to run after data is available has to be triggered from writing the async call back. Modifying the example code from the question
loadMessagesFromFirebase(chatId){
let data = {};
let output = {};
const dataRef = firebase.database().ref('chats').child(chatId);
dataRef.once('value', datasnap=>{
data = datasnap.val();
// Do something really smart with the data and assign it to output
......
output = data;
// set updates to the state
this.setState({output: output});
})
}
Note the use of once instead of the on function.
The reason for this is that for getting subscription to the observer on can lead to the callback being triggered every time there is a change in data. This can lead to undesirable consequences if the component was designed to only get data once.
Further reading https://firebase.google.com/docs/database/web/read-and-write
In case it is desirable to have the component updated every time there is a data change then use the on function to set up a subscription to that data. However in such a case it is important to cancel the subscription inside of componentWillUnmount
https://firebase.google.com/docs/reference/js/firebase.database.Reference#off
This can be summarized as follows
`
// inside componentDidMount
this.onValueChange = ref.on('value', function(dataSnapshot) { ... });
// Sometime later.... Inside componentWillUnmount
ref.off('value', this.onValueChange);`
Figured it out thanks to vvs' comment. Everything that has to be done asynchronously has to go into that dataref.on() function.
loadMessagesFromFirebase(chatId){
let messageList = [];
let data = {};
let message = {};
const dataRef = firebase.database().ref('chats').child(chatId);
dataRef.on('value', datasnap=>{
data = datasnap.val()
for (var sayer in data){
for (var m in data[sayer]){
message = {sayer: sayer, text: m.text, time: new Date(m.created)};
messageList.push(message);
}
}
messageList.sort((a,b) => (a.time > b.time) ? 1: -1);
this.setState({messageList: messageList});
this.setState({{loading: false}})
})
}
And this is how the function is called:
componentDidMount(){
this.loadMessagesFromFirebase(this.state.chatId);
}
(Having firebase.initializeapp(FirebaseConfig) in there actually causes problems- it has to run once, and earlier, such as when setting up the app)
I recently ran into an issue with firebase (onSnapshot) realtime updates. The problem is that onSnapshot updates the state whenever a document is (created, deleted, updated) which overrides the state.
In other words, let's say I have a variable called state.
let state = null;
// And when I visit (**/homepage**) ,,,, onSnapshot runs.
firebase.collection(someCollection).onSnapshot((snapshot) => {
state = (we save documents we got from collection in state variable);
})
// I display these documents on the /homepage.
// Now, I click and call a function (orderByTitle)
// this function gets docs from firebase ... ordered by title.
async function orderByTitle(){
let docs = await firebase.collection(someCollection).orderBy("title").get();
state = docs; // (it overrides the "state" with docs oredered by title & display on page.
}
// Now, I delete one of the docs.
// The problem starts here as it triggers (onSnapshot) again
// and my ("state" variable) gets override with "unordered docs" again.
So, my question is how you prevent (onSnapshot) from overriding your current state or do you manage two different states? And if you manage two different states then how you remove the current elements from the DOM which are using old state and you force them to use other state.
The .onSnapshot function will update the data each time there's a change in the database which is why state keeps changing.
It appears that you want to read the data once. From the docs
You can listen to a document with the onSnapshot() method. An initial
call using the callback you provide creates a document snapshot
immediately with the current contents of the single document. Then,
each time the contents change, another call updates the document
snapshot
on the other hand, if you want to read data once and not have it continually update use the .get function. Here's an example of getting all documents from a collection once, with no further notifications.
db.collection(someCollection).get().then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
// doc.data() is never undefined for query doc snapshots
console.log(doc.id, " => ", doc.data());
});
});
More reading can be found in the documentation Get Data Once
I have a collection of items in a Firebase Realtime database. Clients subscribe to modifications in the /items path of the database. But this has the effect of sending all items to the client each time a single item is added, updated or deleted. This could be up to 1000 items being sent to the client just because an item text has been updated with as little as one character.
This code works, but does not behave the way I want:
export const startSubscribeItems = () => {
return (dispatch, getState) => {
return new Promise(resolve => {
database.ref('items')
.orderByChild(`members/${uid}`)
.equalTo(true)
.on('value', (snapshot) => {
let items = []
snapshot.forEach( (childSnap) => {
const id = childSnap.key
const item = {id, ...childSnap.val()}
items.push(item)
})
dispatch(setItems(items))
resolve()
})
})
}
}
I wish to make this more network cost effective by only sending the item that has been updated - while keeping client subscriptions.
My initial thought was to implement a subscription for each item:
export const startSubscribeSingleItems = () => {
return (dispatch, getState) => {
return new Promise(resolve => {
database.ref('items')
.orderByChild(`access/members/${uid}`)
.equalTo(true)
.once('value', (snapshot) => {
let items = []
snapshot.forEach( (childSnap) => {
const id = childSnap.key
const item = {id, ...childSnap.val()}
items.push(item)
// .:: Subscribe to single item node ::.
database.ref(`items/${id}`).on('value', (snap)=>{
// Some logic here to handle updates and deletes (remove subscription)
})
})
dispatch(setItems(items))
resolve()
})
})
}
}
This seems a bit cumberstone, and only handles updates and deletes. It does not handle the case of additions made by another client. Additions would have to happen via a separate database node (eg. 'subscriptionAdditions//')? Also - initial load would have to clear all items in "subscriptionAdditions//" since first load reads all items.
Again, cumberstone. :/
In conclusion; Is there a simple and/or recommended way to achieve subscribing to single items while taking several clients into account?
Kind regards /K
Firebase Realtime Database synchronizes state between the JSON structure on the server, and the clients that are observing that state.
You seem to want to synchronize only a subset of that state, as far as I can see mostly about recent changes to the state. In that case, consider modeling the state changes themselves in your database.
As you work with NoSQL databases more, you'll see that is quite common to modify your data model to allow each use-case.
For example, if you only need the current state of nodes that have changed, you can add a lastUpdated timestamp property to each node. Then you can query for only the updates nodes with:
database.ref('items')
.orderByChild('lastUpdated')
.startAt(Date.now())
If you want to listen for changes since the client was last online, you'll want to store the timestamp that they were last online somewhere, and use that instead of Date.now().
If you want to synchronize all state changes, even if the same node was changed multiple times, you'll need to store each state change in the database. By keeping those with chronological keys (such as those generated by push()) or storing a timestamp for each, you can then use the same logic as before to only read state change that your client hasn't processed yet.
Also see:
NoSQL data modeling
How to only get new data without existing data from a Firebase?
Retrieve only childAdded from firebase to my listener in firebase