I have a collection of items in a Firebase Realtime database. Clients subscribe to modifications in the /items path of the database. But this has the effect of sending all items to the client each time a single item is added, updated or deleted. This could be up to 1000 items being sent to the client just because an item text has been updated with as little as one character.
This code works, but does not behave the way I want:
export const startSubscribeItems = () => {
return (dispatch, getState) => {
return new Promise(resolve => {
database.ref('items')
.orderByChild(`members/${uid}`)
.equalTo(true)
.on('value', (snapshot) => {
let items = []
snapshot.forEach( (childSnap) => {
const id = childSnap.key
const item = {id, ...childSnap.val()}
items.push(item)
})
dispatch(setItems(items))
resolve()
})
})
}
}
I wish to make this more network cost effective by only sending the item that has been updated - while keeping client subscriptions.
My initial thought was to implement a subscription for each item:
export const startSubscribeSingleItems = () => {
return (dispatch, getState) => {
return new Promise(resolve => {
database.ref('items')
.orderByChild(`access/members/${uid}`)
.equalTo(true)
.once('value', (snapshot) => {
let items = []
snapshot.forEach( (childSnap) => {
const id = childSnap.key
const item = {id, ...childSnap.val()}
items.push(item)
// .:: Subscribe to single item node ::.
database.ref(`items/${id}`).on('value', (snap)=>{
// Some logic here to handle updates and deletes (remove subscription)
})
})
dispatch(setItems(items))
resolve()
})
})
}
}
This seems a bit cumberstone, and only handles updates and deletes. It does not handle the case of additions made by another client. Additions would have to happen via a separate database node (eg. 'subscriptionAdditions//')? Also - initial load would have to clear all items in "subscriptionAdditions//" since first load reads all items.
Again, cumberstone. :/
In conclusion; Is there a simple and/or recommended way to achieve subscribing to single items while taking several clients into account?
Kind regards /K
Firebase Realtime Database synchronizes state between the JSON structure on the server, and the clients that are observing that state.
You seem to want to synchronize only a subset of that state, as far as I can see mostly about recent changes to the state. In that case, consider modeling the state changes themselves in your database.
As you work with NoSQL databases more, you'll see that is quite common to modify your data model to allow each use-case.
For example, if you only need the current state of nodes that have changed, you can add a lastUpdated timestamp property to each node. Then you can query for only the updates nodes with:
database.ref('items')
.orderByChild('lastUpdated')
.startAt(Date.now())
If you want to listen for changes since the client was last online, you'll want to store the timestamp that they were last online somewhere, and use that instead of Date.now().
If you want to synchronize all state changes, even if the same node was changed multiple times, you'll need to store each state change in the database. By keeping those with chronological keys (such as those generated by push()) or storing a timestamp for each, you can then use the same logic as before to only read state change that your client hasn't processed yet.
Also see:
NoSQL data modeling
How to only get new data without existing data from a Firebase?
Retrieve only childAdded from firebase to my listener in firebase
Related
I am working with ReactJS and Google Firestore. I have a component called GameEntryForm, where you can select from a list of users stored in Firestore. In order to get this list, when I render the GameEntryForm component, I make a query to Firestore. Below is how I am getting the list.
I was wondering if there was a better or faster way to do this. My concern is that as the number of users increases, this could be a slow operation.
function GameEntryForm() {
// prevent rendering twice
const effectRan = useRef(false);
const [usersList, setUsersList] = useState(new Map());
useEffect(() => {
if (effectRan.current === false) {
const getUsers = async () => {
const q = query(collection(firestore, "users"));
const querySnapshot = await getDocs(q);
querySnapshot.forEach((doc) => {
setUsersList(new Map(usersList.set(doc.data().uid, doc.data())));
});
};
getUsers();
return () => {
effectRan.current = true;
};
}
}, []);
}
Your code looks fine at first glance, but
here are many ways to mitigate this issue some of them are as follows:
Implement Pagination Functionality to limit the number of documents that are returned by the query, for more about this topic go through this docs
Use Firestore Offline Caching feature through persistence like one provided here. I understand that your user will be added constantly so there’s not much improvement with this method but you can trigger a new request to the db based on the changed type. This is nicely explained in this thread
You can also use the above caching with a global state management solution(Redux, Context API) and only fetch the list of users once. This way, the list of users would be accessible to all components that need it, and you would only have to make the query once. Someone has created an example for how this will work although not using firestore though.
Last but not least use Real Time lister to View changes between snapshots as provide here in official docs This works great with the offline Caching option.
here is the code im using to listen to changes in my firestore database:
async mounted() {
let id = [];
let orders = [];
await db.collection("orders").onSnapshot(doc => {
includeMetadataChanges: true;
doc.docs.forEach(x => {
id.push(x.id);
let z = Object.assign(x.data(), { id: x.id });
orders.push(z);
});
});
I'm using vuejs, adding this listener on the mount stage so the arrays depending on this snapshot keep refreshing. but I'm facing a problem which is when changes happen to the database my new snapshot adds the data to the array which results in duplicate keys all over, I can't find an efficient way to reset the arrays on each snapshot before inserting the new version.
id array is an array I use to extract the id then insert it inside the orders array so I can use it internally.
Edit:
async mounted() {
let id = [];
let orders = [];
await db.collection("orders").onSnapshot(doc => {
includeMetadataChanges: true;
orders = []
id = []
doc.docs.forEach(x => {
id.push(x.id);
let z = Object.assign(x.data(), { id: x.id });
orders.push(z);
});
});
console.log(orders)
when I reset orders array in callback I get an empty array.
Edit 2- i found the error:
I miscalculated where to place the save the array.
this.$store.dispatch("mystore/saveOrders", orders);
I should have placed it inside the onSnapshot function so each time it runs I do the save, at first I had in the mounted function right after the onsnapshot listener but I had to reset orders as mentioned in the answer by Frank van Puffelen.
You have two options here:
The simplest one is to clear the array every time your callback gets invoked, because doc.docs contains all relevant data to rebuild the UI anyway. So this would be calling orders = [] at the top of the callback.
If you want more granular control over updating the UI, you can iterate over doc.docChanges, which allows you to view the changes between the snapshots. So with this you can determine what documents were added to the snapshot (initially that'll be all of them), which ones were removed, and which docs were updated.
Many UI frameworks these days perform minimal updates to the UI based on the data you provide them with, so I'd definitely recommend checking whether that is the case for Vue too before taking the second approach.
I have a react query to get user data like this
const { data: queryInfo, status: queryInfoLoading } = useQuery('users', () =>
getUsers()),
);
I then have a sibling component that needs the same data from the get users query. Is there a way to get the results of the get users query without re-running the query?
Essentially, I would like to do something like this
const userResults = dataFromUserQuery
const { data: newInfo, status: newInfoLoading } = useQuery('newUserData', () =>
getNewUsers(userResults.name)),
)
As suggested in this related question (how can i access my queries from react-query?), writing a custom hook and reusing it wherever you need the data is the recommended approach.
Per default, react-query will trigger a background refetch when a new subscriber mounts to keep the data in the cache up-to-date. You can set a staleTime on the query to tell the library how long some data is considered fresh. In that time, the data will always come from the cache if it exists and no refreshes will be triggered.
I have a master page that is a list of items, and a details page where I fetch and can update an Item. I have the following hooks based upon the react-query library:
const useItems = (options) => useQuery(["item"], api.fetchItems(options)); // used by master page
const useItem = id => useQuery(["item", id], () => api.fetchItem(id)); // used by details page
const useUpdateItem = () => {
const queryClient = useQueryClient();
return useMutation(item => api.updateItem(item), {
onSuccess: ({id}) => {
queryClient.invalidateQueries(["item"]);
queryClient.invalidateQueries(["item", id]);
}
});
};
The UpdatePage component has a form component that takes a defaultValue and loads that into it's local "draft" state - so it's sort of "uncontrolled" in that respect, I don't hoist the draft state.
// UpdatePage
const query = useItem(id);
const mutation = useUpdateItem();
return (
{query.isSuccess &&
!query.isLoading &&
<ItemForm defaultValue={query.data} onSubmit={mutation.mutate} />
}
);
The problem is after I update, go to Master page, then back to Details page, the "defaultValue" gets the old item before the query completes. I do see it hitting the API in the network and the new value coming back but it's too late. How do I only show the ItemForm after the data is re-queried? Or is there a better pattern?
My updateItem API function returns the single updated item from the server.
I used setQueryData to solve this.
const useUpdateItem = () => {
const queryClient = useQueryClient();
// Note - api.updateItem is return the single updated item from the server
return useMutation(item => api.updateItem(item), {
onSuccess: data => {
const { id } = data;
// set the single item query
queryClient.setQueryData('item', id], data);
// set the item, in the all items query
queryClient.setQueryData(
['item'],
// loop through old. if this item replace, otherwise, don't
old => {
return old && old.map(d => (d.id === id ? data : d));
}
);
}
});
};
I will say, react-query is picky about the key even if it is fuzzy. Originally my id was from the url search params and a string, but the item coming back from the db an int, so it didn't match. So a little gotcha there.
Also, when I go back to the Master list page, I see the item change, which is kind of weird to me coming from redux. I would have thought it was changed as soon as I fired the synchronous setQueryData. Because I'm using react-router the "pages" are complete remounted so not sure why it would load the old query data then change it.
isLoading will only be true when the query is in a hard loading state where it has no data. Otherwise, it will give you the stale data while making a background refetch. This is on purpose for most cases (stale-while-revalidate). Your data stays in the cache for 5 minutes after your detail view unmounts because that’s the default cacheTime.
Easiest fix would just set that to 0 so that you don’t keep that data around.
You could also react to the isFetching flag, but this one will always be true when a request goes out, so also for window focus refetching for example.
Side note: invalidateQueries is fuzzy per default, so this would invalidate the list and detail view alike:
queryClient.invalidateQueries(["item"])
I had the same issue today. After scanning your code it could be the same issue.
const useItem = id => useQuery(["item", id], () => api.fetchItem(id)); // used by details page
The name of the query should be unique. But based on you details the ID changes depends on the item. By that you call the query "item" with different IDs. There for you will get the cached data back if you have done the first request.
The solution in my case was to write the query name like this:
[`item-${id}`...]
I have a "users" table with a "state" column that is jsonb. The logic for updating this state is complicated and is implemented in a "true" programming language (e.g. JavaScript).
I have multiple servers providing an API for updating the state. The load balancing between these servers is non-deterministic, so update requests might not always go to the same server.
The API will grab the current state (SELECT), run the function to update it (updateState) and then update the database (UPDATE) with the result.
It might look like this:
const updateHandler = async (id) =>
const { state } = await db.one(
`SELECT * FROM users WHERE id = $<id> LIMIT 1`,
{ id });
const nextState = updateState(state);
await db.query(
`UPDATE users SET state = $<nextState> WHERE id = $<id>`,
{ id, nextState });
});
Because I have multiple servers, I want to ensure that when the state is updated, it is the result of calling updateState on the state value that is already there. Each server should have to queue (or perhaps throw an error) if another server is already performing an update.
What is the best strategy for doing this?
Using Optimistic Locking Pattern, you need just check the current state to validate your transaction
UPDATE users
SET state = $<nextState>
WHERE id = $<id> and state=$<currentState>
Check the nb row updating to detect the conflict :
1 row => it's ok
0 row => you have a conflict (another process have
already update the same id data)