I have a "users" table with a "state" column that is jsonb. The logic for updating this state is complicated and is implemented in a "true" programming language (e.g. JavaScript).
I have multiple servers providing an API for updating the state. The load balancing between these servers is non-deterministic, so update requests might not always go to the same server.
The API will grab the current state (SELECT), run the function to update it (updateState) and then update the database (UPDATE) with the result.
It might look like this:
const updateHandler = async (id) =>
const { state } = await db.one(
`SELECT * FROM users WHERE id = $<id> LIMIT 1`,
{ id });
const nextState = updateState(state);
await db.query(
`UPDATE users SET state = $<nextState> WHERE id = $<id>`,
{ id, nextState });
});
Because I have multiple servers, I want to ensure that when the state is updated, it is the result of calling updateState on the state value that is already there. Each server should have to queue (or perhaps throw an error) if another server is already performing an update.
What is the best strategy for doing this?
Using Optimistic Locking Pattern, you need just check the current state to validate your transaction
UPDATE users
SET state = $<nextState>
WHERE id = $<id> and state=$<currentState>
Check the nb row updating to detect the conflict :
1 row => it's ok
0 row => you have a conflict (another process have
already update the same id data)
Related
I am working with ReactJS and Google Firestore. I have a component called GameEntryForm, where you can select from a list of users stored in Firestore. In order to get this list, when I render the GameEntryForm component, I make a query to Firestore. Below is how I am getting the list.
I was wondering if there was a better or faster way to do this. My concern is that as the number of users increases, this could be a slow operation.
function GameEntryForm() {
// prevent rendering twice
const effectRan = useRef(false);
const [usersList, setUsersList] = useState(new Map());
useEffect(() => {
if (effectRan.current === false) {
const getUsers = async () => {
const q = query(collection(firestore, "users"));
const querySnapshot = await getDocs(q);
querySnapshot.forEach((doc) => {
setUsersList(new Map(usersList.set(doc.data().uid, doc.data())));
});
};
getUsers();
return () => {
effectRan.current = true;
};
}
}, []);
}
Your code looks fine at first glance, but
here are many ways to mitigate this issue some of them are as follows:
Implement Pagination Functionality to limit the number of documents that are returned by the query, for more about this topic go through this docs
Use Firestore Offline Caching feature through persistence like one provided here. I understand that your user will be added constantly so there’s not much improvement with this method but you can trigger a new request to the db based on the changed type. This is nicely explained in this thread
You can also use the above caching with a global state management solution(Redux, Context API) and only fetch the list of users once. This way, the list of users would be accessible to all components that need it, and you would only have to make the query once. Someone has created an example for how this will work although not using firestore though.
Last but not least use Real Time lister to View changes between snapshots as provide here in official docs This works great with the offline Caching option.
for example, i've a component to fetch and call an API, but that query by default is enabled: false and i should fire that by onClick:
const query = useQuery('key', fetch, { enabled: false })
const exec = () => query.refetch()
return <button onClick={exec}>Load</button>
But i've a new API call after every clicks on the button, actually i want to cancel re-calling the API still the cached data is available and is not stale...
I there any way to implement something like refetch to retrieve cached data but without re-calling the API? our basically react-query has a re-call for any data reteive?
in fact, our data doesn't change frequently and is fix for 2-3 days...
other words, our clients frequently work with nested drop-downs with same API calls and i want to reduce same key queries... imagine that, something like category to select a brand for products
Thanks
You can set staleTime and cacheTime options of your query to Infinity to keep your cache always available
https://react-query.tanstack.com/reference/useQuery
imagine that, something like category to select a brand for products
This is a classic example for react-query where you want to put all dependencies of your query into your cache key (see the official docs). That way, the caches won't override each other, and you'll instantly get the values back from the cache if they are available. Not that you will also get a background refetch, and that's where staleTime comes in. If you don't want that, set a higher staleTime to only retrieve the value from the cache if it exists.
To illustrate, let's take your example of a select of categories for products:
function MyComponent() {
const [category, setCategory] = useState(null)
const { data } = useQuery(['key', category], () => fetchProducts(category), { enabled: !!brand, staleTime: 1000 * 60 * 3 })
<CategorySelect onSelect={setCategory} />
}
Multiple things going on here:
I have some local state, where I store the selected category
This selection drives the query. The query is disabled as long as I have no selection, but enables once the user makes a selection
every time the category changes, the query key changes, which will make react-query trigger a refetch automatically, for the new data
I've set the staleTime to 3 minutes. If I chose a category that I have already chosen, I will get data from the cache. Within 3 minutes, I will only get it from the cache. If my data is older, I will get it from the cache and the data will also be updated in the background.
I have a react query to get user data like this
const { data: queryInfo, status: queryInfoLoading } = useQuery('users', () =>
getUsers()),
);
I then have a sibling component that needs the same data from the get users query. Is there a way to get the results of the get users query without re-running the query?
Essentially, I would like to do something like this
const userResults = dataFromUserQuery
const { data: newInfo, status: newInfoLoading } = useQuery('newUserData', () =>
getNewUsers(userResults.name)),
)
As suggested in this related question (how can i access my queries from react-query?), writing a custom hook and reusing it wherever you need the data is the recommended approach.
Per default, react-query will trigger a background refetch when a new subscriber mounts to keep the data in the cache up-to-date. You can set a staleTime on the query to tell the library how long some data is considered fresh. In that time, the data will always come from the cache if it exists and no refreshes will be triggered.
I have a master page that is a list of items, and a details page where I fetch and can update an Item. I have the following hooks based upon the react-query library:
const useItems = (options) => useQuery(["item"], api.fetchItems(options)); // used by master page
const useItem = id => useQuery(["item", id], () => api.fetchItem(id)); // used by details page
const useUpdateItem = () => {
const queryClient = useQueryClient();
return useMutation(item => api.updateItem(item), {
onSuccess: ({id}) => {
queryClient.invalidateQueries(["item"]);
queryClient.invalidateQueries(["item", id]);
}
});
};
The UpdatePage component has a form component that takes a defaultValue and loads that into it's local "draft" state - so it's sort of "uncontrolled" in that respect, I don't hoist the draft state.
// UpdatePage
const query = useItem(id);
const mutation = useUpdateItem();
return (
{query.isSuccess &&
!query.isLoading &&
<ItemForm defaultValue={query.data} onSubmit={mutation.mutate} />
}
);
The problem is after I update, go to Master page, then back to Details page, the "defaultValue" gets the old item before the query completes. I do see it hitting the API in the network and the new value coming back but it's too late. How do I only show the ItemForm after the data is re-queried? Or is there a better pattern?
My updateItem API function returns the single updated item from the server.
I used setQueryData to solve this.
const useUpdateItem = () => {
const queryClient = useQueryClient();
// Note - api.updateItem is return the single updated item from the server
return useMutation(item => api.updateItem(item), {
onSuccess: data => {
const { id } = data;
// set the single item query
queryClient.setQueryData('item', id], data);
// set the item, in the all items query
queryClient.setQueryData(
['item'],
// loop through old. if this item replace, otherwise, don't
old => {
return old && old.map(d => (d.id === id ? data : d));
}
);
}
});
};
I will say, react-query is picky about the key even if it is fuzzy. Originally my id was from the url search params and a string, but the item coming back from the db an int, so it didn't match. So a little gotcha there.
Also, when I go back to the Master list page, I see the item change, which is kind of weird to me coming from redux. I would have thought it was changed as soon as I fired the synchronous setQueryData. Because I'm using react-router the "pages" are complete remounted so not sure why it would load the old query data then change it.
isLoading will only be true when the query is in a hard loading state where it has no data. Otherwise, it will give you the stale data while making a background refetch. This is on purpose for most cases (stale-while-revalidate). Your data stays in the cache for 5 minutes after your detail view unmounts because that’s the default cacheTime.
Easiest fix would just set that to 0 so that you don’t keep that data around.
You could also react to the isFetching flag, but this one will always be true when a request goes out, so also for window focus refetching for example.
Side note: invalidateQueries is fuzzy per default, so this would invalidate the list and detail view alike:
queryClient.invalidateQueries(["item"])
I had the same issue today. After scanning your code it could be the same issue.
const useItem = id => useQuery(["item", id], () => api.fetchItem(id)); // used by details page
The name of the query should be unique. But based on you details the ID changes depends on the item. By that you call the query "item" with different IDs. There for you will get the cached data back if you have done the first request.
The solution in my case was to write the query name like this:
[`item-${id}`...]
I have a collection of items in a Firebase Realtime database. Clients subscribe to modifications in the /items path of the database. But this has the effect of sending all items to the client each time a single item is added, updated or deleted. This could be up to 1000 items being sent to the client just because an item text has been updated with as little as one character.
This code works, but does not behave the way I want:
export const startSubscribeItems = () => {
return (dispatch, getState) => {
return new Promise(resolve => {
database.ref('items')
.orderByChild(`members/${uid}`)
.equalTo(true)
.on('value', (snapshot) => {
let items = []
snapshot.forEach( (childSnap) => {
const id = childSnap.key
const item = {id, ...childSnap.val()}
items.push(item)
})
dispatch(setItems(items))
resolve()
})
})
}
}
I wish to make this more network cost effective by only sending the item that has been updated - while keeping client subscriptions.
My initial thought was to implement a subscription for each item:
export const startSubscribeSingleItems = () => {
return (dispatch, getState) => {
return new Promise(resolve => {
database.ref('items')
.orderByChild(`access/members/${uid}`)
.equalTo(true)
.once('value', (snapshot) => {
let items = []
snapshot.forEach( (childSnap) => {
const id = childSnap.key
const item = {id, ...childSnap.val()}
items.push(item)
// .:: Subscribe to single item node ::.
database.ref(`items/${id}`).on('value', (snap)=>{
// Some logic here to handle updates and deletes (remove subscription)
})
})
dispatch(setItems(items))
resolve()
})
})
}
}
This seems a bit cumberstone, and only handles updates and deletes. It does not handle the case of additions made by another client. Additions would have to happen via a separate database node (eg. 'subscriptionAdditions//')? Also - initial load would have to clear all items in "subscriptionAdditions//" since first load reads all items.
Again, cumberstone. :/
In conclusion; Is there a simple and/or recommended way to achieve subscribing to single items while taking several clients into account?
Kind regards /K
Firebase Realtime Database synchronizes state between the JSON structure on the server, and the clients that are observing that state.
You seem to want to synchronize only a subset of that state, as far as I can see mostly about recent changes to the state. In that case, consider modeling the state changes themselves in your database.
As you work with NoSQL databases more, you'll see that is quite common to modify your data model to allow each use-case.
For example, if you only need the current state of nodes that have changed, you can add a lastUpdated timestamp property to each node. Then you can query for only the updates nodes with:
database.ref('items')
.orderByChild('lastUpdated')
.startAt(Date.now())
If you want to listen for changes since the client was last online, you'll want to store the timestamp that they were last online somewhere, and use that instead of Date.now().
If you want to synchronize all state changes, even if the same node was changed multiple times, you'll need to store each state change in the database. By keeping those with chronological keys (such as those generated by push()) or storing a timestamp for each, you can then use the same logic as before to only read state change that your client hasn't processed yet.
Also see:
NoSQL data modeling
How to only get new data without existing data from a Firebase?
Retrieve only childAdded from firebase to my listener in firebase