i have a parse.com based app with offline capabilities where the whole database is stored locally (localStorage on web clients and parse.com local database on mobile clients). I am looking for a design solution to efficiently update the local database with latest changes in the remote database. The options that I could think of are:
Journaling with code triggers. Setup cloud code triggers (afterSave, afterDelete) for every object and add a log to the journal table every time an object has been saved or destroyed. The clients will query the table for updates and remember lastUpdateTime for subsequent requests.
Pros: a) we can have a very detailed summary what has been changed and who made the change. b) all the changes are instantly available for other clients (e.g. table call be polled for notifications in real time with little delays)
Cons: a) there may be too many entries in the table
Journaling with background job. Setup a background job that queries all tables by updatedAt, populates journal table and saves the lastUpdateTime for subsequent requests.
Pros: a) less entries in the journal table
Cons: a) changes are available with unpredictable delay (not suitable for real time notifications?) b) cannot track deletes, there's still a need to setup another table to track deletes or implement soft-delete c) less details in the log (e.g. when object is created by one user and deleted by another user, we will not know who created an object)
No journal. All clients query all tables by updatedAt and store lastUpdateTime for subsequent requests.
Pros: a) easy to implement, b) changes are instantly available
Cons: a) same problem with deletes as in 2, b) inefficient (i believe that querying 20+ tables by all clients is not a good idea
We also have an UI where user can look through the recent activity (who changed what), so I kind of lean towards number 1 approach, but the potential size of the table is worrying me.
Client needs ability to recover irrespective of current state. This is critical if you are using local storage that may get cleared by the user. In that case you need a recoverable state. Additionally the client needs to be able to fetch only the transaction required / relevant to it.
Implementing a transaction store on the backend
Creating a recovery mechanism in case the localstorage offline is corrupted
Journaling with code triggers or use of event source db type mechanism so that you have complete history and can use that to build tables for the client.
In conclusion - Modified Journaling with Code Triggers (Modified to recover and storing the state for client in server and using that to query the data)
Related
I have an application where users keep writing data to the database that only needs to last for 1 minute. For example, Alice stores {foo: bar} at 10:36:00pm, and at 10:37:00pm, that entry should automatically be deleted from the database to free up memory, as it will never be used again.
Is there a way to do this, or do I have no choice but to manually have users delete their temp data after a minute? That wouldn't be ideal, because if a user "forgets" to (e.g. they lose their internet connection or close the app before they delete), then that entry will stay taking up space forever unless I set up a worker manually cleaning the database of these old entries.
For the record, the intended use case is to have Firestore act as a WebRTC signalling server, where users temporarily signal each other to establish a p2p connection.
As far as I can tell, there’s no such thing in Firestore. This mechanism is commonly described as TTL (Time To Live).
You can, however, roll your own TTL mechanism using Cloud Tasks.
For your use case, very “short-term”, heavy-rotation, ephemeral, and presumably small data, you may also consider Redis, which has built-in TTL and higher performance. Depending on your volume, may turn out cheaper as well.
I'm using Firebase to allow users to save certain information about their logged-in profiles. In total I have 4 different collections, and all of which contain UID-matching documents.
When my site loads up, I need to immediately access data from all 4 collections for a given user's UID. This means that right now, whenever a user reloads my application they generate 4 get requests. However users of my application tend to not change their stored information very often (perhaps once every 20 or so times they load the application), so there's really no need for this data to be re-requested every time.
I've already taken steps to cache all of this data within the user's browser local storage, but here is where my problem lays: how can I know when the data I have cached is out of date?
In the past I've created similar cache mechanisms which ultimately rely on a 'version' flag. Both the data source and the user's local storage contain a 'version'. When both match, the application knows that it has the latest data, however when they are out of sync, the application knows it needs to re-request the data. The 'version' flag on the data source is ultimately changed every time an update occurs which impacts the user in question.
Ideally I want to end up with this flow:
User loads application.
Application compares local data version with version contained in Firebase.
If versions match, use local data.
If versions do not match, request all data.
Is it possible to do something like this with Firebase? Right now I can't see this possible without having to generate an extra set call every time the user's data is modified to change this version flag.
If this isn't possible, how can I know when a user's data needs updating without using up multiple requests?
If it helps, I'm using Firestore and this is a web application with no backend.
Intro:
I've got a complex and long lasting query on the back-end, feeding back the angular app on the front-end.
Currently the angular app uses the cached data on the back-end rather than reading directly from the complex query, which would take few minutes. The cache gets warm every morning and every night.
As users make changes to the UI, and save the data, which is then passed onto the server side, and saved to database. At that time the UI is up to date until the user refreshes the page. At the same time database is up to date, but the cache is stale.
So when the user refreshes the page the stale cache values are displayed on the page.
More info:
I'm now thinking of ways to refresh the cache, and any advice from more experienced folks would be most welcome.
My idea is to refresh the cache by a cache job (one at a time), which is queued as soon as user saves something. The job will have the relevant info what changed, and the whole cache won't have to be recalculated but rather just the bit which changed.
Question part:
What technique can I use to keep the user up to date with the data even if the user refreshes the page? Should I save the 'deltas', on the client side in a form of indexedDB or localstorage, at the same when the data is sent to server. So when the page refreshes the user reads the data from the localstorage or indexed db.
I'm still thinking this through, obviously I don't have much experience in this, any comments on the directions I've taken so far?
Basically I can change anything including back-end/front-end/caching it's still in the POC phase, I'm just trying to be as informed as possible to what worked for other people.
Update
Little more background. I'm working on a index like page, so there are more than one records that can be edited inline.
Also I'm doing some transformation of the flat db records on the back-end, before dumping them into the map like structure, and passing it to the front-end in a form of json.
I would think the simplest way would be to make sure you know the time the cache was created. When you make changes, save the current state of the page in localStorage, along with the time of the cache. When you load the page, you get the cached data, check it's time to see if it is more recent than your localStorage version. If it is, use the cache, if not, reload your data from localStorage since it has the cached data PLUS your changes already.
Your question is too long, let me summarize the facts.
You have a lot of information in the database
Direct search query takes several minutes
To provide fast search, you use cache which is updated two times a day
When user changes the data, database is updated and cache is not, so web page shows outdated information from cache.
This looks like a typical cache using scenario and the solution is obvious: you should update the cache with deltas as soon as database is changed. The real implementation will depend on your application architecture and cache structure.
The typical workflow for your problem would be:
def updateRequest(Request req) {
def tx = db.startTransaction();
tx.execute(createUpdate(req.getData()));
tx.commit(); // if transaction fails, cache is not updated
cache.update(req.getData()); // can be done in background, if you return delta
}
It seems that you are storing your data in tables and you use those tables with a complex query to build a JSON configuration to render your index.html file. I avoided this problem by avoiding tables and using a NoSQL solution. I build the JSON configuration object on the client side and store that JSON configuration object in a NoSQL collection. I do a simple query using the URL to grab the JSON configuration object and render the index.html file.
I have a little experience storing the JSON configuration object with AWS DynamoDB, and if I need to get faster I will probably switch to AWS ElastiCache.
The key is that you need to cache your JSON configuration object with a useful key like the site hostname or some other base URL and use that as your source of truth for index.html rendering.
I have an ionic app and a Parse.com backend. My users can perform CRUD functions on exercise programmes, changing every aspect of the programme including adding, deleting, editing the exercises within it.
I am confused about when to save, when to call the server and how much data can be held in services / $rootScope?
Typical user flow is as below:
Create Programme and Client (Create both on server and store data in $localStorage).
User goes to edit screen where they can perform CRUD functions on all exercises within the programme. Currently I perform a server call on each function so it is synced to the backed.
The user may go back and select a different programme - downloading the data and storing it localStorage again.
My question is how can I ensure that my users data is always saved to the server and offer them a responsive fast user experience.
Would it be normal to have a timeout function that triggers a save periodically? On a mobile the amount of calls to the server is quite painful over a poor connection.
Any ideas on full local / remote sync with Ionic and Parse.com would be welcome.
From my experience, the best way to think of this is as follows:
localStorage is essentially a cache layer, which if up to date is great because it can reduce network calls. However it is limited to the current session, and should be treated as volatile storage.
Your server is your source of truth, and as such, should always be updated.
What this means is, for reads, localstorage is great, you don't need to fetch your data a million times if it hasn't changed. For writes, always trust your server for long term storage.
The pattern I suggest is, on load, fetch any relevant data and save it to local storage. Any further reads should come from local storage. Edits, should go directly to the server, and on success, you can write those changes to localstorage. This way, if you have an error on save, the user can be informed, and/or you can use localstorage as a queue to continue trying to post the data to the server until a full success.
This is called "offline sync" or sometimes "4 ways data binding". The point is to cache data locally and sync it with a remote backend. This is a very common need, but the solutions are unfornately not that common... The ideal flow would follows this philosophy:
save data locally
try to sync it with server (performing auto merges)
And
Periodically sync, along with a timer and maybe some "connection resumed" event
This is very hard to achieve manually. If been searching modules for a long time, and the only ones that come to my mind don't realy fit your needs (become they often are backend providers that give you frontend connectors; and you already have an opiniated backend), but here they are anyway:
Strongloop's Loopback.io
Meteor
PouchDB
I'm building an app that uses Angular.js for the front and Socket.IO & Redis on Express on the back.
The base usage of sockets is to allow one type of users to push items to lists that are consumed by groups of a second type of users.
A simple example:
Students can push messages into a class list and only teachers of this class can see the list.
I'm trying to sync the list between multiple teachers that are connected at different times,
the lists are stored in a Redis store and I'm wondering if the correct approach to sync clients:
A. Send the list on each update - saving the need of having to manage sync in the client and having potential missmatches.
B. Send the list only on connection and apply incremental updates on successive events.
I'm sure this has been addressed in the past as it seems quite a basic issue with socket communication but I was not able to find a definitive answer.
Thanks!
If the list is not particularly large, then I'd think you want to go with something simple. The simplest thing I can think of is as follows:
Student creates change to the list and sends message to the server (which could be an ajax call, doesn't have to be a web socket).
Server receives message and puts it into the appropriate list storage.
Server then looks for any clients monitoring that list and sends an update message to them.
Teacher connects to the server. Any lists that the teacher is monitoring are sent in their entirety to the teacher and they are subscribed to updates for those lists.
This way, you're never actually doing sync which simplifies matters a lot - you're just doing download list and then incremental updates. There's only one master store. If a client goes off-line, they just get a fresh copy of the list and resubscribe to updates when they come back on-line. Avoiding sync makes the whole solution a lot simpler. This assumes the data is not particularly large so it's feasible to just get a fresh copy of the list as needed.
If you do want to do sync, then a fairly straightforward technique is to maintain one master copy of the store on the server and have every change transaction coin a monotonically increasing transaction ID. Then, each synced copy can just keep track of the last transaction ID that they synced and request all transactions since then. The data store needs to keep track of all changes as transactions (often by writing to a transaction log for each transaction or perhaps a feature in some databases) so any given set of transactions can be played back for any client that is syncing.