I am looking to create an App (a game) which will get its data from an API but once it has that data will work offline if necessary. I therefore want to:
Keep a copy of the data in localStorage
Be able to operate offline with the stored data it has if no network connectivity by caching the results of API calls
Unfortunately its been some time since I last did something similar and I can code it manually, but is there a more modern way of doing this? I wish to have permanent access to about 1000 objects which get their data from an API call https://myapi/{name}. If I have a local copy of the object, I only need to try and update it if more than 24 hours has elapsed, but even if that fails due to no connectivity I can use the existing data if I have it.
So my thoughts are that I can subscribe to a Redux store to save to local storage, but how do I manage the API calls and updating of the objects? I am thinking of using something like react-query or RTK Query but don't know how to go about this. I can already query the API using RTK Query but am uncertain how to use it to get my specialised requirements addressed.
Related
Firebase Realtime Database offers powerful synchronisation across clients... But it does seem to have rather high bandwidth charges.
I was hence wondering if it is possible to perform an "initial sync" on clients, so to speak, where we load a reasonably recent version of the database from elsewhere (with lower bandwidth charges), and then have the Firebase SDK (flutter, web) sync from there instead of downloading all the required parts of the database. Maybe a "load from JSON" sort of function or similar.
Seeing as to how the Firebase SDKs do seem to store some state locally on clients (for offline operation) and sync up with Firebase once online again, I was wondering if it might be possible to:
set the local state to our "recent version" downloaded from another source, along with that version's timestamp
trick the client SDK into thinking that we are recovering from an "offline" state
then let the SDK communicate with the Firebase server to get changes since our last timestamp
Are there any approaches to achieve the effect of saving bandwidth through performing an initial sync from a cheaper source? Thanks in advance!
Unfortunately every get() and every realtime listener will load all data it needs from the database. I had the same idea when started working with the Firebase RTDB but unfortunately at this moment it's not possible.
The only way to reduce the bandwith is to read data in smaller chunkgs as possible.
You might want to take a look at AceBase, which is an open source alternative for the Firebase RTDB. It offers the same functionality, has powerful indexing and querying options, offline support, synchronization etc. Easy to setup and you can host it anywhere you want. You can even use it as a standalone realtime database in the browser, so you could also use it in combination with Firebase to perform custom synchronization between your front and backend db's. AceBase is free and its full source code is public.
I have a stored procedure that take too much time to run as it internally does multiple joins on different tables, does group by, order by and then returns response to nodeJS server. The server than passes on this data received from mysql db to angular/client.
Because the client has to wait for too long to let http request complete. I was thinking of better approaches that will shorten the http request completion time.
Need suggestions on how can i handle or make the data in advance that a user will need later.Like adding a small cache database on server that will store user based data and clears out when user session is destroyed.If i go with this approach than which db will be best?
Or instead of having cache Db can i have a json file on server to store users data and use it?
Having a json file on server would be too cumbersome to handle in cases like updation and query in my opinion. Please have a look at memory based dbs for caching purposes if the data doesnt change too often. My recommendation is redis just because i have the best hands on it. However there are many other other options that you can find out with a quick serch like memcached.
Caching strategy only works if the data you are caching doesnt change too quickly. You can explore different schema strategies for the same.
Im coding a static page app using Angular, which shows various Instagram and Twitter posts of the company, and shows the details of the members. I have few questions regarding this, and would like any help.
Firstly, I have about 100+ contacts to display on the first page. Should I create a Json by myself and retrieve it from the service, or should I create a backend and save it there ? I do not have any backend as of now.
Other thing, I was able to retrieve Instagram Json with media content using their API, the doubt im facing is, once I have the call done, will the Json change automatically when the user adds/edits their posts? Or will the Json be the same as I first called it with? Any help is appreciated. Thanks.
For your case, as you have fewer data using Firebase is the best approach. If you write a backend and maintaining it would cost you more. You can use Firebase service URL to retire those records. In future, if you want to add more data it would be easy.My suggestion is Firebase.
Should I create a Json by myself and retrieve it from the service, or should I create a backend and save it there ?
Are you revealing credentials or other sensitive information in the client? That would be one reason to have a backend apart from Instagram or Twitter. Do you envision exhausting API rate limits of Instagram or Twitter APIs? That would be another reason; you could cache results in your backend to reduce external API traffic. Do you need to process (reduce? translate?) the data before it gets to the client, or are you satisfied with performing any processing on the client (e.g. is it fast enough)?
TL;DR: It depends a lot on your particular requirements.
If you do want a backend, the recommendation in the answer from #praneeth-reddy to use Firebase is excellent. If you only need processing/transformation but no caching or separate storage, then AWS Lambda may also be worth considering. If you need more control (build vs. buy), you could write your own backend.
...will the Json change automatically when the user adds/edits their posts? Or will the Json be the same as I first called it with?
Angular can help you update content automatically if the client side data (think browser JavaScript memory) changes via its automatic change detection functionality, but you would have to provide your own logic (e.g. in Angular services perhaps leveraging RxJS) to update the client side data based on data from the APIs. You could poll to update periodically, or for better performance listen for changes using an asynchronous event/push mechanism such as websockets or streams.
Intro:
I've got a complex and long lasting query on the back-end, feeding back the angular app on the front-end.
Currently the angular app uses the cached data on the back-end rather than reading directly from the complex query, which would take few minutes. The cache gets warm every morning and every night.
As users make changes to the UI, and save the data, which is then passed onto the server side, and saved to database. At that time the UI is up to date until the user refreshes the page. At the same time database is up to date, but the cache is stale.
So when the user refreshes the page the stale cache values are displayed on the page.
More info:
I'm now thinking of ways to refresh the cache, and any advice from more experienced folks would be most welcome.
My idea is to refresh the cache by a cache job (one at a time), which is queued as soon as user saves something. The job will have the relevant info what changed, and the whole cache won't have to be recalculated but rather just the bit which changed.
Question part:
What technique can I use to keep the user up to date with the data even if the user refreshes the page? Should I save the 'deltas', on the client side in a form of indexedDB or localstorage, at the same when the data is sent to server. So when the page refreshes the user reads the data from the localstorage or indexed db.
I'm still thinking this through, obviously I don't have much experience in this, any comments on the directions I've taken so far?
Basically I can change anything including back-end/front-end/caching it's still in the POC phase, I'm just trying to be as informed as possible to what worked for other people.
Update
Little more background. I'm working on a index like page, so there are more than one records that can be edited inline.
Also I'm doing some transformation of the flat db records on the back-end, before dumping them into the map like structure, and passing it to the front-end in a form of json.
I would think the simplest way would be to make sure you know the time the cache was created. When you make changes, save the current state of the page in localStorage, along with the time of the cache. When you load the page, you get the cached data, check it's time to see if it is more recent than your localStorage version. If it is, use the cache, if not, reload your data from localStorage since it has the cached data PLUS your changes already.
Your question is too long, let me summarize the facts.
You have a lot of information in the database
Direct search query takes several minutes
To provide fast search, you use cache which is updated two times a day
When user changes the data, database is updated and cache is not, so web page shows outdated information from cache.
This looks like a typical cache using scenario and the solution is obvious: you should update the cache with deltas as soon as database is changed. The real implementation will depend on your application architecture and cache structure.
The typical workflow for your problem would be:
def updateRequest(Request req) {
def tx = db.startTransaction();
tx.execute(createUpdate(req.getData()));
tx.commit(); // if transaction fails, cache is not updated
cache.update(req.getData()); // can be done in background, if you return delta
}
It seems that you are storing your data in tables and you use those tables with a complex query to build a JSON configuration to render your index.html file. I avoided this problem by avoiding tables and using a NoSQL solution. I build the JSON configuration object on the client side and store that JSON configuration object in a NoSQL collection. I do a simple query using the URL to grab the JSON configuration object and render the index.html file.
I have a little experience storing the JSON configuration object with AWS DynamoDB, and if I need to get faster I will probably switch to AWS ElastiCache.
The key is that you need to cache your JSON configuration object with a useful key like the site hostname or some other base URL and use that as your source of truth for index.html rendering.
I have an ionic app and a Parse.com backend. My users can perform CRUD functions on exercise programmes, changing every aspect of the programme including adding, deleting, editing the exercises within it.
I am confused about when to save, when to call the server and how much data can be held in services / $rootScope?
Typical user flow is as below:
Create Programme and Client (Create both on server and store data in $localStorage).
User goes to edit screen where they can perform CRUD functions on all exercises within the programme. Currently I perform a server call on each function so it is synced to the backed.
The user may go back and select a different programme - downloading the data and storing it localStorage again.
My question is how can I ensure that my users data is always saved to the server and offer them a responsive fast user experience.
Would it be normal to have a timeout function that triggers a save periodically? On a mobile the amount of calls to the server is quite painful over a poor connection.
Any ideas on full local / remote sync with Ionic and Parse.com would be welcome.
From my experience, the best way to think of this is as follows:
localStorage is essentially a cache layer, which if up to date is great because it can reduce network calls. However it is limited to the current session, and should be treated as volatile storage.
Your server is your source of truth, and as such, should always be updated.
What this means is, for reads, localstorage is great, you don't need to fetch your data a million times if it hasn't changed. For writes, always trust your server for long term storage.
The pattern I suggest is, on load, fetch any relevant data and save it to local storage. Any further reads should come from local storage. Edits, should go directly to the server, and on success, you can write those changes to localstorage. This way, if you have an error on save, the user can be informed, and/or you can use localstorage as a queue to continue trying to post the data to the server until a full success.
This is called "offline sync" or sometimes "4 ways data binding". The point is to cache data locally and sync it with a remote backend. This is a very common need, but the solutions are unfornately not that common... The ideal flow would follows this philosophy:
save data locally
try to sync it with server (performing auto merges)
And
Periodically sync, along with a timer and maybe some "connection resumed" event
This is very hard to achieve manually. If been searching modules for a long time, and the only ones that come to my mind don't realy fit your needs (become they often are backend providers that give you frontend connectors; and you already have an opiniated backend), but here they are anyway:
Strongloop's Loopback.io
Meteor
PouchDB