How to integrate Redux with very large data-sets and IndexedDB - javascript

I have an app that uses a sync API to get its data, and requires to store all the data locally.
The data set itself is very large, and I am reluctant to store it in memory, since it can contains thousands of records. Since I don't think the actual data structure is relevant, let's assume I am building an email client that needs to be accessible offline, and that I want my storage mechanism to be IndexedDB (which is async).
I know that a simple solution would be to not have the data structure as part of my state object and only populate the state with the required data (eg - store email content on state when EMAIL_OPEN action is triggered). This is quite simple, especially with redux-thunk.
However, this would mean I need to compromise 2 things:
The user data is no longer part of the "application state", although in truth it is. Since the sync behavior is complex, and removing it from the app state machine will hurt the elegance of the redux concepts (the way I understand them)
I really like the redux architecture and would like all of my logic to go through it, not just the view state.
Are there any best-practices on how to use redux with a not-in-memory state properties? The thing I find hardest to wrap my head around is that redux relies on synchronous APIs, and so I cannot replace my state object with an async state object (unless I remove redux completely and replace it with my own, async implementation and connector).
I couldn't find an answer using Google, but if there are already good resources on the subject I would love to be pointed out as well.
UPDATE:
Question was answered but wanted to give a better explantation into how I implemented it, in case someone runs into it:
The main idea is to maintain change lists of both client and server using simply redux reducers, and use a connector to listen to these change lists to update IDB, and also to update the server with client changes:
When client makes changes, use reducers to update client change list.
When server sends updates, use reducers to update server change list.
A connector listens to store, and on state change updates IDB. Also maintain internal list of items that were modified.
When updating the server, use list of modified items to pull delta from IDB and send to server.
When accessing the data, use normal actions to pull from IDB (eg using redux-thunk)
The only caveat with this approach is that since the real state is stored in IDB, so we do lose some of the value of having one state object (and also harder to rewind/fast-forward state)

I think your first hunch is correct. If(!) you can't store everything in the store, you have to store less in the store. But I believe I can make that solution sound much better:
IndexedDB just becomes another endpoint, much like any server API you consume. When you fetch data from the server, you forward it to IndexedDB, from where your store is then populated. The store gets just what it needs and caches it as long as it doesn't get too big or stale.
It's really not different than, say, Facebook consuming their API. There's never all the data for a user in the store. References are implemented with IDs and these are loaded when required.
You can keep all your logic in redux. Just create actions as usual for user actions and data changes, get the data you need and process it. The interface is still completely defined by the user data because you always have the information in the store that is needed to GET TO the rest of it when needed. It's just somewhat condensed, i. e. you only save the total number of messages or the IDs of a mailbox until the user navigates to it.

Related

Is it ok to use localStorage instead of Redux or Context API?

I've been working with React for a little while, and after some time i started to ask myself, why not store every piece of data that my components need to share in localstorage instead of using Redux. Redux need so much boilerplate while localstorage is so simple. You can say that you cant storage object in localstorage but you can JSON.stringfy them, and on recovery just parse them back. So problaby there's something that i cant see properly about that, cause Redux is largely used and localstorage is normally used just to save data you dont wish to loss on refresh and things like that.
This question was on my head when I started developing react apps.
There are many reasons than below to use redux over localStorage.
but at least
Using Redux alongside react-redux is not only for store data. don't forget that changing in a state will reRender All components that listen to that state. and that is really what react-redux does.
stringify/parse from localStorage will make your app (on high-scale) slower and also will not sync all components while changing state.
Personal Recommendation After more than 4 years of developing React Apps, use REDUX with easy API like redux-toolkit or rematch
Redux and localStorage have different use cases actually. Redux you'll use to manage your application state across multiple components.
Local Storage you'll use to persist properties in the browser for later usage. The problem is that any change on your localStorage won't reflect on your application. You'll have to do it manually.
The purpose of react-redux is to allow other components to connect to state and then react on changes. You are loosing the whole scope of using react-redux/context api.
The answer is in your question, yes local storage is only used for storing data in the browser while redux and context api solve some different problem. It separates your data layer from your view to easily manage your data state. If the app is not really big then you should consider going with Context API.
You can read this article for more info.
Note, stringifying and parsing itself is a pretty heavy operations for larger datasets.
It's probably OK to use localstorage instead of Redux. It's possible to have changes in localstorage have immediate effect on subscribed react components.
The people at marmelab who built react-admin transitioned from using redux to what they call The Store.
React-admin contains a global, synchronous, persistent store for
storing user preferences. Think of the Store as a key-value database
that persists between page loads.
The store uses the browser local storage (or a memory storage when
localStorage isn’t available). The store is emptied when the user logs
out.
When using react-admin changes in localstorage have immediate effect on subscribed react components. Checkout the readme and try the demo. Do some fiddling with data in localstorage manually, and see how react components rerender. I was amazed when I saw that for the first time.
I think it's ok. It depends on your requirement. In our situation, we need to split a big project into several small projects, whether use Redux or Mobx both cause problem very difficult, so we totally remove Redux and Mobx, just use LocalStorage to save all states. We know it will be slow and cannot rerender when state changes, but we want to accept it, and even add a Refresh button on some page if need to get state from local storage. So the answer is: it's ok to totally remove Redux, just see if you want to accept it or not.
Why use Redux instead of localStorage:
Disk space: You will probably not be deleting data from local storage every time the user quits your website. (you could with onbeforeonunload event enter link description here but it doesn't look like a good pracitce).
Security: If you are saving user's data, you would have to be careful on not mixing users data saved on localStorage.
Why not use Redux instead of localStorage:
"update the state": The truth is that you can listen to changes on localStorage and then change the state. So, I think, this is not a good reason. For example:
window.addEventListener('storage', (event) => { setState("changed")});
Simplicity: as Zhang Buzz, redux can be a pain.
Each case is different and you'll needs to weight the pros and cons to make a good decision.

Which type of data and when should I initialize in the vuex state?

I'm starting to learn about vuex. I have the question should I set this data from the Store or should I load it on the component?
For example, In my app, I load all users (firebase) from a Store Action and read it from the Getters, but when it comes to load one user data, should I fetch it from the state and then to the component or from the component itself?
I just want to make sure to make my life easier when updating or adding a new feature on the app.
Your store should be your only source of truth for global state.
This means that any component that read data should read from the store. This ensures consistency. Of course each component may need some local data, that it gets using other means and can alter the data that it gets from the store, but it should read from there.
Now you only have a dilemma regarding who should write to the store (using mutations and actions). This depends heavily on your use cases. It is perfectly acceptable to write from the components itself.
There are use cases where populating some data from outside any component makes sense. Probably the most common case is auth credentials. It is easier and cleaner to populate auth credentials before mounting the app for instance so you have one choke point for dealing with authed/unauthed users.
Bottom line is, writing to the store depends on your use case and there's no silver bullet here.

What is the proper way to get data out of Relay Modern's internal store?

We are currently exploring some of the undocumented apis in Relay Modern, and so far the best way we have found to grab data out of the Relay Store for use in the application is to call environment.lookup with a selector derived from a graphql query.
this happened because it seems like the RecordSource.get method returns the object from the store but doesn't fetch any data for nodes nested under it. Is there a better way to fetch an object and all connected nodes to it?
Our use case is we are using applyOptimisticUpdate to update Relay's store so that changes that are made prior to saving on our admin application are visible to all components that have requested that piece of data. So once we are done making these changes we'd like to requery the relay Store to get the current state of a record, clean it up for real mutation purposes, and send the updated payload to the server.
Any insights would be appreciated and I will add documentation to Relay with findings (if so desired).
Relay exposes a commitLocalUpdate function, which takes the environment and an updater function as arguments. The updater works the same way as the ones you use in mutations -- it receives a proxy of the store which you can modify and return. You could use the body of this function to read data from the store and emit any side-effects you need.
It feels a bit like abusing the API though, so there's probably a more correct way of doing this.

Partial state changes for Vaadin's AbstractJavascriptComponent

I'm implementing a JavaScript-based Vaadin component that will need to show and update a relatively large data set. I'm doing this by extending AbstractJavaScriptComponent.
I'm trying to keep the JS side as "dumb" as possible, delegating user interactions to the server using RPC, and which updates the shared state. Then the JS connector wrapper's onStateChange function is called with the new state, which causes the DOM to be updated accordingly.
I have 2 problems:
I don't want to transfer the whole data set each time a small part gets updated.
I don't want to entirely rebuild the UI each time either.
I can solve the second problem by keeping the previous state and comparing parts of it to find out what changed and only make the necessary DOM changes.
But that still leaves the first problem.
Do I have to stop using Vaadin's shared state mechanism and instead only use RPC for communicating the changes to the state?
Update:
I've been doing some testing, and it certainly appears that Vaadin's shared state mechanism is horrible in terms of efficiency:
Whenever the component calls getState() in order to update some property in the state object (or even without updating anything), the whole state object is transferred. The only way to avoid this, as far as I can see, is to not use the shared state mechanism and instead use RPC calls to communicate specific state changes to client.
There are some issues with the RPC approach that will need to be resolved, for example: if you change a value multiple times within a single request/response cycle, you don't want to make the RPC call multiple times. Instead, you want only the last value to be sent just like the shared state mechanism only sends the final state in the response. You can keep dirty flags for each part of the state that you want to send separately (or just keep a copy of the previous state and compare), but then you need to somehow trigger the RPC call at the end of the request handling. How can this be done?
Any ideas on this are welcome!
Update 2:
Vaadin 8 fixes the root issue: it sends only the changed state properties. Also, it doesn't call onStateChange() on the JS connector anymore when only doing an RPC call (and not changing any state).
OP is correct in stating that shared state synchronisation is inefficient for AbstractJavaScriptComponent-based components. The entire state object is serialised and made available to the Javascript connector's onStateChange method whenever the connector is marked as dirty. Other non-javascript components handle state updates more intelligently by only sending changes. The exact place in the code where this happens is line 97 in com.vaadin.server.LegacyCommunicationManager.java
boolean supportsDiffState = !JavaScriptConnectorState.class
.isAssignableFrom(stateType);
I'm not sure why state update is handled differently for AbstractJavaScriptComponent-based components. Maybe it's to simplify the javascript connector and remove the need to assemble a complete state object from deltas. It would be great if this could be addressed in a future version.
As you suggest, you could dispense with JavaScriptComponentState completely and rely on server->client RPC for updates. Keep dirty flags in you server-side component or compare old state and new state by any mechanism you want.
To coalesce the changes and send only one RPC call for each change, you could override beforeClientResponse(boolean initial) in your server-side component. This is called just before sending a response to the client and is your chance to add a set of RPC calls to update the client-side component.
Alternatively, you could override encodeState where you have free-reign to send exactly whatever JSON you like to the client. You could choose to add a list of changes to the base JSON object returned by super.encodeSate. Your javascript connector could interpret as appropriate in its onStateChange method.
Edited to add: calling getState() in your server-side component will mark the connector as dirty. If you want to get state without marking it as dirty then use getState(false) instead.
Following our discussion about this, I've created a drop-in replacement for AbstractJavaScriptComponent that transmits state deltas and includes some extra enhancements. It's in the very early stages but should be useful.
https://github.com/emuanalytics/vaadin-enhancedjavascript
The solution is deceptively simple: basically re-enabling state difference calculation by bypassing this line of code in com.vaadin.server.LegacyCommunicationManager.java:
boolean supportsDiffState = !JavaScriptConnectorState.class
.isAssignableFrom(stateType);
The implementation of the solution is complicated by the fact that the Vaadin classes are not easily extended so I've had to copy and re-implement 6 classes.
Vaadin's shared state works exactly like you want out of the box: when a component is added to the DOM first time, the whole shared state is transferred from server to client, so that it's possible to display the component. After that, only changes are transferred. For example, one changes the caption of a visible component by calling component.setCaption("new caption"), Vaadin only transfers that new caption text to client and "merges" that to the client-side shared state instance of the component.

dojo.store.Observable, JSON REST and queryEngine

Does anybody know how to use the JsonRest store in dojo witn an Observable weapper, like the one in dojo.store.Observable?
What do I need, server side, to implement the store and make it work as an Observable one? What about the client side?
The documentation says http://dojotoolkit.org/reference-guide/1.7/dojo/store/Observable.html
If you are using a server side store like the JsonRest store, you will need to provide a queryEngine in order for the update objects to be properly included or excluded from queries. If a queryEngine is not available, observe listener will be called with an undefined index.
But, I have no idea what they mean. I have never created a store myself, and am not 100% familiar with queryEngine (to be honest, I find it a little confusing). Why is queryEngine needed? What does the doc mean by "undefined index"? And how do you write a queryEngine for a JsonRest store? Shouldn't I use some kind of web socket for an observable REST store, since other users might change the data as well?
Confused!
I realize this quesiton is a bit old, but here's some info for future reference. Since this is a multi-part question, I'll break it down into separate pieces:
1) Server-side Implementation of JsonRest
There's a pretty decent write up on implementing the server side of JsonRest Store. It shows exactly what headers JsonRest will generate and what content will be included in the rest. It helps form a mental model of how the JsonRest api is converted into HTTP.
2) Query Engine
Earlier in the same page, how query() works client side is explained. Basically, the query() function needs to be able to receive an object literal (ex: {title:'Learning Dojo',categoryid:5}) and return the objects in the store that match those conditions. "In the store" meaning already loaded into memory on the client, not on the server.
Depending on what you're trying to do, there's probably no need to write your own queryEngine anyway -- just use the built-in SimpleQueryEngine if you're building your own custom store. The engine just needs to be handed an object literal and it adds the whole dojo query() api for you.
3) Observables
My understanding is that the Observables monitor client side changes in the collection of objects (ex: adding or removing a result) or even within a specific object (ex: post 5 has changed title). It does NOT monitor changes that happen server-side. It simply provides a mechanism to notify other aspects of the client-side app that data changed so that all aspects of the page stay synchronized.
There's a whole write up on using Observables under the headings 'Collection Data Binding' and 'Object Data Binding: dojo/Stateful'.
4) Concurrency
There's two things you'd want to do in order to keep your client side data synchronized with the server side data: a) polling for changes from other users on the server, b) using transactions to send data to the server.
a) To poll for changes to the data, you'd want to have your object store track the active query in a variable. Then, use setTimeout() or setInterval() to run the query in the background again every so often. Make sure that widgets or other aspects of your application use Observables to monitor changes in the query result set(s) they depend on. That way, changes on the server by other users would automatically be reflected throughout your application.
b) Use transactions to combine actions that must be combined. Then, make sure the server sends back HTTP 200 Status codes (meaning 'It Worked!'). If the transactions returns a HTTP status in the 400s, then it didn't work for some reason, and you need to requery the data because something changed on the backend. For example, the record you want to update was deleted, so you can't update it. There's a write up on transactions as well under the heading 'Transactional'

Categories