I'm using PouchDB to store values that come from a database (values sent by the server) but also for some values that are only set by the user from front-side (and so should stay local).
Those front-side values should persist accross multiple connections to the app. That's why I store them in PouchDB.
Since I don't care about the revision system (only one user can modify them from one location - its browser), I'm wondering if there is a way to tell PouchDB to say so. I know there is a force parameter for calls, but there is still a need for _rev property in those cases...
Is there a way to get rid of the notion of revision and conflict for one table ?
Cheers
Related
In the design stage for an app that collects large amounts of data...
Ideally, I want it to be an offline-first app and was looking to Pouchdb/Counchdb - However, the data needs to be kept for years for legal reasons, and my concern is that this is going to consume too much local storage over time.
My thoughts were:
handle sync between pouchdb and couchdb myself, allowing me to purge inactive documents from the local store without impacting the couchdb. This feels messy and probably a lot of work
Build a local store using dexie.js and completely write the sync function. It also looks hard work, but may be less as I'm not trying to mess with a sync function
Search harder :)
Conceptually, I guess I'm looking for a 'DB cache' - holding active json document versions and removing documents that have not been touched for X period. It might be that 'offline' mode is handled separate to the DB cache..
Not sure yet if this is the correct answer..
setup a filter on couchdb to screen out old documents (lets say we have a 'date_modified' field in the doc and we filter out any docs with date_modified older than one month)
have a local routine on the client that deletes documents from the local pouchdb that are older than one month ( actually using the remove() method against the local pouchdb, not updating it with _deleted:true) - from https://pouchdb.com/2015/04/05/filtered-replication.html it appears removed documents don't sync.
docs updated on the Pouchdb will replicate normally
there might be a race condition here for replication, we'll see
Which is the best approach for an app that aims to filter data, like 5000+ records, by keeping the response speed in focus?
Filter local memory arrays
Query to db through http API request calls
For my app I use angularjs, php and SQLite3. Right now I load all record from slite db to my table and then filter this field by search. All works great, but when I exceed 3000 records I notice a certain slowing down. By limiting the search on two fields, I get better performance.
My doubt is if changing the model and querying the db I get a better performance or not.
Local array advantages
I can use JavaScript Array map() Method
low consuming data bandwidth
I can see all records in table before filter
I can work, after loading data, in offline.
Local array disadvantages
slowing down performance over 2000 record.
So can you help me to evaluate advantages and disadvantages if I make http API call for any filter action request keeping in focus the performances?
I can't tell about caching in PHP, but for the AngularJS end, there's an approach you can follow:
When the user searches for the first time, fetch the result(s) from db.
Make 2 copies of the data: one presented to the user directly, another can be stores in a local json with a key value pair approach.
3.Next time the user searches for anything, look into the local json first for the result. If the data is present locally, no need for the db query, else make the db query and repeat step 2.
The idea is not to make user wait for every search, you cannot simply call all 5000+ records at once and store locally, and you definitely cannot make db queries every-time since RDMS having that much records simply have low performance issues.
So this seems best to me.
Are there any patterns for cases when something (in my case it's a filters) is stored on client (e.g. localStorage) and you need to run a script once per user/version to migrate data you store. For example, initially there is a filter saved in localStorage with a key myFilter after some time you decide that you need to separate filters per environment, so you need separate dev-myFilter, train-myFilter, etc. You update your code to work with environment-dependant filters, but there are users who have old myFilter and you want with next deployed version to run script which will update the key of saved filter if there is one.
Question is - what are patterns/best practices for that?
I don't know about "best practices", but the obvious technical solution, just like with any API or storage format, is to store a version number alongside the data. If you didn't do so from the start, assume version == 1 when absent.
You may be able to avoid this if the data structure is so unique between versions that the version can be determined simply by examining it.
Either way, you simply perform the translation whenever you spot that the user's data is in the old format.
The downside of this is that you have to keep checking; for a web application this is unlikely to be a bottleneck, but if you can make your data forward-compatible from the outset then you may save a bit of processing time on each request. But for the data to be useful you've got to read it anyway, so a little branching for as long as you wish to maintain backward-compatibility is, again, unlikely to be a big problem.
I'm trying to get my head around the use of IndexDB. I have an SQL database which I access via REST and I'm planning on providing some local caching using IndexDB.
My SQL structure uses a large (and variable) number of tables, each table storing an array of data (time sequence and value) for a specific sensor value. Ideally, I would have assumed I'd create a new object store for each of my tables from MySQL. However, it seems that you can only create an object store when the database is opened which is a bit of a pain.
So, I see a number of options -:
I could use a single object store and add two indexes - one for the time,
and one for the sensor. I'm a little worried that this might have
performance issues, but I'm not sure how data is stored under the
hood.
I could probably detect a new sensor somehow, and open the
database with a new version number. This just feels a little wrong to
me.
I could alternatively use different databases for each sensor,
but I've read somewhere that it's not recommended to use multiple
databases (although it's unclear why since this is possibly the
easiest solution).
I'd welcome any thoughts people have regarding the best structure for this sort of data, that will provide good performance.
If your data sets are independent, example you don't need to combine results from multiple sensors, I suggest you to split them in different tables and/or different databases. Different database option is more convenient for deleting data.
IndexedDB database limit for performance in a single database is for more than 50K data, depending on browser and hardware. I have a couple tests which can measure the speed, just tweak the object size that is inserted and you can test your use case.
If you have less than 10K data per sensor (object store/database) you won't hit big performance issues. One common mistake when inserting batch of data is separate transaction for each insert - this is completely unnecessary, since you can store 10K data with one transaction. If you are working with even larger data set, you can separate the inserting into couple transaction, so you won't block the reading of that database.
Also for every transaction that you do in IndexeDB you need to open a connection, some people use the approach for keeping the one connection alive and reusing it, I prefer the closing and opening a separate connection for each transaction.
Also for faster access, you can store all database info into Local Storage, that way you can track how many databases you have and descriptions for each of them.
Additionally you can take a look at this similar question
Coming from Python/Java/PHP, I'm now building a website. On it I want a list of items to be updated in near-realtime: if items (server side) get added to or deleted from the list, this should be updated on the webpage. I made a simple API call which I now poll every second to update the list using jQuery. Because I need some more lists to be kept updated on the same page I'm afraid this will turn into more than 10 server calls per second from every single open browser, even if nothing gets updated.
This seems not like the logical way to do it, but I don't really know how else to do it. I looked at Meteor, but since the webpage I'm building is part of a bigger system I'm rather restricted in my choices of technology (basic LAMP setup).
Could anybody enlighten me with a tip from the world of real-time websites on how to efficiently keep a list updated?
You can use WebSocket(https://code.google.com/p/phpwebsocket/ ) technology.
but php is not the best language for implement it
A way to work this is using state variables for the different types of data you want to have updated (or not).
In order to avoid re-querying the full tables even if the data set in them has not changed in relation to what a particular client has displayed at any given time, you could maintain a state counter variable for the data type on the server (for example in a dedicated small table) and on the client in a javascript variable.
Whenever an update is done on the data tables on the server, you update the state counter there.
Your AJAX polling calls would then query this state counter, compare it to the corresponding javascript variable, and only do a data-update call if it has changed, updating the local javascript variable to what the server has.
In order to avoid having to poll for each datatype separately, you might want to use an JS object with a member for each datatype.
Note: yes this is all very theoretical, but, hey, so is the question ;)