I am looking at writing a mobile web application with offline support, and would prefer to use the HTML5 offline capabilities instead of something like phonegap.
A requirement of the app will be that it store 2 collections of different types of objects (lets call them Books and Authors) which, when online, it will download from the server. If it didn't need to be offline then I'd be using some flavour of SQL tables or (preferably) a document-based DB to store these items, but having done some basic investigation I haven't found anything that feels "right" that works offline.
My question is: what is the best strategy to store those items? What I have considered so far:
HTML5 localStorage - Seems to have relatively wide support but is limited to key value pairs, which would mean a slightly unpleasant data structure - maybe having a total-authors key and then a series of author-1....author-n key-document pairs. This feels wrong to me - like I am trying to work around the intent of the storage.
CouchDb - I've worked with CouchDb quite a bit in the past so would love to use it or an equivalent, but "proper" CouchDb would require a native app (with which I have no experience), and the 2 in-browser equivalents I have seen don't quite seem to do the trick: Lawnchair doesn't seem to add much to the localStorage option and BrowserCouch doesn't appear to have had much activity recently (which always makes me wary).
WebDb - Looks like it would be perfect...except it seems to be officially discontinued and doesn't look to be widely supported
Gears - Discontinued
IndexedDB - Limited browser support for now (at least according to caniuse)
I'm convinced there must be something out there that does what I want, and I've just managed to somehow overlook it. Can anyone suggest the way this ought to be done (as opposed to a way it could be done)?
There's no browser-native simple solution as far I know; you'll have to either go with localStorage or develop for both WebDB and IndexedDB.
But non-native you could use a abstraction layer like lawnchair (uses localStorage by default, plugins available for indexeddb, webdb, ...) or the one SO-user Gatapia wrote for picnet (uses indexeddb & webdb & gears) that you can find on github.
Related
Since being asked to develop a web-app for someone, I have ben thinking about the whole project. One of the main things that the frontend needs is the ability to operate offline. At first it seemed that maintaining the application offline would be easy:
Important information from the database could be replicated into indexedDB.
Storage API's would be useful for storing tidbits of info.
the Application Cache could handle storing assets offline.
My ideas seemed solid, until I did some research. The application cache has been deprecated. Apparently, it had some issues and wasn't as great as I thought. Now it seems nearly impossible to build offline apps. Through research and thought, I have considered a few solutions, but they all have some sort of flaw.
One article I read considered using localStorage for storing assets. This seems ok, I guess, as the application would be single paged, but assets such as CSS, JavaScript libraries, and images would large, and while I could compress them, it seems kind of hacky to store them as strings in localStorage.
MDN pointed me to Service workers. These seem good, but also overcomplicated and their browser support just wont work for me.
I considered using the File API instead of localStorage to handle assets. The problem is that the File API only seems to work with user interaction such as file upload or drag and drop which is not what I need. I would need just to write to files using JavaScript behind the scenes. Even if that, however, I would expect a performance hit especially with user with slower disks.
As you can see from my solutions one of the main factors is speed. I suppose a procedure like this could be isolated from the main application using WebWorkers, but even then, the feeling of storing files in localStorage is not a good one.
I don't believe than any of these solutions are viable ones, but I cannot be too sure. How should I go about storing assets for offline applications? Ideally, I would like mobile support, but as of now I am looking for a solution that:
Will not have a serious degrade on performance
and
Semantically looks good and does not use any hacks or bad practices.
What solutions do I have available? Are any of my above solutions decent?
Application Cache has only just been deprecated by Firefox a couple of weeks ago, but to me it seems a rather rash move on their part as they haven't finished the replacement yet! See https://www.fxsitecompat.com/en-US/docs/2015/application-cache-api-has-been-deprecated/ and https://bugzilla.mozilla.org/show_bug.cgi?id=1204581 (in particular: "We have not shipped service workers yet, and our implementation of Cache API without service workers is not quite useful for replacing appcache yet.")
I reckon it will be at least a couple of years until AppCache is removed from browsers, and as you've discovered, right now it's your only real choice for cross-browser compatibility. As Service Workers become more mature, a wrapper will probably be developed to ease transition from AppCache to the SW equivalent. (Sounds possible, at least: Is Service Worker intended to replace or coexist with Appcache?)
Which brings me to my next point: for offline database stuff, I recommend LocalForage, Mozilla's wrapper for various offline storage options. It will choose the best available option on the user's browser, saving you the hassle of deciding. I've just used it on a project, it's really simple to use. https://mozilla.github.io/localForage/
Speed-wise you'll probably be pleasantly surprised. Even using LocalStorage (which is synchronous, so blocks execution while running) you might never notice any delay in real-world use.
I'm building a relatively complex and data heavy web application in AngularJS. I'm planning to use php as a RESTful backend (with symfony2 and FOSRESTbundle). I have spent weeks looking around for different solutions to on/off line synchronization solutions and there seem to be many half solutions (see list below for some examples). But non of them seem to fit my situation perfectly. How do I go about deciding which strategy will suite me?
What issues that might determine “best practices” for building an on/off line synchronization system in AngularJS and symfony 2 needs some research, but on the top of my head I want to consider things like speed, ease of implementation, future proof (lasting solution), extensibility, resource usage/requirements on the client side, having multiple offline users editing the same data, how much and what type of data to store.
Some of my requirements that I'm presently aware of are:
The users will be offline often and then needs to synchronize (locally created) data with the database
Multiple users share some of the editable data (potential merging issues needs to be considered).
User's might be logged in from multiple devices at the same time.
Allowing large amount of data to be stored offline(up to a gigabyte)
I probably want the user to be able to decide what he wants to store locally.
Even if the user is online I probably want the user to be able to choose whether he uses all (backend) data or only what's available locally.
Some potential example solutions
PouchDB - Interesting strategies for synchronizing changes from multiple sources
Racer - Node lib for realtime sync, build on ShareJS
Meteor - DDP and strategies for sync
ShareJS - Node.js operational transformation, inspired by Google Wave
Restangular - Alternative to $resource
EmberData - EmberJS’s ORM-like data persistence library
ServiceWorker
IndexedDB Polyfill - Polyfill IndexedDB with browsers that support WebSQL (Safari)
BreezeJS
JayData
Loopback’s ORM
ActiveRecord
BackBone Models
lawnchair - Lightweight client-side DB lib from Brian Leroux
TogetherJS - Mozilla Labs’ multi-client state sync/collaboration lib.
localForage - Mozilla’s DOMStorage improvement library.
Orbit.js - Content synchronization library
(https://docs.google.com/document/d/1DMacL7iwjSMPP0ytZfugpU4v0PWUK0BT6lhyaVEmlBQ/edit#heading=h.864mpiz510wz)
Any help would be much appreciated :)
You seem to want a lot of stuff, the sync stuff is hard... I have a solution to some of this stuff in an OSS library I am developing. The idea is that it does versioning of local data, so you can figure out what has changed and therefore do meaningful sync, which also includes conflict resolution etc. This is sort-of the offline meteor as it is really tuned to offline use (for the London Underground where we have no mobile data signals).
I have also developed an eco system around it which includes a connection manager and server. The main project is at https://github.com/forbesmyester/SyncIt and is very well documented and tested. The test app for the ecosystem will be at https://github.com/forbesmyester/SyncItTodoMvc but I have yet to write virtually any docs for it.
It is currently using LocalStorage but will be easy to move to localForage as it actually is using a wrapper around localStorage to make it an async API... Another one for the list maybe?
To work offline with your requeriments I suggest to divide problem into two scenarios: content (html, js, css) and data (API REST).
The content
Will be stored offline by appcache for small apps or for advanced cases with the awesome serviceworkers. Chrome 40+.
The data
Require solve the storage and synchronization and It becames a more difficult problem.
I suggest a deep reading of the Differential Synchronization algorimth, and take next tips in consideration:
Frontend
Store the resource and shadow (using for example url as key) into the localstorage for small apps or into more advanced alternatives (pouchdb,indexdb,...). With the resource you could work offline and when needs synchronize with the server use jsonpath to get diffs between the resource-shadow and to send it to server the PATCH request.
Backend
At backend take in consideration storage the shadow copies into redis.
The two sides (Frontend/Backend) needs to identify the client node, to do so you could use x- syn-token at HTTP header (send it in all request of the client with angular interceptors).
https://www.firebase.com/
it's reliable and proven, and can be used as a backend and sync library for what you're after. but, it costs, and requires some integration coding.
https://goinstant.com/ is also a good hosted option.
In some of my apps, I prefer to have both: syncing db source AND another main database. (mogno/express, php/mysql, etc..). then each db handles what's its best with, and it's features (real-time vs. security, etc...). This is true regardless to sync-db provider (be it Racer or Firebase or GoInstant ...)
The app I am developing has many of the same requirements and is being built in AngularJS. In terms of future proofing, there are two main concerns that I have found, one is hacking attempts requiring encryption and possible use of one time keys and an backend key manager and the other is support for WebSQL being dropped by the standards consortium in preference to indesedDB. So finding an abstraction layer that can support both is important. The solution set I have come up with is fairly straight forward. Where offline data is is loaded first into the UI and a request goes out to the REST Server if in an online state. As for resolving data conflicts in a multi user environment, that becomes a business rule decision. My decision was to simplify the matter and not delve into data mergers but to use a microtime stamp comparison to determine which version should be kept and pushed out to clients. When in offline mode, store data as a dirty write and the push to server when returning to an online state.
Or use ydn-db, which I am evaluating now as it has built in support for AWS and Google cloud storage built in.
Another suggestion:
Yjs leverages an OT-like algorithm to share a wide range of supported data types, and you have the option to store the shared data in IndexedDB (so it is available for offline editing).
I am developing a JavaScript SPA using DurandalJs and BreezeJs as main technologies in the client side. I would like to know what libraries I can use to do my application support different languages. I have been doing a little of searching and I have found that JED.js (http://slexaxton.github.com/Jed/) can be a good option for this task.
However I am not very sure, can somebody recommend nay library for this task, or give me some opinion about JED.js.
Also I would like to ask about some library for store information in the browser. Can somebody recommend me one? I have thought in amplify.js, it gives good functionality for doing request to the server and caching this information. But in my application I am using breezejs, so this part is not needed. I only need to store some data like the username and some little information.
Very thanks.
MY SOLUTION:
I have implemented my system using amplify.storage for storing information in the browser and i18next (http://i18next.com/) to make my application avaliable in different human-languages.
I am not much help on the first part of your question. I'm quite interested in the response myself!
I do have some thoughts on local storage. AmplifyJS is really three independent libraries. One of them, amplify.store, is dedicated to the local storage issue and is worth looking at. You might look at lawnchair. Finally, look at the Breeze DocCode sample for examples of storing full and partial entity caches in browser local storage; it's primitive but gives you hints.
EDIT#2 - The replies until now (after 2 days) are personal opinions and preferences and not an analysis of the various options that an offline-phoneGap app has to store simple data easily across all relevant devices. As such I haven't accepted any answer, but I am following this question.
I am a little confused about which format of persistent data I should be looking into for a PhoneGap web app I'm building. I've been researching this but things are not clear given my mediocre requirements.
The app is an educational app with about 100 or so multiple choice questions and some memorization games attached.
The app once downloaded can remain offline.
It is for all phonegap supported devices.
The only data that I want to read and write is the user's performance, number of times wrong in total, per card etc and any high scores for games.
This is all very basic information and could be held in very simple js objects.
I would like it to be a fairly simple solution and very easy to maintain/repeat.
What would be my best option? The phonegap file api? json/lawnchair? local-storage? cookies? Would there be a way to 'update' the app and keep it as an object in the javascript? websql? sqilite? Storage API?
Some of these seem overkill.
EDIT
Are there differences in devices and I should do some device detection and use different technologies?
I personally like localStorage. It's straight forward and works great for most situations.
If you are just recording the data you mention above, localStorage would be perfect. I would just seralise your data objects by turning them into a string using say JSON.stringify(), then when pulling it back in use JSON.parse() to turn it back into a useable JS object.
How about try out my library http://dev.yathit.com/ydn-db/getting-started.html backed by IndexedDB (excellent performance, query by index scan), WebSQL (good performance, SQL query) or localStorage (fair performance, no query, get by key, 2.5 MB limit).
db = new ydn.db.Storage('test-store');
db.put('store1', {test: 'Hello World!'}, 123);
req = db.get('store1', 123);
req.done(function(record) {
console.log(record);
});
High performance while still go easy.
Don't like library dependency, take raw source code at https://bitbucket.org/ytkyaw/ydn-db
It looks like these are good ones, though I haven't tried them.
localForage
PouchDB
If you're using the ionic framework which uses AngularJS, I like ngStorage. This one I have tried and it's awesome.
I use localStorage to keep my persistent data, but it is somehow not reliable. I have seen some data lost, but I don't know why. But my persistent data usage is not that critical so I don't mind these inconsistencies.
But your case seems more important. I would store my persistent data in Documents folder, with File API.
Phonegap has local storage support for SQL Lite
http://docs.phonegap.com/en/2.2.0/cordova_storage_storage.md.html#Storage
Sorry I don't have more info. I was interested in this topic and happened to come across it.
We would like to use the HTML5 sessionStorage (or something like it) in a new project but you can't rely on it on some browsers. There's this project (http://code.google.com/p/sessionstorage/) which provides an implementation that works a lot of places, yet oddly it doesn't use the HTML5 version if it is available. It always does its own thing.
There's this project (https://github.com/jas-/jQuery.handleStorage) over on GitHub
but its emulation layer is cookies (bleh!).
Note: The above is incorrect, see the answer from jas- below where he explains that I was mistaken about that project.
Is there anything better out there that offers a good implementation like the first project but falls back automatically to HTML5 if it is available? We might retrofit the first project to automatically fall back to HTML5 if available but would rather not build that if there's already a good solution out there.
I know you said that you wanted a solution that did it all but what's wrong with something like this?
if(!window.sessionStorage){
$.getScript("/google_code_sessionStorage.js");
}
There's a couple of YUI utilities for this, depending on whether you're using 2 or 3:
YUI 2: Storage Utility
YUI 3: Storage Lite
Both will use HTML5 session storage if available.
Munsch,
I realize this post is several years old but I wanted to clarify the project you mentioned, I am the author of jQuery.handleStorage and it does not use cookies as its default method of client storage.
If you read the documentation for the project you will see that one of its three client storage options is to use cookies in the event a clients browser does not support the HTML5 localStorage and/or sessionStorage API's.
With that said, if you need a more robust and flexible client storage tool I would recommend secStore.js as it does not implicitly require binding to a form, supports the HTML5 client storage options, gracefully degrades to cookie in the event of a legacy browser and optionally uses the SJCL API to provide transparent encryption of possible sensitive data.
I ended up doing more research on this because I liked both of the answers I got, but I wanted something more like the YUI 3: Storage Lite but that didn't require YUI. Something that was framework agnostic or jQuery friendly would work a lot better for us.
In the end I found a couple of possibilities and one stood out as what we'll try and put to use: Lawnchair
Also considered: store.js, AmplifyJS