The Setup:
Hello, my org has built an Angular PWA for our users to work offline in the field on their corporate Surface Go devices. This PWA needs to review and edit various data while users are disconnected, so we researched and settled on working with the IndexedDB (Idb) in the browser. Users have the ability to download various files, PDFs, and images while in the office, go offline and into the field, capture new photos within the app (Idb), and upload this all back to our servers upon return.
We store large JSON and blob objects in the Idb to keep track of everything and users have a good workflow to follow for downloading, fielding, and uploading. We also keep a close watch on the Storage within the browser to ensure the Surfaces do meet the limits.
The PWA is loaded in MS Edge (Version 96.0.1054.62 (Official build) (64-bit)) on Windows and the PWA is installed onto their desktop to function like a native app. The Surfaces have 8GB of RAM and plenty of hard drive space for storage.
The Problem:
This has all worked great for a few months but users have now started experiencing issues with the PWA crashing upon navigating to our app. The browser is opened, navigate to the URL, the app loads for 2-3 seconds, and then the entire browser crashes (all tabs and windows). Users have reported uploading and storing photos (3MB each) when the first crash happens. Users are then out of commission and can no longer access the app by any means, so there defiantly seems like a tipping point somewhere.
We have narrowed it down to a memory (RAM) issue with the browser. When the app starts up and establishes a connection to the Idb, the memory spikes dramatically based on the amount of data stored within it. Looking at the local File System in Windows the users that have been affected have upwards of 4.5GB+ stored in their Idb (this is everything).
Loading the browser .exe into Visual Studio and navigating to the app we can see the result. As soon as the call the indexeddb.open happens, a spike in memory is shown.
Chrome (in this instance) throws memory errors.
We have also tried to increase the amount of memory available to the browser by following these posts with no difference. We have been able to copy out the Idb files from the affected users (C:\Users{user}\AppData\Local\Microsoft\Edge\User Data\Default\IndexedDB), swapped them into our machines, and been able to reproduce the crashing locally.
System memory shows the same spike, but there is obviously a lot more memory to give.
As it stands, our users are dead in the water and unable to open the app and unable to access their data. The raw Idb files are unreadable and every solution we have taken to extract the data (https://github.com/amyboyd/indexeddb-to-json) just fails in the same way.
Questions:
Can we allocate more memory to the browser through some other means?
Can we extract the data from the Idb without the browser?
Can we avoid the memory issues when opening the Idb some other way?
OK, so months later we were able to implement a different approach that goes around the IndexedDB and uses the File System API. This has its own drawbacks but this fixed all my issues by getting raw files out of the IndexedDB.
Related to the IndexedDB itself, we found some very specific issues around the images we were storing there, as they were all being base64 encoded and stored as strings. It seems that when the IndexedDB stores these large strings (3-4MB) it has to decode them when it wakes up to correctly render them on demand at a later time.
Based on the number of large strings it has to do this for when making the initial connection to the database, it causes a huge memory spike (decoding hundreds of base64 strings) and ultimately crashes the browser as the IndexedDB and browser engine just completely crashes.
We were not able to dig too deep into how and what the IndexedDB does under the covers as some proprietary magic from the creators at Google? I'm hoping someone with more knowledge on standards at the WWWC can chime and provide more data on this. I would love to know how it works under the covers and why we can't access the raw data on the file system.
TLDR; Don't store large base64 encoded strings in the IndexedDB.
I'm using WebBluetooth API in Chrome 80 with experimental flag enabled.
I can retrieved all nearby devices' info with navigator.bluetooth.requestLEScan.
However, when it comes to the current device that I'm making scan from, there doesn't seem to be a way to retrieve information about it.
Is there anyway that I can retrieve information of the current bluetooth device?
I couldn't find anything from here, here, nor here
Web Bluetooth does not have a way to get information about the central device. Please use the GitHub Issue that you filed to discuss use cases for this feature.
I work on an old application that used ApplicationCache to work offline. Because of its deprecation, we would like to move to Service Workers to achieve the same goal.
Firefox 67 works very well with my current service worker implementation: when I first access to the application, every file listed as 'to-be-cached' is effectively downloaded and cached. The app can then be accessed offline.
Nevertheless, Chrome 74 and Android WebView (which looks like to be based on Chrome 73 - inside a Cordova app) have a slightly different comportment. When I first access to the application, a request per 'to-be-cached' file is put inside the cache. Navigating through the app works great when I'm online. But then I switch offline and only pages I have already accessed to are now accessible.
Is this a bug or a feature? Whatever it is, is there any workaround?
Finally, here is what I have understood: Firefox compares cached resources on their URLs (string). Thus, giving a list of URLs (string) is sufficient for Firefox to cache them and retrieve them later with a Request object (sent when browsing the web application).
Chrome apparently compares cached resources on another value (I haven't managed to find which one). Then, giving a list of URLs (string) wasn't sufficient for Chrome. Request objects and URLs (string) were not recognized as the same.
Based on W3C's specification cache.addAll method, Chrome behaviour should be the right one. But Firefox behaviour is simplier.
I've been playing around with the chrome.storage.sync API as part of a Google Chrome extension that I'm building.
The API makes it clear that if you sign in to the Chrome browser with your Google account and use chrome.storage.sync.set then all the data that is set will be accessible next time you sign in to a Chrome browser with the same Google account and use chrome.storage.sync.get.
What the API doesn't make particularly clear is how chrome.storage.sync behaves when not signed in to the Chrome browser.
From my experiments it appears that, when not signed in to the Chrome browser, chrome.storage.sync.set and chrome.storage.local.set save to different places.
It says in the API:
When Chrome is offline, Chrome stores the data locally. The next time
the browser is online, Chrome syncs the data. Even if a user disables
syncing, storage.sync will still work. In this case, it will behave
identically to storage.local.
It appears that the place where
Chrome stores the [synced] data locally
is different to where chrome.storage.local.set stores it. Can anyone confirm if this is true?
When Chrome.storage.sync is unable to connect to the internet, it stores data in a new, temporary place locally. It does not store it in Chrome.storage.local and is still accessed through Chrome.storage.sync.
The confusion seems to be in the language. Chrome.storage.local is a separate location, and Chrome.storage.sync will behave LIKE Chrome.storage.local. It does not use the same storage location
Actually chrome description has clearly stated on https://developer.chrome.com/extensions/storage
When Chrome is offline, Chrome stores the data locally. The next time
the browser is online, Chrome syncs the data. Even if a user disables
syncing, storage.sync will still work. In this case, it will behave
identically to storage.local.
I have an offline web application using appcaching. I need to provide it about 10MB - 20MB of data that it will save (client-side) consisting mainly of PNG image files. The operation is as follows:
Web application downloads and installs in appcache (uses manifest)
Web app requests from server PNG data files (how? - see alternatives below)
Occasionally web app resyncs with server, and does small partial updates/deletes/additions to PNG database
FYI: Server is a JSON REST server, that can place files in wwwroot for pickup
Here is my current analysis of client-based "databases" that handle binary blob storage
SEE UPDATE at Bottom
AppCache (via manifest add all the PNG and then update on demand)
CON: any change of a PNG database item will mean complete download of all items in manifest (Really bad news!)
WebStorage
CON: Designed for JSON storage
CON: can only store blobs via base64 encoding (probably fatal flaw due to cost of de-encoding)
CON: Hard limit of 5MB for webStorage http://htmlui.com/blog/2011-08-23-5-obscure-facts-about-html5-localstorage.html
PhoneGap & SQLLite
CON: Sponsor will reject it as a native app requiring certification
ZIP file
Server creates a zip file, places it in wwwroot, and notifies client
user has to manually unzip (At least that is how I see it) and save to client file system
Web app uses FileSystem API to reference files
CON: ZIP might be too large (zip64?), long time to create
CON: Not sure if FileSystem API can always read out of the sandbox (I think so)
USB or SD card (back to the stone age....)
The user will be local to the server before going offline
So we could have him insert a SD card, let the server fill it with PNG files
Then the user will plug it into the laptop, tablet
Web app will use FileSystem API to read the files
CON: Not sure if FileSystem API can always read out of the sandbox (I think so)
WebSQL
CON: w3c has abandoned it (pretty bad)
I might consider a Javascript wrapper that uses IndexedDB and WebSQL as a fall-back
FileSystem API
Chrome supports read/write of blobs
CON: not clear about IE and FireFox (IE10, has non-standard msSave)
caniuse.com reports IOS and Android support (but again, is this just r/w of JSON, or does it include the full blob API for writing?
CON: FireFox folks dislike FileSystem API & not clear if they are supporting saving blobs: https://hacks.mozilla.org/2012/07/why-no-filesystem-api-in-firefox/
PRO: Much faster than IndexedDB for blobs according to jsperf http://jsperf.com/indexeddb-vs-localstorage/15 (page 2)
IndexedDB
Good support in IE10, FireFox (save, read blobs)
Good speed and easier management than a file system (deletes, updates)
PRO: see speed tests: http://jsperf.com/indexeddb-vs-localstorage/15
See this article on storing and display of images in IndexedDB: https://hacks.mozilla.org/2012/02/storing-images-and-files-in-indexeddb/
CON: I confirmed that Chrome does not yet support blob writing (current bug, but not clear when it will be fixed)
UPDATE: A June 2014 blogpost suggests Chrome now supports blobs in IndexedDB
UPDATE: This caniuse/indexeddb confirms: "Chrome 36 and below did not support Blob objects as indexedDB values."; suggesting >Chrome36 supports Blob objects.
LawnChair JavaScript wrapper http://brian.io/lawnchair/
PRO: very clean wrapper for IndexedDB, WebSQL or whatever database you have (think polyfill)
CON: cannot store binary blobs, only data:uri (base64 encoding) (probably fatal flaw due to cost of de-encoding)
IndexedDB JQUERY polyFill https://github.com/axemclion/jquery-indexeddb
Parashuram has writtent a nice JQUERY wrapper for the raw IndexedDB interface
PRO: greatly simplifies using IndexedDB, I was hoping to add a shim/polyfill for Chrome FileSystemAPI
CON: It should handle blobs, but I was unable to get it to work
idb.filesystem.js http://ericbidelman.tumblr.com/post/21649963613/idb-filesystem-js-bringing-the-html5-filesystem-api
Eric Bidelman # Google has written a well tested PolyFill the FileSystem API that uses Indexed DB as a fall back
PRO: FileSystem API is well suited for storing blobs
PRO: works great on FireFox and Chrome
PRO: great for synchronizing with cloud based CouchDB
CON: no clear why, but it is not working on IE10
PouchDB JavaScript Library http://pouchdb.com/
great for syncing a CouchDB with a local DB (uses either WebSQL or IndexedDB (not my problem though)
CON: NO CONS, PouchDB now supports binary blobs for all recent browsers (IE, Chrome, Firefox, Chrome on mobile, etc.) as well as many older browsers. That was not the case when I first did this post.
NOTE: to see a data:uri encoding of PNG I created an example at: http://jsbin.com/ivefak/1/edit
Desired/Usefull/Uneeded Features
No native (EXE, PhoneGap, ObjectiveC, etc) app on client (pure web application)
Only needs to run on latest Chrome, FireFox, IE10 for laptops
Strongly want same solution for Android Tablet (IOS would be nice too) but only need one browser to work (FF, Chrome, etc.)
Fast initial DB population
REQUIREMENT: Very fast retrieval of images by web application from storage (DB, file)
Not meant for consumers. We can restrict browsers, and ask user to do special setup & tasks, but let's minimize that
IndexedDB Implementations
There is an excellent article on how IE,FF,and Chrome internally implement this at: http://www.aaron-powell.com/web/indexeddb-storage
In short:
IE uses the same database format as Exchange and Active Directory for IndexedDB
Firefox is using SQLite so are kind of implementing a NoSQL database in to SQL database
Chrome (and WebKit) are using a Key/ Value store which has heritage in BigTable
My Current Results
I chose to use an IndexedDB approach (and polyfill with FileSystemAPI for Chrome until they ship blob support)
For fetching the tiles, I had a dilemna since the JQUERY folks are kvetching about adding this to AJAX
I went with XHR2-Lib by Phil Parsons, which is very much like JQUERY .ajax() https://github.com/p-m-p/xhr2-lib
Performance for 100MB downloads (IE10 4s, Chrome 6s, FireFox 7s).
I could not get any of the IndexedDB wrappers to work for blobs (lawnchair, PouchDB, jquery-indexeddb, etc.)
I rolled my own wrapper, and performance is (IE10 2s, Chrome 3s, FireFox 10s)
With FF, I assume we are seeing the performance issue of using a relational DB (sqllite) for a non-sql storage
NOTE, Chrome has outstanding debug tools (developer tab, resources) for inspecting the state of the IndexedDB.
FINAL Results posted below as answer
Update
PouchDB now supports binary blobs for all recent browsers (IE, Chrome, Firefox, Chrome on mobile, etc.) as well as many older browsers. That was not the case when I first did this post.
Results Offline blob cache for PNG slippy maps
Testing
171 PNG files (total of 3.2MB)
Platforms tested: Chrome v24, FireFox 18, IE 10
Should also work with Chrome & FF for Android
Fetch from web server
using XHR2 (supported on almost all browsers) for blob download from web server
I went with XHR2-Lib by Phil Parsons, which is very much like JQUERY .ajax()
https://github.com/p-m-p/xhr2-lib
Storage
IndexedDB for IE and FireFox
Chrome: Polyfill (blob stored using FileSystem API, reference kept in IndexedDB) polyfill
A Must read article on "How the browsers store IndexedDB data"
http://www.aaron-powell.com/web/indexeddb-storage
Note: FireFox uses SQLlite for the NOSQL IndexedDB. That might be the reason for the slow performance. (blobs stored separately)
Note: Microsoft IE uses the extensible storage engine:
http://en.wikipedia.org/wiki/Extensible_Storage_Engine
Note: Chrome uses LevelDB http://code.google.com/p/leveldb/
Display
I am using Leaflet http://leafletjs.com/ to show the map tiles
I used the functional tile layer plugin by Ishmael Smyrnow for fetching the tile layer from the DB
https://github.com/ismyrnow/Leaflet.functionaltilelayer
I compared the DB-based tiles layer with a purely local (localhost://) storage
There is no noticeable difference in performance! between using IndexedDB and local files!
Results
Chrome: Fetch (6.551s), Store (8.247s), Total Elapsed Time: (13.714s)
FireFox: Fetch (0.422s), Store (31.519s), Total Elapsed Time: (32.836s)
IE 10: Fetch (0.668s), Store: (0.896s), Total Elapsed Time: (3.758s)
For your requirements I suggest that developing a new polyfill based on two others:
FileSystem API to IndexedDB and IndexedDB to WebSQL — is the best option.
The former one will enable support for storing blobs in Chrome (FileSystem API) and Firefox (IndexedDB), while the latter should provide the support for Android and iOS (WebSQL). What is needed is just making these polyfills work together, and I suppose it's not hard.
NB: Since I couldn't find any information on the web about this, you should test if storing blobs using the WebSQL polyfill will work on iOS and Android. It looks like it should work though:
var sql = ["CREATE TABLE", idbModules.util.quote(storeName), "(key BLOB", createOptions.autoIncrement ? ", inc INTEGER PRIMARY KEY AUTOINCREMENT" : "PRIMARY KEY", ", value BLOB)"].join(" ")
Source
I have map caching examples(open example, discover regions and zooms, switch offline and discovered regions will availaible).
There are map.js - map layer for offline tiles, storage.js - storage implementation based on IndexedDb and WebSQL (but this just test implementation with poor performance).
For site files (html, css, js and etc.) I prefer use application cache.
For storage I prefer use Indexed DB (support blob), Web SQL (only base64), FileWriter (support blob, but only chrome). Frankly storage is big issue for this. You need the fastest key value solution that will mix them all. I think is good decision to use exist solution.
For fetching I used canvas with CORS. But I thinking about WebWorkers and XHR2 and this can be better instead canvas because canvas have some troubles with CORS in different browsers and other (for example this title was stored bad in opera).
Additional information about sizes for 2 billion city (Minsk):
Zoom - 9, tiles - 2, size - 52 kb, with previous - 52 kb;
Zoom - 10, tiles - 3, size - 72 kb, with previous - 124 kb;
Zoom - 11, tiles - 7, size - 204 kb, with previous - 328 kb;
Zoom - 12, tiles - 17, size - 348 kb, with previous - 676 kb;
Zoom - 13, tiles - 48, size - 820 kb, with previous - 1.5 mb;
Zoom - 14, tiles - 158, size - 2.2 mb, with previous - 3.7 mb;
Zoom - 15, tiles - 586, size - 5.5 mb, with previous - 9.3 mb;
Zoom - 16, tiles - 2264, size - 15 mb, with previous - 24.3 mb;
A few years back (not exactly the stone age), I was using a signed java applet that would query its server for syncing/updating requirements, download appropriate files from the server and save them on the user's filesystem (not a database). That solution might work for you, although you will need someone to write the applet and sign it. For database solutions, such an applet can use the jdbc available for most databases using localhost on a suitable port (e.g., 3306 for MySQL). I believe the applet tag is deprecated in Html5 but it still works. No experience on Android tablets, so can't comment on that part.