The Setup:
Hello, my org has built an Angular PWA for our users to work offline in the field on their corporate Surface Go devices. This PWA needs to review and edit various data while users are disconnected, so we researched and settled on working with the IndexedDB (Idb) in the browser. Users have the ability to download various files, PDFs, and images while in the office, go offline and into the field, capture new photos within the app (Idb), and upload this all back to our servers upon return.
We store large JSON and blob objects in the Idb to keep track of everything and users have a good workflow to follow for downloading, fielding, and uploading. We also keep a close watch on the Storage within the browser to ensure the Surfaces do meet the limits.
The PWA is loaded in MS Edge (Version 96.0.1054.62 (Official build) (64-bit)) on Windows and the PWA is installed onto their desktop to function like a native app. The Surfaces have 8GB of RAM and plenty of hard drive space for storage.
The Problem:
This has all worked great for a few months but users have now started experiencing issues with the PWA crashing upon navigating to our app. The browser is opened, navigate to the URL, the app loads for 2-3 seconds, and then the entire browser crashes (all tabs and windows). Users have reported uploading and storing photos (3MB each) when the first crash happens. Users are then out of commission and can no longer access the app by any means, so there defiantly seems like a tipping point somewhere.
We have narrowed it down to a memory (RAM) issue with the browser. When the app starts up and establishes a connection to the Idb, the memory spikes dramatically based on the amount of data stored within it. Looking at the local File System in Windows the users that have been affected have upwards of 4.5GB+ stored in their Idb (this is everything).
Loading the browser .exe into Visual Studio and navigating to the app we can see the result. As soon as the call the indexeddb.open happens, a spike in memory is shown.
Chrome (in this instance) throws memory errors.
We have also tried to increase the amount of memory available to the browser by following these posts with no difference. We have been able to copy out the Idb files from the affected users (C:\Users{user}\AppData\Local\Microsoft\Edge\User Data\Default\IndexedDB), swapped them into our machines, and been able to reproduce the crashing locally.
System memory shows the same spike, but there is obviously a lot more memory to give.
As it stands, our users are dead in the water and unable to open the app and unable to access their data. The raw Idb files are unreadable and every solution we have taken to extract the data (https://github.com/amyboyd/indexeddb-to-json) just fails in the same way.
Questions:
Can we allocate more memory to the browser through some other means?
Can we extract the data from the Idb without the browser?
Can we avoid the memory issues when opening the Idb some other way?
OK, so months later we were able to implement a different approach that goes around the IndexedDB and uses the File System API. This has its own drawbacks but this fixed all my issues by getting raw files out of the IndexedDB.
Related to the IndexedDB itself, we found some very specific issues around the images we were storing there, as they were all being base64 encoded and stored as strings. It seems that when the IndexedDB stores these large strings (3-4MB) it has to decode them when it wakes up to correctly render them on demand at a later time.
Based on the number of large strings it has to do this for when making the initial connection to the database, it causes a huge memory spike (decoding hundreds of base64 strings) and ultimately crashes the browser as the IndexedDB and browser engine just completely crashes.
We were not able to dig too deep into how and what the IndexedDB does under the covers as some proprietary magic from the creators at Google? I'm hoping someone with more knowledge on standards at the WWWC can chime and provide more data on this. I would love to know how it works under the covers and why we can't access the raw data on the file system.
TLDR; Don't store large base64 encoded strings in the IndexedDB.
Related
How much data can we keep in the browser, so that we can have access to the files again after closing / reopening the browser?
LocalStorage seems to allow ~ 5 or 10 MB (depending on the browser).
Other systems might be available (such as IndexedDB), as detailed here, but it seems quite complex.
Question: is there a simple-to-use technique (working at least for Chrome+Firefox, or even better if totally cross-browser, and both mobile and desktop) allowing to store more than 100MB (up to 500MB)?
Use-case: a website allowing to play piano with high quality samples (let's say 150 MB audio samples), and prevent to have to re-download the data each time we open the page.
PS: I'm preferably looking for a solution not requiring to install an extension or a browser plugin.
Yes!
As mentioned in the comments as well IndexedDB is the way to go. PouchDB has a nice FAQ about the storage limits between different browsers.
Though Firefox has no upper limit besides disk space, if your application wishes to store more than 50MB locally [In IndexedDB], Firefox will ask the user using a non-modal dialog to confirm that this is okay.
Android works the same as Chrome as of 4.4+ (IndexedDB), while older versions can store up to 200MB (WebSQL).
And from the Chrome webpage.
Temporary storage is shared among all web apps running in the browser.{#It is also shared across all offline APIs, such as App Cache, IndexedDB, and File System. However, it does not include web storage APIs like Local Storage and Session Storage, which still has a limit of 5 MB per origin.#} The shared pool can be up to 1/3 of the of available disk space. Storage already used by apps is included in the calculation of the shared pool; that is to say, the calculation is based on (available storage space + storage being used by apps) * .333 .
Each app can have up to 20% of the shared pool. As an example, if the total available disk space is 60 GB, the shared pool is 20 GB, and the app can have up to 4 GB. This is calculated from 20% (up to 4 GB) of 1/3 (up to 20 GB) of the available disk space (60 GB).
Context: My client-sided web application should work offline and only depend on HTML and Javascript. The target device is a mobile phone with android, but nice would be a solution which works also in Chrome and Firefox. The pictures should be saved locally on the device and available to save them later on a server.
Approach: With getUserMedia i'm able to take a picture and draw it with canvas. Now i'm stucked where i have to save the picture manually. Last approach was to save it in localStorage to access it later.
Question: Is there another way to avoid saving the picture in the localStorage? At least i could accept solutions which exclusively targeting mobile browser for android (maybe jquery, phonegap,... ?)
I'd appreciate your input.
You can use IndexedDB or WebSQL with phonegap to store your image data.
However, first try using HTTP cache. Set long expiry headers, Cache-control: public. If you don't need long-term off-line capability, it will "just work".
These links will help you with IndexedDB and WebSQL:
http://docs.phonegap.com/en/3.3.0/cordova_storage_storage.md.html
http://plugreg.com/plugin/DickvdBrink/cordova-indexeddb
On an iPhone device, on an HTML page you can always compose an sms (via an sms:1234567890 URL), and if WhatsApp is installed, you can compose a WhatsApp message (via the WhatsApp protocol).
Is it possible to detect such features on an HTML page?
Following a few hours of research + consulting with top experts I came to the conclusion that there is no legitimate way of achieving the goal of detecting whether a specific app is installed on the device without having the browser re-direct to the app if it is installed.
In iOS for example, there was an app just removed from the App Store because of violating the rules: SysSecInfo. The app managed to pull list of all running processes for example.
From: https://www.sektioneins.de/en/blog/16-05-09-system-and-security-info.html
See https://developer.apple.com/videos/play/wwdc2015/703/ "App
Detection" starting at 08:34
During this talk they discuss several APIs used to gather information
about processes currently running on your system (around 12:12 in the
video) and claimed to have fixed them. However as so often Apple has
only partially fixed the problems they claim to have fixed. Therefore
they have actually never stopped malicious applications from gathering
information about what other applications run currently on your
device, but only removed access to detail information that is only
relevant for harmless system information tools anyway.
System and Security Info is therefore still able to show the list of
running processes and enriches this list with information from the
codesigning information including the list of entitlements running
processes have.
I am developping an app for Ipad (2 and 3).
The app needs a continuous synchronization with a remote web app. Point is, my app needs to keep "listening" whenever there is connectivity in order to receive updates (lots of data), alerts,... etc.
PS: Before going further I would like to highlight that my app is not targeted to the appstore and moreover it can't rely on push notifications.
What I am confused about is the multitasking on the ipad. I would like my app's local db to stay synchronized by communicating with the remote web app. And hence my questions are:
Does any of the ipads have a real multitasking? (like android's where you can have "services") Meaning that my app or at least a small part of it would still operate even if it is not active on the screen.
How can I achieve that using phonegap standard framework?
If the above points show that at some extent it's not possible. Is there still a possibility to extend phonegap with a plugin and make this happen?
If the above three points are unfortunately fully negative; how would you address the problem by keeping in mind that the syncrhonization involves a lot of data and is mandatory to cover the app's usability?
Does any of the ipads have a real multitasking? (like android's where
you can have "services") Meaning that my app or at least a small part
of it would still operate even if it is not active on the screen.
IOS only has limited background services - in that your app goes into a background state when not active. Really only limited to receiving push notifications. As Phonegap uses a webview and javascript that is only active when the app is active in the foreground.
How can I achieve that using phonegap standard framework?
If you want to keep data connection open you might have to look at development of an application for a computer tablet, rather then an IOS device.
If the above points show that at some extent it's not possible. Is
there still a possibility to extend phonegap with a plugin and make
this happen?
No.
If the above three points are unfortunately fully negative; how would
you address the problem by keeping in mind that the syncrhonization
involves a lot of data and is mandatory to cover the app's usability?
You only option (if you can only use phonegap) is to have the app active all the time, and set a constant sync using Jquery & XML/JSON data. Of course this depends on what type of data you are going to show. For showing a page of data - similar to a webpage you won't have any problems - as long as you understand once you change apps that data won't be synced.
I would like my app's local db to stay synchronized by communicating with the remote web app.
If I were you, I'd be questioning why would I need to do that? Why not just make the app works online? You can have all the assets locally, and just setup some sort of API to retrieve whatever data you need to display on demand.
Otherwise, you'll need to hack your way through to do all the things you wanted, plus you need offline / online syncing. Since you have most data available offline, I assume the app also works offline? And you'll have a really bad time working on offline / online syncing...
I am working on a site for mobile devices. The site is available through normal web browsers and also through an app which is just a browser shell and brings up the mobile site. In our efforts to speed up the loading of the site in mobile we have reduced requsts, made use of data uris, etc. Recently we have started using localStorage to save styles and JavaScript data to the device.
Why you may ask?
In our testing, mobile browsers maintain their cache throughout their session and when the browser is closed and re-opened. The app maintains its cache as long as it is being used, but when it is closed and re-opened it re-requests everything, thus slowing down that initial load.
The problem is, we have styles and JavaScript that are specific for if you are in the browser or in the app for a few small things. We've seen a few things break around these subtle differences and my best theory is that localStorage is shared between the browser and the app. And a user that uses both the site and the app may have problems if the localStorage was set by one and needs something else for the other.
I can't find any documentation that confirms this theory or not, and short of creating an app just to test this I figured I'd ask if anyone has any ideas?
If you trust Apple...
Like cookies, storage objects are a shared resource common to web
content served from the same domain. All pages from the same domain
share the same local storage object. Frames and inline frames whose
contents are from the same origin also share the same session storage
object because they descend from the same window.
Because the storage resource is shared, scripts running in multiple
page contexts can potentially modify the data stored in a storage
object that is actively being scrutinized or modified by a script
running on a different page. If your scripts do not notice these
changes, you may not get the results you expect.
If you are populating your app with data from the same place as the web app, I would suspect there are some keys being modified by the other one. I know that using sessionStorage.clear() will wipe out keys if the web app and offline app load data from the same domain.
As Chiguireitor said it depends on what mobile OS the user is using, but in my experience iOS 4 & 5 share the same localStorage whether you're accessing the mobile app through the Safari browser or as a homescreen web app. And of course if you package it with something like Phonegap it acts as its own app therefore its localStorage is not shared.