We Keep Coding

How to increase browser localstorage size [duplicate]

I've written a webapp that allows you to store the images in the localStorage until you hit save (so it works offline, if signal is poor).
When the localStorage reaches 5MB Google Chrome produces an error in the javascript console log:
Uncaught Error: QUOTA_EXCEEDED_ERR: DOM Exception 22
How do I increase the size of the localStorage quota on Google Chrome?

5MB is a hard limit and that is stupid. IndexedDB gives you ~50MB which is more reasonable. To make it easier to use try Dexie.js https://github.com/dfahlander/Dexie.js
Update:
Dexie.js was actually still an overkill for my simple key-value purposes so I wrote this much simpler script https://github.com/DVLP/localStorageDB
with this you have 50MB and can get and set values like that
// Setting values
ldb.set('nameGoesHere', 'value goes here');
// Getting values - callback is required because the data is being retrieved asynchronously:
ldb.get('nameGoesHere', function (value) {
console.log('And the value is', value);
});
Copy/paste the line below so ldb.set() and ldb.get() from the example above will become available.
!function(){function e(t,o){return n?void(n.transaction("s").objectStore("s").get(t).onsuccess=function(e){var t=e.target.result&&e.target.result.v||null;o(t)}):void setTimeout(function(){e(t,o)},100)}var t=window.indexedDB||window.mozIndexedDB||window.webkitIndexedDB||window.msIndexedDB;if(!t)return void console.error("indexDB not supported");var n,o={k:"",v:""},r=t.open("d2",1);r.onsuccess=function(e){n=this.result},r.onerror=function(e){console.error("indexedDB request error"),console.log(e)},r.onupgradeneeded=function(e){n=null;var t=e.target.result.createObjectStore("s",{keyPath:"k"});t.transaction.oncomplete=function(e){n=e.target.db}},window.ldb={get:e,set:function(e,t){o.k=e,o.v=t,n.transaction("s","readwrite").objectStore("s").put(o)}}}();

You can't, it's hard-wired at 5MB. This is a design decision by the Chrome developers.
In Chrome, the Web SQL db and cache manifest also have low limits by default, but if you package the app for the Chrome App Store you can increase them.
See also Managing HTML5 Offline Storage - Google Chrome.

The quota is for the user to set, how much space he wishes to allow to each website.
Therefore since the purpose is to restrict the web pages, the web pages cannot change the restriction.
If storage is low, you can prompt the user to increase local storage.
To find out if storage is low, you could probe the local storage size by saving an object then deleting it.

You can't but if you save JSON in your localStorage you can use a library to compress data like : https://github.com/k-yak/JJLC
demo : http://k-yak.github.io/JJLC/

Here you can test your program , you should handle also the cases when the cuota is exceed

https://stackoverflow.com/a/5664344/2630686 The above answer is much amazing. I applied it in my project and implement a full solution to request all kinds of resource.
// Firstly reference the above ldb code in the answer I mentioned.
export function get_file({ url, d3, name, enable_request = false }) {
if (name === undefined) { // set saved data name by url parsing alternatively
name = url.split('?')[0].split('/').at(-1).split('.')[0];
}
const html_name = location.href.split('/').at(-1).split('.')[0]
name = `${html_name}_${name}`
let ret = null;
const is_outer = is_outer_net(url); // check outer net url by its start with http or //
// try to access data from local. Return null if not found
if (is_outer && !enable_request) {
if (localStorage[name]) {
ret = new Promise(resolve => resolve(JSON.parse(localStorage[name])));
} else {
ret = new Promise(r => {
ldb.get(name, function (value) {
r(value)
})
});
}
} else {
ret = new Promise(r => r(null))
}
ret.then(data => {
if (data) {
return data
} else {
const method = url.split('.').at(-1)
// d3 method supported
if (d3 && d3[method]) {
ret = d3[method](url)
} else {
if (url.startsWith('~/')) { // local files accessed supported. You need a local service that can return local file data by requested url's address value
url = `http://localhost:8010/get_file?address=${url}`
}
ret = fetch(url).then(data => {
// parse data by requested data type
if (url.endsWith('txt')) {
return data.text()
} else {
return data.json()
}
})
}
ret = ret.then(da => {
data = da
if (is_outer) { // save data to localStorage firstly
localStorage[name] = JSON.stringify(data);
}
}).catch(e => { // save to ldb if 5MB exceed
ldb.set(name, data);
}).finally(_ => {
return data;
});
}
})
return ret;
}

How to use local storage to store data from an API

I want to get data from an API only once in a while(say, once every hour) and store it locally and use that data on my website without having to call that api again and again everytime the person refreshes the browser. How can we achieve this. Can we use localStorage for that purpose. If yes then how?
I am using this:
fetch("https://coronavirus-tracker-api.herokuapp.com/deaths")
.then(res=>{
return res.json();
})
.then(data=>{
localStorage.setItem("data",JSON.stringify(data));
console.log(localStorage.getItem("data"))
})
but this would call the api everytime the page reloads. But instead of calling to the api again and again I want to store data in the localstorage and get data to view on the page from there.

It depends actually on which quantity of data you want to store. Generally you prefers to use the localStorage when you need to deals with small amount of data.
Another alternative is also possible, it's the IndexedDB which is more compliant and allow you to store more data.
You can find the API here: https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API
You can follow also some tutorials about IndexedDB to see actually how it works.
Finally, you can find the localStorage vs. IndexedDB usage response here: https://softwareengineering.stackexchange.com/questions/219953/how-is-localstorage-different-from-indexeddb
But if you want to steal use the localStorage, then you can check before fetching your data if the key storage "data" is used :
const data = localStorage.getItem('data');
if (!data) {
// then fetch your data
}
Be careful, the localStorage always store Key Value pairs, and the value will always be a string. So if you want to deals with your value when you retrieved it, do not forget to JSON.parse(data)
Edit: Refresh data outdated when re-opening tab
To update your data every 3-4 hours, you can store the date when you fetched the data. You need to update a little but the treatment of your response in the promise results :
const getDataFromLocalStorage = () => {
const dataStringified = localStorage.getItem('data');
return data && JSON.parse(dataStringified) || null;
};
const areDataOutdated = (receivedAt) => {
if (!dataReceivedAt || isNaN(Date.parse(receivedAt)) {
return true;
}
// Basically, we take the actual date, and we removed 4 hours
const checkDate = new Date(new Date().getTime() - (60 * 60 * 4 * 1000));
// If the data received is lower than the checkDate, it means that data are outdated.
return new Date(receivedAt).getTime() < checkDate.getTime();
};
const data = getDataFromLocalStorage();
if (!data || areDataOutdated(data && data.receivedAt)) {
// then fetch your data
fetch("https://coronavirus-tracker-api.herokuapp.com/deaths")
.then(res=> {
// instead of storing directly your response, construct a object that will store also the date
localStorage.setItem("data", JSON.stringify({response: res.json(), receivedAt: new Date()}));
console.log(localStorage.getItem("data"))
})
}
Edit 2: Refresh the data while staying on page
const getDataFromLocalStorage = () => {
const dataStringified = localStorage.getItem('data');
return data && JSON.parse(dataStringified) || null;
};
const fetchData = () => {
fetch("https://coronavirus-tracker-api.herokuapp.com/deaths")
.then(res=> {
// instead of storing directly your response, construct a object that will store also the date
localStorage.setItem("data", JSON.stringify({response: res.json(), receivedAt: new Date()}));
console.log(localStorage.getItem("data"))
})
}
setInterval(() => {
fetchData();
}, 1000 * 60 * 60 * 4) // every 4 hours, we'll fetch the data.
const data = getDataFromLocalStorage();
if (!data) {
fetchData();
}
But you can combine also with the outdated data check from Edit 1.

How to edit request url in service worker?

I'm using cache first caching strategy for my pwa, for every GET request I first look if that request exists in cache, if it does I return it and update the cache.
The problem is that users can switch between multiple projects, so when they switch to another project,
the first time they open some url, they get the stuff from previous project if it exists in cache.
My solution is to try to add GET parametar ?project=projectId(project=2 for example) in the service worker, so each project would have its own version of the request saved in the cache.
I wanted to concatinate project id to the event.request.url, but I've read here that it is read only.
After doing that, hopefully I would have urls like this in cache:
Instead of: https://stackoverflow.com/questions
I would have: https://stackoverflow.com/questions?project=1
And: https://stackoverflow.com/questions?project=2
So I would get questions from the project I'm on, instead of just getting questions from previous project is /questions is saved in cache already.
Is there a way to edit request url in service worker?
My service worker code:
self.addEventListener('fetch', function(event) {
const url = new URL(event.request.clone().url);
if (event.request.clone().method === 'POST') {
// update project id in service worker when it's changed
if(url.pathname.indexOf('/project/') != -1 ) {
// update user data on project switch
let splitUrl = url.pathname.split('/');
if (splitUrl[2] && !isNaN(splitUrl[2])) {
console.log( user );
setTimeout(function() {
fetchUserData();
console.log( user );
}, 1000);
}
}
// do other unrelated stuff to post requests
.....
} else { // HANDLE GET REQUESTS
// ideally,here I would be able to do something like this:
if(user.project_id !== 'undefined') {
event.request.url = event.request.url + '?project=' + user.project_id;
}
event.respondWith(async function () {
const cache = await caches.open('CACHE_NAME')
const cachedResponsePromise = await cache.match(event.request.clone())
const networkResponsePromise = fetch(event.request.clone())
if (event.request.clone().url.startsWith(self.location.origin)) {
event.waitUntil(async function () {
const networkResponse = await networkResponsePromise.catch(function(err) {
console.log( 'CACHE' );
// return caches.match(event.request);
return caches.match(event.request).then(function(result) {
// If no match, result will be undefined
if (result) {
return result;
} else {
return caches.open('static_cache')
.then((cache) => {
return caches.match('/offline.html');
});
}
});
});
await cache.put(event.request.clone(), networkResponse.clone())
}())
}
// news and single photos should be network first
if (url.pathname.indexOf("news") > -1 || url.pathname.indexOf("/photos/") > -1) {
return networkResponsePromise || cachedResponsePromise;
}
return cachedResponsePromise || networkResponsePromise;
}())
}
});

It's possible to use any URL as a cache key when reading/writing to the Cache Storage API. When writing to the cache via put(), for instance, you can pass in a string representing the URL you'd like to use as the first parameter:
// You're currently using:
await cache.put(event.request.clone(), networkResponse.clone())
// Instead, you could use:
await cache.put(event.request.url + '?project=' + someProjectId, networkResponse.clone())
But I think a better approach that would accomplish what you're after is to use different cache names for each project, and then within each of those differently-named caches you would not have to worry about modifying the cache keys to avoid collisions.
// You're currently using:
const cache = await caches.open('CACHE_NAME')
// Instead, you could use:
const cache = await caches.open('CACHE_NAME' + someProjectId)
(I'm assuming that you have some reliable way of figuring out what the correct someProjectId value should be inside of the service worker, based on which client is making the incoming request.)

How to handle localstorage in asynchronous way?

In my typescript application, i am having two files say,
File 1 and File 2,
Whereas in File 1, i would like to store a value in localstorage like,
private load() {
return this.entityService
.load(this.$scope.projectRevisionUid)
.then(resp => {
localStorage.removeItem('employeerates');
this.$scope.employeeRates = resp.employeeRates;
return this.refreshCostRate(...resp.employeeRates)
.then(() =>
localStorage.setItem(
'employeerates',
JSON.stringify(this.$scope.employeeRates)
)
)
.then(() => this.refreshBillRate(...resp.employeeRates))
.then(() => resp.employeeRates.forEach(erm => this.calculate(erm)))
.then(() => DatepickerUtil.reinitializeDatepickers(this.$scope));
})
}
In File 2, i am having the following,
const employeerates = JSON.parse(
localStorage.getItem('employeerates')
);
if (employeerates && employeerates.length != null) {
employeerates.forEach((element: any) => {
if (
this.employee.getUid() === element.user.personUid &&
element.internalRate
) {
this.cost_rate_uom = element.internalRate * this.uom_factor;
this.cost_rate_per_hour =
this.cost_rate_uom / this.uom_factor;
this.cost_rate.setValue(this.ap4_cost_rate_per_hour);
}
});
}
Here setting localstorage in File 1 is asynchronous, i am unable to fetch the data at right time in File 2..
Kindly help me to achieve the result of getting localstorage in file 2 without using setTimeOut (because it doesnot solve my issue as i have already checked).
Please help me to pass localstorage value from one file to another which is async..
Update:
I couldn't get any other method of passing the data this.$scope.employeeRates from file 1 to file 2, for which i have used this localstorage method.. So after the async function this.refreshCostRate(...resp.employeeRates) i need to call the localstorage but before that itself my file 2 runs, but above the line this.refreshCostRate(...resp.employeeRates), if i set the localstorage then i am getting localstorage in file 2, but the case is after refresh function only i will get the exact value..
If you suggest any other way of passing data from one ts file to another ts file, then it would also be helpful.. Thing is after this.refreshCostRate(...resp.employeeRates) i will get the exact value for this.$scope.employeeRates which i need to send to file 2..

Yes we can achieve using storage change event listener
window.addEventListener("storage", ()=>{alert("D")});
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/onChanged

First, I'm not sure what framework you're using, though since I really don't know any frameworks that wouldn't help me anyway. What I think might work is for the "file 1" code to expose the Promise object created each time that rate table needs to be updated. It's generally considered "bad" to have global variables, and indeed it only has to be globally reachable. As long as code running anywhere in the page can find it, that's good enough; for the example I'll use a global.
So in "file 1" you have:
localStorage.removeItem('employeerates');
this.$scope.employeeRates = resp.employeeRates;
return this.refreshCostRate(...resp.employeeRates)
.then(() =>
localStorage.setItem(
'employeerates',
JSON.stringify(this.$scope.employeeRates)
)
)
That code clears the rate table storage and starts the process to get new rates. What I suggest is to store the Promise in the global:
localStorage.removeItem('employeerates');
this.$scope.employeeRates = resp.employeeRates;
window.employeeRatesPromise = this.refreshCostRate(...resp.employeeRates)
.then(() =>
localStorage.setItem(
'employeerates',
JSON.stringify(this.$scope.employeeRates)
)
);
return window.employeeRatesPromise;
Now, in "file2", you can do everything as a .then() callback:
if (window.employeeRatesPromise) {
window.employeeRatesPromise.then(() => {
const employeerates = JSON.parse(
localStorage.getItem('employeerates')
);
if (employeerates && employeerates.length != null) {
employeerates.forEach((element: any) => {
if (
this.employee.getUid() === element.user.personUid &&
element.internalRate
) {
this.cost_rate_uom = element.internalRate * this.uom_factor;
this.cost_rate_per_hour =
this.cost_rate_uom / this.uom_factor;
this.cost_rate.setValue(this.ap4_cost_rate_per_hour);
}
});
}
else {
// whatever makes sense when there's no data at all
}
If the rate table update has completed, then the global Promise will be resolved and the function passed to .then() runs basically immediately. That can happen more than once on the same Promise object. However, when the "file 1" update process is still pending, the "file 2" code will wait for the local storage to be updated.
}
}

Inserting into Collection after Promises in a Meteor Method

I'm using this Gumroad-API npm package in order to fetch data from an external service (Gumroad). Unfortunately, it seems to use a .then() construct which can get a little unwieldy as you will find out below:
This is my meteor method:
Meteor.methods({
fetchGumroadData: () => {
const Gumroad = Meteor.npmRequire('gumroad-api');
let gumroad = new Gumroad({ token: Meteor.settings.gumroadAccessKey });
let before = "2099-12-04";
let after = "2014-12-04";
let page = 1;
let sales = [];
// Recursively defined to continue fetching the next page if it exists
let doThisAfterResponse = (response) => {
sales.push(response.sales);
if (response.next_page_url) {
page = page + 1;
gumroad.listSales(after, before, page).then(doThisAfterResponse);
} else {
let finalArray = R.unnest(sales);
console.log('result array length: ' + finalArray.length);
Meteor.call('insertSales', finalArray);
console.log('FINISHED');
}
}
gumroad.listSales(after, before, page).then(doThisAfterResponse); // run
}
});
Since the NPM package exposes the Gumorad API using something like this:
gumroad.listSales(after, before, page).then(callback)
I decided to do it recursively in order to grab all pages of data.
Let me try to re-cap what is happening here:
The journey starts on the last line of the code shown above.
The initial page is fetched, and doThisAfterResponse() is run for the first time.
We first dump the returned data into our sales array, and then we check if the response has given us a link to the next page (as an indication as to whether or not we're on the final page).
If so, we increment our page count and we make the API call again with the same function to handle the response again.
If not, this means we're at our final page. Now it's time to format the data using R.unnest and finally insert the finalArray of data into our database.
But a funny thing happens here. The entire execution halts at the Meteor.call() and I don't even get an error output to the server logs.
I even tried switching out the Meteor.call() for a simple: Sales.insert({text: 'testing'}) but the exact same behaviour is observed.
What I really need to do is to fetch the information and then store it into the database on the server. How can I make that happen?
EDIT: Please also see this other (much more simplified) SO question I made:
Calling a Meteor Method inside a Promise Callback [Halting w/o Error]

I ended up ditching the NPM package and writing my own API call. I could never figure out how to make my call inside the .then(). Here's the code:
fetchGumroadData: () => {
let sales = [];
const fetchData = (page = 1) => {
let options = {
data: {
access_token: Meteor.settings.gumroadAccessKey,
before: '2099-12-04',
after: '2014-12-04',
page: page,
}
};
HTTP.call('GET', 'https://api.gumroad.com/v2/sales', options, (err,res) => {
if (err) { // API call failed
console.log(err);
throw err;
} else { // API call successful
sales.push(...res.data.sales);
res.data.next_page_url ? fetchData(page + 1) : Meteor.call('addSalesFromAPI', sales);
}
});
};
fetchData(); // run the function to fetch data recursively
}