We Keep Coding

How to increase browser localstorage size [duplicate]

I've written a webapp that allows you to store the images in the localStorage until you hit save (so it works offline, if signal is poor).
When the localStorage reaches 5MB Google Chrome produces an error in the javascript console log:
Uncaught Error: QUOTA_EXCEEDED_ERR: DOM Exception 22
How do I increase the size of the localStorage quota on Google Chrome?

5MB is a hard limit and that is stupid. IndexedDB gives you ~50MB which is more reasonable. To make it easier to use try Dexie.js https://github.com/dfahlander/Dexie.js
Update:
Dexie.js was actually still an overkill for my simple key-value purposes so I wrote this much simpler script https://github.com/DVLP/localStorageDB
with this you have 50MB and can get and set values like that
// Setting values
ldb.set('nameGoesHere', 'value goes here');
// Getting values - callback is required because the data is being retrieved asynchronously:
ldb.get('nameGoesHere', function (value) {
console.log('And the value is', value);
});
Copy/paste the line below so ldb.set() and ldb.get() from the example above will become available.
!function(){function e(t,o){return n?void(n.transaction("s").objectStore("s").get(t).onsuccess=function(e){var t=e.target.result&&e.target.result.v||null;o(t)}):void setTimeout(function(){e(t,o)},100)}var t=window.indexedDB||window.mozIndexedDB||window.webkitIndexedDB||window.msIndexedDB;if(!t)return void console.error("indexDB not supported");var n,o={k:"",v:""},r=t.open("d2",1);r.onsuccess=function(e){n=this.result},r.onerror=function(e){console.error("indexedDB request error"),console.log(e)},r.onupgradeneeded=function(e){n=null;var t=e.target.result.createObjectStore("s",{keyPath:"k"});t.transaction.oncomplete=function(e){n=e.target.db}},window.ldb={get:e,set:function(e,t){o.k=e,o.v=t,n.transaction("s","readwrite").objectStore("s").put(o)}}}();

You can't, it's hard-wired at 5MB. This is a design decision by the Chrome developers.
In Chrome, the Web SQL db and cache manifest also have low limits by default, but if you package the app for the Chrome App Store you can increase them.
See also Managing HTML5 Offline Storage - Google Chrome.

The quota is for the user to set, how much space he wishes to allow to each website.
Therefore since the purpose is to restrict the web pages, the web pages cannot change the restriction.
If storage is low, you can prompt the user to increase local storage.
To find out if storage is low, you could probe the local storage size by saving an object then deleting it.

You can't but if you save JSON in your localStorage you can use a library to compress data like : https://github.com/k-yak/JJLC
demo : http://k-yak.github.io/JJLC/

Here you can test your program , you should handle also the cases when the cuota is exceed

https://stackoverflow.com/a/5664344/2630686 The above answer is much amazing. I applied it in my project and implement a full solution to request all kinds of resource.
// Firstly reference the above ldb code in the answer I mentioned.
export function get_file({ url, d3, name, enable_request = false }) {
if (name === undefined) { // set saved data name by url parsing alternatively
name = url.split('?')[0].split('/').at(-1).split('.')[0];
}
const html_name = location.href.split('/').at(-1).split('.')[0]
name = `${html_name}_${name}`
let ret = null;
const is_outer = is_outer_net(url); // check outer net url by its start with http or //
// try to access data from local. Return null if not found
if (is_outer && !enable_request) {
if (localStorage[name]) {
ret = new Promise(resolve => resolve(JSON.parse(localStorage[name])));
} else {
ret = new Promise(r => {
ldb.get(name, function (value) {
r(value)
})
});
}
} else {
ret = new Promise(r => r(null))
}
ret.then(data => {
if (data) {
return data
} else {
const method = url.split('.').at(-1)
// d3 method supported
if (d3 && d3[method]) {
ret = d3[method](url)
} else {
if (url.startsWith('~/')) { // local files accessed supported. You need a local service that can return local file data by requested url's address value
url = `http://localhost:8010/get_file?address=${url}`
}
ret = fetch(url).then(data => {
// parse data by requested data type
if (url.endsWith('txt')) {
return data.text()
} else {
return data.json()
}
})
}
ret = ret.then(da => {
data = da
if (is_outer) { // save data to localStorage firstly
localStorage[name] = JSON.stringify(data);
}
}).catch(e => { // save to ldb if 5MB exceed
ldb.set(name, data);
}).finally(_ => {
return data;
});
}
})
return ret;
}

Fetching api response slow and not be stable

I need to get the value (totaluser) as soon as possible , but when the server change the data (totaluser) , the reponse is not stable about the time returning , sometimes get the reponse for 30ms and sometimes get for 5000ms , can someone give any solution to get this more stable and get better about the performance ? , i just want to get the value as soon as possible when the server update it?
var refreshIntervalId = setInterval(function () {
fetch('**the link for api **')
.then(res => res.json()).then((out) => {
var variable = out.totalUser[0].count;
if( variable > 20 ){
// do something ...
}
})
}, 10);

Javascript: append to an array not working

I am trying to append numbers that I get from an api call (a promise) into an array. When I test the array's length it's always returning 1 as if each api call resets the array and puts in a new number.
here's the code:
The API call
wiki()
.page("COVID-19_pandemic_in_Algeria")
.then((page) => page.fullInfo())
.then((info) => {
(data.confirmed.value = info.general.confirmedCases),
(data.recovered.value = info.general.recoveryCases),
(data.deaths.value = info.general.deaths);
});
const data = {
confirmed: { value: 0 },
deaths: { value: 0 },
recovered: { value: 0 },
};
Now I want to put the deaths count into an array, so that I have a list of numbers over the next days to keep track of.
function countStats() {
const counter = [];
var deathCounter = data.deaths.value;
counter.push(deathCounter);
console.log(counter.length);
return counter;
}
countStats();
every time the functions run (wiki() and countStats()) the counter array's length is always 1. Why is that?

Unless ...
the data source provides multi-day data, or
you are going to run an extremely long javascript session (which is impractical and unsafe),
... then javascript can't, on its own, meet the objective of processing/displaying data arising from multiple days'.
Let's assume that the data source provides data that is correct for the current day.
You will need a permanent data store, in which scraped data can be accumulated, and retreived on demand. Exactly what you choose for your permanent data store is dependant on the environment in which you propose to run your javascript (essentially client-side browser or server-side NODE), and that choice is beyond the scope of this question.
Your master function might be something like this ...
function fetchCurrentDataAndRenderAll() {
return fetchCurrentData()
.then(writeToFile)
.then(readAllFromFile)
.then(data => {
// Here, you have the multi-day data that you want.
return renderData(data); // let's assume the data is to be rendered, say as a graph.
})
.catch(error => {
// something went wrong
console.log(error);
throw error;
});
}
... and the supporting functions might be something like this:
function fetchCurrentData() {
return wiki() // as given in the question ...
.page("COVID-19_pandemic_in_Algeria")
.then(page => page.fullInfo())
.then(info => ({
'timeStamp': Date.now(), // you will most likely need to timestamp the data
'confirmed': info.general.confirmedCases,
'recovered': info.general.recoveryCases,
'deaths': info.general.deaths
}));
}
function writeToFile(scrapedData) {
// you need to write this ...
// return Promise.
}
function readAllFromFile() {
// you need to write this ...
// return Promise.
}
function renderData(data) {
// you need to write this ...
// optionally: return Promise (necessary if rendering is asynchronous).
}

You can use Promise.all(). I take it that you'll not be requesting the same page 10 times but requesting a different page in each call e.g. const Pages = ['COVID-19_pandemic_in_Algeria','page2','page3','page4','page5','page6','page7','page8','page9','page10']. Then you could make the 10 calls as follows:
//const wiki = ......
const Pages = ['COVID-19_pandemic_in_Algeria','page2','page3','page4','page5','page6','page7','page8','page9','page10'];
let counter = [];
Promise.all(
Pages.map(Page => wiki().page(Page))
)
.then(results => {
for (page of results) {
let infoGeneral = page.fullInfo().general;
counter.push(infoGeneral.deaths);
}
console.log( counter.length ); //10
console.log( counter ); //[10 deaths results one for each page]
})
.catch(err => console.log(err.message));

How to edit request url in service worker?

I'm using cache first caching strategy for my pwa, for every GET request I first look if that request exists in cache, if it does I return it and update the cache.
The problem is that users can switch between multiple projects, so when they switch to another project,
the first time they open some url, they get the stuff from previous project if it exists in cache.
My solution is to try to add GET parametar ?project=projectId(project=2 for example) in the service worker, so each project would have its own version of the request saved in the cache.
I wanted to concatinate project id to the event.request.url, but I've read here that it is read only.
After doing that, hopefully I would have urls like this in cache:
Instead of: https://stackoverflow.com/questions
I would have: https://stackoverflow.com/questions?project=1
And: https://stackoverflow.com/questions?project=2
So I would get questions from the project I'm on, instead of just getting questions from previous project is /questions is saved in cache already.
Is there a way to edit request url in service worker?
My service worker code:
self.addEventListener('fetch', function(event) {
const url = new URL(event.request.clone().url);
if (event.request.clone().method === 'POST') {
// update project id in service worker when it's changed
if(url.pathname.indexOf('/project/') != -1 ) {
// update user data on project switch
let splitUrl = url.pathname.split('/');
if (splitUrl[2] && !isNaN(splitUrl[2])) {
console.log( user );
setTimeout(function() {
fetchUserData();
console.log( user );
}, 1000);
}
}
// do other unrelated stuff to post requests
.....
} else { // HANDLE GET REQUESTS
// ideally,here I would be able to do something like this:
if(user.project_id !== 'undefined') {
event.request.url = event.request.url + '?project=' + user.project_id;
}
event.respondWith(async function () {
const cache = await caches.open('CACHE_NAME')
const cachedResponsePromise = await cache.match(event.request.clone())
const networkResponsePromise = fetch(event.request.clone())
if (event.request.clone().url.startsWith(self.location.origin)) {
event.waitUntil(async function () {
const networkResponse = await networkResponsePromise.catch(function(err) {
console.log( 'CACHE' );
// return caches.match(event.request);
return caches.match(event.request).then(function(result) {
// If no match, result will be undefined
if (result) {
return result;
} else {
return caches.open('static_cache')
.then((cache) => {
return caches.match('/offline.html');
});
}
});
});
await cache.put(event.request.clone(), networkResponse.clone())
}())
}
// news and single photos should be network first
if (url.pathname.indexOf("news") > -1 || url.pathname.indexOf("/photos/") > -1) {
return networkResponsePromise || cachedResponsePromise;
}
return cachedResponsePromise || networkResponsePromise;
}())
}
});

It's possible to use any URL as a cache key when reading/writing to the Cache Storage API. When writing to the cache via put(), for instance, you can pass in a string representing the URL you'd like to use as the first parameter:
// You're currently using:
await cache.put(event.request.clone(), networkResponse.clone())
// Instead, you could use:
await cache.put(event.request.url + '?project=' + someProjectId, networkResponse.clone())
But I think a better approach that would accomplish what you're after is to use different cache names for each project, and then within each of those differently-named caches you would not have to worry about modifying the cache keys to avoid collisions.
// You're currently using:
const cache = await caches.open('CACHE_NAME')
// Instead, you could use:
const cache = await caches.open('CACHE_NAME' + someProjectId)
(I'm assuming that you have some reliable way of figuring out what the correct someProjectId value should be inside of the service worker, based on which client is making the incoming request.)

How to transfer data from one file to another?

I am having two files such as,
employee-rates-controller.ts:
private load() {
return this.entityService
.load(this.$scope.projectRevisionUid)
.then(resp => {
localStorage.removeItem('employeerates');
this.$scope.employeeRates = resp.employeeRates;
return this.refreshCostRate(...resp.employeeRates)
.then(() =>
localStorage.setItem(
'employeerates',
JSON.stringify(this.$scope.employeeRates)
)
)
.then(() => this.refreshBillRate(...resp.employeeRates))
.then(() => resp.employeeRates.forEach(erm => this.calculate(erm)))
.then(() => DatepickerUtil.reinitializeDatepickers(this.$scope));
})
}
And in another file,
getEmployeeRates.ts:
const employeerates = JSON.parse(
localStorage.getItem('employeerates')
);
if (employeerates && employeerates.length != null) {
employeerates.forEach((element: any) => {
if (
this.employee.getUid() === element.user.personUid &&
element.internalRate
) {
this.cost_rate_uom = element.internalRate * this.uom_factor;
this.cost_rate_per_hour =
this.cost_rate_uom / this.uom_factor;
this.cost_rate.setValue(this.ap4_cost_rate_per_hour);
}
});
}
Here you can see,
In first ts file,
localStorage.setItem('employeerates',JSON.stringify(this.$scope.employeeRates))
And in second ts file receiving the data,
const employeerates = JSON.parse(localStorage.getItem('employeerates'));
I couldn't not find any problem if i add very few employees but when i keep on adding employee which means storing them into localstorage, i am getting the error at certain point of time when the data size was huge and it blocks the entire process.
The error was,
QuotaExceededError: Failed to execute 'setItem' on 'Storage': Setting
the value of 'employeerates' exceeded the quota.
So i would like to get some good solution in transferring any large data from one file to another without using the localstorage..
As the application is made in Angularjs and Typescript combination, i couldn't find out right solution as i am new to this scenario.
Edit:
Instead of first TS file, i am also able to get the value in this file.
employeeratemodel.ts:
export class EmployeeRateModel {
public uid: string;
.
.
.
public internalRate: number; // Getting the value here
}
How to fetch this value inside the second ts getEmployeeRates.ts: file..
My try:
import { EmployeeRateModel } from '../component/employee-rates/model/employee-rate.model';
constructor() {
const data = new EmployeeRateModel();
console.log(data) // {} // Gives empty object.. I need to fetch the internalRate from it..
}
Here if i get the data then it will be ease for me to get the internalRate which is needed for calculation, but as everything returns empty, this also fails for me..
Kindly help me to fix it in appropriate way, stucked for long..

use IndexedDB, it is a large-scale, NoSQL storage system. It lets you store just about anything in the user's browser and it has a huge limit based on the client computer approx 20% of total storage.
NPM Package Angular IndexedDB

Chrome's Local Storage default size is 10 Mb (https://en.wikipedia.org/wiki/Web_storage), so clearing your chrome's local storage could help you, if your data is larger than the limits, you may need to find alternatives like storing on blob storage and accessing the contents from blob itself.
For the reference, you can handle the error if you want using the code below.
try {
var counter = 1;
var stringData = "AddLocalStorageTillItIsFull";
for (var i = 0; i <= counter; counter + 1) {
stringData += stringData;
localStorage.setItem("localStorageData", stringData);
console.log(stringData);
console.log(counter);
}
}
catch (e) {
// When local storage is full, it goes hits this carch block
console.log("Local Storage is full, Please clear local storage data to add more");
}