Javascript variable VS Object storage for large data script execution - javascript

I'm actually running a JS script (ES6 and Guzzle) in the browser (it have to run in a browser, no NodeJS).
This script is calling some xml files, and store results for later usage (I output the convert and output then process it to be able to import it in a database).
So this script will generate an array containing thousands and thousands of small JS object (from XML parsing).
As the script take really long to run, I'm looping on my URL array (I have a list of all file URL), and storing query result into a classic JS variable, and local storage after jsonEncode. As it's JSON-encoded, the localStorage value is erased every time and a new bigger string is saved for the same key.
My question :
Is it better to use only a classic variable? Or only the local storage?
Is there any other way to store a large amount of data for a script? (temporary blob, text file, DOM append...)
From my tests, after 3-4k files queried and result stored, the browser starts to slow down a lot and drastically reduce the number of HTTP request/minutes.
Thanks !
Notes :
It have to run in a browser (I need some dynamic DOM data, it's an internal dashboard that display stats, with user inputs for live settings).
It need to run only on latest Chrome or Firefox

the localStorage value is erased every time and a new bigger string is saved for the same key.
This deserialization-append-serialization process is what slows down the page. Instead you could store each entry in its own key, that way appending is much more performant:
class PersistentArray {
constructor(name) {
this.name = name;
this.length = +localStorage.getItem(name) || 0;
}
push(value) {
set(this.length, value);
}
set(index, value) {
if(index >= this.length)
localStorage.setItem(this.name, this.length = index + 1);
localStorage.setItem(this.name + index, JSON.stringify(value));
}
get(index) {
return JSON.parse(localStorage.getItem(this.name + index));
}
*[Symbol.iterator] () {
for(let i = 0; i < this.length; i++)
yield this.get(i);
}
}
That way you can easily push values as:
const pages = new PersistentArray("pages");
// ... somewhen
pages.push({ value: "whatever" });
When all the data is there:
// Turn into a real in memoy array:
const result = [...pages];
// Dynamically load:
for(const page of pages)
console.log(page);

Related

Project works fine on localhost but the build gets an issue

I am working on a Preact-CLI project with a Preact-Router and it works fine on the localhost. But the production doesn't work well after the build.
I have created a one page object which gets its content dynamically from a JSON file (inside the project not external). So I've loaded the same page object 2 times for each different page.
I get the page url (using this.props.permalink) and compare it with the JSONObject.title. If they are the same I want to get the corresponding JSON content to display it on the corrrct page. Works like a charm on localhost, but not in production.
Issue:
Somehow all pages get the content of the first JSON element. First I thought it was a server issue but I was wrong. The builded files are wrong after the prerendering/build. So the prerendered html of page B contains the content of the prerendered page A.
My guess is that during the build this.props.permalink doesn't work. How should I handle this?
Additional info:
I use the prerender function but not the service worker for the build.
Thanks!
UPDATE:
I have rewritten the function. I guessed I needed to set the dynamic content through a loop, so that during the build the compiler loops through it and is able to prerender all the pages.
The iteration and setting the state works, but only the final element of the PrerenderUrls array gets stored. So now all pages gets the JSON content of the first element.
componentWillMount() {
for (var i = 0; i <= PrerenderUrls.length; i++) {
// the code you're looking for
let removeDash = new RegExp("-")
var needle = PrerenderUrls[i].title
var needle1 = needle.replace(removeDash, " ")
alert("1")
// iterate over each element in the array
if (needle1 != "Homepage") {
for (var x = 0; x < Data.length; x++) {
// look for the entry with a matching `code` value
let removeDash = new RegExp("-")
var nodash = Data[x].title.replace(removeDash, " ")
var nocaps = nodash.toLowerCase()
if (nocaps == needle1) {
alert("needle2: "+ needle1 + " nocaps: " + nocaps)
//alert("data "+ Data[x].title)
this.setState({
pageTitle: Data[x].title,
descShort: Data[x].descShort,
description: Data[x].desc,
img: Data[x].img
})
alert("state "+ this.state.pageTitle)
}
}
}
}
From your description it seems you have a standard Javascript closure problem. I noticed you use both let and var. If let is supported, use it instead of var. It will automagically solve your closure issues, because let creates variables with the block scope, instead of a function scope. Otherwise, you can try to replicate how let does it under the hood - throw the variable to the callback function. Something in the lines of:
// ...
for (var x = 0; x < Data.length; x++) {
try { throw x }
catch(iterator) {
this.setState({
pageTitle: Data[iterator].title
});
}
}
PS. It is very difficult to follow your code, when it is so specific to your functionality. You could simplify it, and focus on the troubling issue. Most of the code you provided is not relevant to your problem, but makes us going through it anyway.

Iterate through multiple web pages in JS with sleep in between

I was checking some simple solutions for showing multiple web pages for some dashboard and currently fighting with simple HTML page with javascript inside to achieve what I want to see there.
var urls = new Array();
urls[0] = "https://stackoverflow.com/"
urls[1] = "https://www.google.com"
var arrayLength = urls.length;
for (var i = 0; i < arrayLength; i++) {
window.location.assign(urls[i]);
sleep(3000);
}
function sleep(milliseconds) {
var start = new Date().getTime();
for (var i = 0; i < 1e7; i++) {
if ((new Date().getTime() - start) > milliseconds) {
break;
}
}
}
Currently this page opens only first page (after some time) and looks like it doesn't do iteration trough other pages. Maybe you could help me to make it work? I want to rotate those pages forever on screen (will add some infinite while loop after making this part working).
Currently this page opens only first page (after some time) and looks
like it doesn't do iteration trough other pages.
Once you change your window.location, and go to the first url from the array, you are losing all of your JS code (as it is not present in just opened url any more).
You can do this by installing a chrome plugin (which will not lose your JS after window.location change).
The plugin will run the added JS at DOMContentLoaded (no need to attach any event listener).
I needed also to do this, check things on the page, store some information and move on to the next page. I know, this can be done with Python and other stuff but by doing this it can be done on the FE side also.
I used the localStorage to store my information.
I pasted this into the browser console to prepare all the stuff and clean the localStorage:
// clear the localStorage
localStorage.clear();
// set an array that will keep all our pages to iterate into the localStorage
localStorage.setItem(
"pages",
JSON.stringify([
"https://my-page-1.html",
"https://my-page-2.html",
"https://my-page-3.html",
"https://my-page-4.html",
])
);
// set an array that will keep our findings
localStorage.setItem("resultArray", JSON.stringify([]));
// move to the first page of the iteration
window.location.href = "https://my-page-1.html";
After doing this, I opened the plugin interface and added the following code:
(function check() {
// array saved into the localStorage that contains all the pages to iterate
const pagesArray = JSON.parse(localStorage.getItem("pages"));
// array to store your stuff
const resultArray = JSON.parse(localStorage.getItem("resultArray"));
// whatever you want to check on that page
const myFancyCondition = true;
if (myFancyCondition) {
// push any data to the array so that you can check it later
resultArray.push({
page: pagesArray[0],
message: "I found what I was looking for!",
});
}
//remove the current page from the array
pagesArray.shift();
//reset the array value after the first page was already checked
localStorage.setItem("pages", JSON.stringify(pagesArray));
//store the array data
localStorage.setItem("resultArray", JSON.stringify(resultArray));
// quit if the iteration is over and there are no more pages to check
if(!pagesArray.length) return;
//go to the next page
window.location.href = pagesArray[0];
})();
Then, to check the results you just need to read the data from the localStorage like:
JSON.parse(localStorage.getItem('resultArray'))
I hope this helps :)!

Reading a subset of documents using Couchbase View

I have around 25M documents in my cluster. I need to read 1M documents at a time without any specific criterion. I don't have the access to the keys. So I need to create a view which will emit documents till I reach a counter which goes up to 1M.
I have written a Map function inside which I am trying to create a static variable, but JS doesn't support static variables. I am not sure how to do this operation. The map function which I have written is just to return 1000 documents and it is full of errors. Can someone help me with this functionality?
function (doc, meta) {
value = foo();
if(value < 1000)
{
emit(meta.id, null);
}else{
return;
}
}
function incrementor(){
if(typeof incrementor.counter == 'undefined'){
incrementor.counter = 0;
}
return ++incrementor.counter;
}
Reading a subset of documents with Views can be done using pagination: http://blog.couchbase.com/pagination-couchbase
The Map function is called for each mutation stored in the bucket so a counter approach like this does not make sense. If you want to split your indexes, you need to do that based on the content of the document. But you should really use pagination. This can also be achieved with N1Ql by the way.

Remember javascript variable across more than one page

I have a database and 2 drop down menus, using javascript I obtain the value from the selected drop down, and then send it to my PHP, the PHP then brings back information from the database and displays it in a table, there are two tables as there are 2 drop downs. Here is my javascript:
function choice1()
{
var x = document.getElementById("select1");
a = x.options[x.selectedIndex].value;
window.location.href = "index.php?a=" + a + "&b=" + b;
}
function choice2()
{
var y = document.getElementById("select2");
b = (y.options[y.selectedIndex].value);
window.location.href = "index.php?a=" + a + "&b=" + b;
}
what happens is it waits for both drop downs to change before changing both of the tables, what I would like it to do is change the table as soon as one changes but keep the other one the same. This I think means the javascript variable a or b needs to be stored so that when the page changes it can be called upon so that the PHP gives the same information for the second table.
You can persist data in many ways:
Examples are:
cookies: Javascript getCookie functions and
https://developer.mozilla.org/en-US/docs/Web/API/document.cookie
localStorage, get variables, hidden input.
I recommend using the localStorage for this case (html5), because it's pretty safe and easy to use.
// getter function of 'persistent' variables.
function getPersistent(myVar) {
// ensure the localStorage object exists.
// return the variable we are looking for if it does or: undefined.
return (window.localStorage)? localStorage.getItem(myVar) : undefined;
}
// setter function of 'persistent' variables.
function setPersistent(myVar, value) {
// ensure the localStorage object exists.
if (window.localStorage) {
// set the variable.
localStorage.setItem(myVar, value);
}
}
// first run (page refresh) returns undefined.
// second run returns 4.
console.log(getPersistent('test'));
// we set the localStorage var 'test' to '4'.
// note that localStorage only saves strings.
// so you need to parse/convert the data if you want to modify.
console.log(setPersistent('test', '4'));
// returns localStorage var 'test' ==> 4.
console.log(getPersistent('test'));
fiddle: http://jsfiddle.net/kychan/2WJX4/
The simplest way to persist values across pages is to use the session storage API. It's much like local storage, but is torn down when the current window / session closes.
Of course, you could always just opt for updating your page by AJAX rather than reloading the document entire,

Mainting the state of array across the entire app using singleton pattern

function MySingletonClass(arg) {
this.arr = [];
if ( arguments.callee._singletonInstance )
return arguments.callee._singletonInstance;
arguments.callee._singletonInstance = this;
this.Foo = function() {
this.arr.push(arg);
// ...
}
}
var a = new MySingletonClass()
var b = MySingletonClass()
Print( a === b ); // prints: true
My requirement is i am pushing objects to an array on each load of window, but when i open the next window the state of the array is not visible.
var arr = [];
arr.push("something");
// It gets pushed.
When i open the new window, the array's length becomes zero again.
There is no way to do this with JavaScript alone. JavaScript is just the language. It doesn't have any direct link to the app, the page or even the browser. JavaScript can be used (and is used) in many other situations, such as in server-side applications and as a plugin language for desktop apps.
Of course, when JavaScript is used in the browser, you do need a way to "communicate", as it were, with the content on page. For this you can use the Document Object Model (DOM) API, which is implemented by every browser that supports JavaScript. To communicate with the browser itself you can use window and other global object. These are sometimes referred to as the Browser Object Model (although it's not an official API).
Now that we know that; is there an API that allows us to maintain state between pages? Yes, there is. In fact, there are several:
HTML5's localStorage
Cookies
Take this example, using localStorage:
// On page 1:
localStorage.setItem("message", "Hello World!");
// On page 2:
var message = localStorage.getItem("message");
if (message !== null) {
alert(message);
}
Easy, right? Unfortunately, localStorage only accepts key/value pairs. To save an array, you'll need to convert it into a string first. You could do this, for example, using JSON:
// On both pages:
var arr = localStorage.getItem("arr");
if (arr === null) {
arr = [];
} else {
arr = JSON.parse(arr);
}
function saveArr() {
localStorage.setItem("arr", JSON.stringify(arr));
}
// On page 1:
console.log(arr); // []
arr.push("Hello");
arr.push("world!");
saveArr();
// On page 2:
console.log(arr); // ["Hello", "world!"]
Keep in mind, though, that localStorage and JSON are both fairly new, so only modern browsers support them. Have a look at emulating localStorage using cookies and at JSON2.js.
For data to persist across an application, there must be a database. Javascript cannot accomplish this because it is client side only and mostly intended as a way to render user interfaces.

Categories