We Keep Coding

Issues executing working JS with Scraping API: Not making complete XHR Calls

Trying to execute some JavaScript via a scraping API, with the goal of scrolling through dynamically loading pages, then parsing the full response. I tested the script in the console at an Auction site (with single page ticked) and it worked as expected.
//Keep scrolling until the previous doc height is equal to the new height
const infinite_scroll = async () => {
let previous = document.body.scrollHeight;
while (true){
window.scrollTo(0, document.body.scrollHeight);
// Fully Wait Until Page has loaded, then proceed
await new Promise(r => setTimeout(r, 2000));
let new_height = document.body.scrollHeight;
console.log('%s Previous Height: ',previous)
console.log(' ')
console.log('%s New Height: ',new_height)
if (new_height == previous){
break
}
else{
previous = new_height
}
}
};
infinite_scroll();
In the network tab it shows 11 successful XHR calls are made from
'372610?pn=6&ipp=10' to '372610?pn=16&ipp=10' ( 'pn', representing the equivalent to a page number and 'ipp', items per page)
However when I try to pass this script to Scrapfly, the API I'm using, It makes only 3 XHR calls and the last one times out, So instead of getting the entire page I get an extra 20 items.
Using the API Demo (would need to sign up for a free account!) It can be reproduced by adding the script, setting the Rendering time to 10000 ms, and adding the following headers;
emailcta : pagehits%3D1%26userdismissed%3Dfalse
cookie: UseInfiniteScroll=true
Looking at the details for the last XHR call, it times out and has a null response. The request headers for the call is identical to the two previous successful ones so I'm not exactly sure what the issue is.
JS_Rendering time doesn't seem to affect this or the wait time within the infinite scroll function.
All the docs say is:
"We provide a way to inject your javascript to be executed on the web page.
You must base64 your script.Your Javascript will be executed after the rendering delay and before the awaited selector (if defined)." They encode the script in base64 then execute it but the result is drastically different from the one in console

How to remove delay from pop-up of chrome extension/Firefox add-on?

I am building a bulk download extension from chrome and firefox to download a large number of files. The pop up of the extension has the cancel/pause/resume features. The pop up works fine when downloading a small number of files. However, when downloading a larger number of files, the pop-up menu takes forever to actually pop-up (or some time it doesn't whatsoever) probably because the extension becomes very busy and processing a lot of files at a time. Is there a way to eliminate this delay?
manifest.json
"default_popup": "html/popup.html"
},
popup.html
let bgPage = chrome.extension.getBackgroundPage(); //Getting the variables from the background page
let idsOfDownload = [];
idsOfDownload = bgPage.downloadIds;
console.log(idsOfDownload);
let cancel = document.createElement("button");
cancel.id = "cancel";
cancel.className = "btn";
cancel.innerHTML='<i class="fa fa-stop-circle"></i> Cancel All</button>';
document.body.appendChild(cancel);
$('body').width(350);
setInterval(function(){
let downloadString = LZString.decompress(localStorage.getItem('downloadLinks'));
if(downloadString === "") return;
let downloadLinks = JSON.parse(downloadString);
if (downloadLinks !== undefined && downloadLinks !== null && downloadLinks !== "null") {
let status = `Total pending download: ${downloadLinks.length}`;
jQuery("#download-status").html(status);
}
},1000);
$(cancel).click(function () {
chrome.runtime.sendMessage({message: "cancel-download"});
});

Generally DOM is slow so if you have a lot of elements you should only display the visible stuff and draw the rest on demand when scrolling. There are many libraries and frameworks for this.
LZString can be very slow with big strings so consider not using it or do it inside a web worker and pass the data via standard DOM messaging.
window.localStorage is synchronous so it can be slow too if the data is big, consider switching to asynchronous storage like chrome.storage.local or IndexedDB.
Most importantly, use devtools profiler instead of guessing. After analyzing the results you'll see what needs fixing/rewriting. The popup has its own devtools (see how to open it).

In my case, the pop up was slow because I was using synchronous code blocks like looping over array with tens of thousand links and as #wOxxOm pointed out, window.localStorage, which ended up blocking the event loop.
Replacing for loop with setInterval( //doSomething, 0) and window.localStorage with chrome.storage.local worked in my case.

location.reload(true) crashing browser tab

I have a website that uses PHP sessions, and I have implemented the following JS code to check every 60 seconds if a user's sessions is still active:
var timeoutInterval = 60000; // 1 minute
function checkTimeout() {
var timeoutWorker = new Worker("/include/cbpull.js");
timeoutWorker.postMessage('/cloud/timeout.php');
timeoutWorker.onmessage = function (result) {
if (result.data['result'] === false) {
location.reload(true);
}
}
}
function sessionTimeout() {
checkTimeout();
setInterval(checkTimeout, timeoutInterval);
}
sessionTimeout();
However, this code crashes the tab in Google Chrome when the session is timed out and location.reload(true) is called. What can I do to make the code work correctly?

Might the following be what's happening? On a session time-out, you reload the page, which immediately triggers sessionTimeout again, which again finds that the session is (still) expired, which reloads the page...

JavaScript notifications not working as expected in function

I recently made a YouTube JukeBox. I have everything I want done, except I am trying to get Notifications to work/show on the play event only if the previous video has ended.
I have tried setting a variable then updating it when the video when a video ends to true, however it keeps returning "undefined" when I check it in the play event.
function onPlayerStateChange(event)
{
var videoData = player.getVideoData();
if (event.data == 0)
{
var videend = 1;
player.loadVideoById($("input[name=video]:radio:checked").closest('tr').next().find('input').prop("checked", true).val());
$('.playlist').scrollTo($("input[name=video]:radio:checked").closest('tr'));
}
if (event.data == 1)
{
$("#info").css("display","none");
if(videend == 1)
{
var videend = 0;
new Notification(videoData['author'], {
dir: "auto",
lang: "",
body: videoData['title'],
icon: "http://vman315.com/base/images/Pinkie-Pie-mustache.png",
});
}
document.title = '\u25B6 ' + videoData['title'];
}
}
Example
If anyone could possibly provide some insight into what I am doing wrong I would be very grateful.

I got this to work for me in Google Chrome Version 29.0.1547.66 m
It appears as though notifications in Chrome are restricted from running automatically and you have to go into your web browser and have them set to either, allow notifications from all websites without asking, or ask each time for a notification.
My browser was set to ask each time for allowing a notification. You do not however have any code in your document to ask for permission to run a notification. When i set my browser to run notifications without asking then it worked properly.
The only other issue being your
"var videend = 1;"
was not being accessed inside of the local function. I moved that inside of the function and then your if statement for notifications started working.
http://jsfiddle.net/83tju/3/
Helpful resources
https://developer.mozilla.org/en-US/docs/Web/API/notification
http://www.w3.org/TR/notifications/#permission
After reading those resources it appears as though the syntax you are using to create notifications does not work on the version of chrome i am using, this might explain why i had to stop my browser from asking for notifications? Not sure. I will have to look at this later on an updated browser.

<video>.currentTIme doesn't want to be set

I'm trying to write a piece of Javascript that switches between two videos at timed intervals (don't ask). To make matters worse, each video has to start at specific place (about ten seconds, and again, don't ask.)
I got the basics working by just using the YUI Async library to fire to switch the videos at intervals:
YUI().use('async-queue', function (Y) {
// AsyncQueue is available and ready for use.
var cumulativeTime = 0;
var q = new Y.AsyncQueue()
for (var x = 0; x < settings.length; x++) {
cumulativeTime = cumulativeTime + (settings[x].step * 1000)
q.add( {
fn: runVideo,
args: settings[x].mainWindow,
timeout: cumulativeTime
})
}
q.run()
});
So far, so good. The problem is that I can't seem to get the video to start at ten seconds in.
I'm using this code to do it:
function runVideo(videoToPlay) {
console.log('We are going to play -> ' + videoToPlay)
var video = document.getElementById('mainWindow')
video.src = '/video?id=' + videoToPlay
video.addEventListener('loadedmetadata', function() {
this.currentTime = 10 // <-- Offending line!
this.play();
})
}
The problem is that this.currentTime refuses to hold any value I set it to. I'm running it through Chrome (the file is served from Google Storage behind a Google App Engine Instance) and when the debugger goes past the line, the value is always zero.
Is there some trick I'm missing in order to set this value?
Thanks in advance.

Try use Apache server.
setCurrentTime not working with some simple server.
ex) python built in server, php built in server
HTTP server should be Support partial content response. (HTTP Status 206)