We Keep Coding

Issues executing working JS with Scraping API: Not making complete XHR Calls

Trying to execute some JavaScript via a scraping API, with the goal of scrolling through dynamically loading pages, then parsing the full response. I tested the script in the console at an Auction site (with single page ticked) and it worked as expected.
//Keep scrolling until the previous doc height is equal to the new height
const infinite_scroll = async () => {
let previous = document.body.scrollHeight;
while (true){
window.scrollTo(0, document.body.scrollHeight);
// Fully Wait Until Page has loaded, then proceed
await new Promise(r => setTimeout(r, 2000));
let new_height = document.body.scrollHeight;
console.log('%s Previous Height: ',previous)
console.log(' ')
console.log('%s New Height: ',new_height)
if (new_height == previous){
break
}
else{
previous = new_height
}
}
};
infinite_scroll();
In the network tab it shows 11 successful XHR calls are made from
'372610?pn=6&ipp=10' to '372610?pn=16&ipp=10' ( 'pn', representing the equivalent to a page number and 'ipp', items per page)
However when I try to pass this script to Scrapfly, the API I'm using, It makes only 3 XHR calls and the last one times out, So instead of getting the entire page I get an extra 20 items.
Using the API Demo (would need to sign up for a free account!) It can be reproduced by adding the script, setting the Rendering time to 10000 ms, and adding the following headers;
emailcta : pagehits%3D1%26userdismissed%3Dfalse
cookie: UseInfiniteScroll=true
Looking at the details for the last XHR call, it times out and has a null response. The request headers for the call is identical to the two previous successful ones so I'm not exactly sure what the issue is.
JS_Rendering time doesn't seem to affect this or the wait time within the infinite scroll function.
All the docs say is:
"We provide a way to inject your javascript to be executed on the web page.
You must base64 your script.Your Javascript will be executed after the rendering delay and before the awaited selector (if defined)." They encode the script in base64 then execute it but the result is drastically different from the one in console

Browser history broken when using push state

I have the following code in my application which tries to retrieve some data from a URL and then update the browser with the new URL.
const response = await fetch(url);
if (response.ok) {
...
window.history.pushState({ loaded: true }, null, url);
}
Now every time I click to re-fetch the data (with a new URL) the URL in the browser is updated successfully. However if I click back it does nothing unless I click enough times for it to go back to the referred page.
My first fix was to change the code above to the following:
const response = await fetch(url);
if (response.ok) {
...
if (window.history.state && window.history.state.loaded)
window.history.pushState({ loaded: true }, null, url);
else
window.history.replaceState({ loaded: true }, null);
}
Now when I click back after it initially loads it goes back to the referred page as I simply replace the state and don't push anything to the history stack, before I would have had to click back twice. However I still have the issue that when I re-fetch the data it will now push to the history stack and nothing happens when I click back.
I thought the easiest thing would be to reload the page when I click back and the data has loaded. Therefore I tried the following:
window.addEventListener('popstate', e => {
if (e.state && e.state.loaded)
window.location.href = window.location.pathname + window.location.search;
});
However now I can't seem to go back to the referred page. This is because even though I reload the page (in the "popstate" event listener) when it fetches the data it says the state has loaded.
I thought this would be null after the reload but since it's not I thought I would force the browser to make sure the initial state is null when it first loads:
window.addEventListener('load', e => {
window.history.replaceState(null, null);
});
However whilst this initially appeared to work I found this was only because I had the developer tools running (with caching disabled).
I'm sure I'm missing something basic as this seems to be quite a common problem. I'd appreciate it if someone could show me what I am doing wrong.

I've found that changing my fetch request to the following seems to work:
await fetch(url, { cache: 'no-store' });
Alternatively adding an addional query string parameter to the fetched url also works.
I'll leave this open for a little bit incase someone has a better solution.

Implementing Dropbox API V2 in Cordova Application

I have a Cordova application with previous Dropbox implementation using rossmartin/phonegap-dropbox-sync-android. Now as the API V1 is going to be deprecated I want to upgrade to Dropbox API V2. I have searched for plugins for Cordova applications using Dropbox API V2 but didn't find any.So I am trying to implement it using dropbox/dropbox-sdk-js.
For Authentication, I am using authenticateWithCordova method which returns me the Access token (Full documentation here).This method returns Access token once the user completes authentication with Dropbox and uses the redirect URL to redirect the user to Cordova application.
This method works perfectly when the user clicks the button for the first time, but when the user clicks the button again calling this method shows a blank screen and return a new access token. How to avoid seeing the blank screen?
This is the method from Dropbox-sdk.js file, which I have called from my application,
DropboxBase.prototype.authenticateWithCordova = function (successCallback, errorCallback)
{
var redirect_url = 'https://www.dropbox.com/1/oauth2/redirect_receiver';
var url = this.getAuthenticationUrl(redirect_url);
var browser = window.open(url, '_blank');
var removed = false;
var onLoadError = function(event) {
// Try to avoid a browser crash on browser.close().
window.setTimeout(function() { browser.close() }, 10);
errorCallback();
}
var onLoadStop = function(event) {
var error_label = '&error=';
var error_index = event.url.indexOf(error_label);
if (error_index > -1) {
// Try to avoid a browser crash on browser.close().
window.setTimeout(function() { browser.close() }, 10);
errorCallback();
} else {
var access_token_label = '#access_token=';
var access_token_index = event.url.indexOf(access_token_label);
var token_type_index = event.url.indexOf('&token_type=');
if (access_token_index > -1) {
access_token_index += access_token_label.length;
// Try to avoid a browser crash on browser.close().
window.setTimeout(function() { browser.close() }, 10);
var access_token = event.url.substring(access_token_index, token_type_index);
successCallback(access_token);
}
}
};
Here is my code which I use to call the method,
function authenticateWithCordova()
{
var dbx = new Dropbox({ clientId: CLIENT_ID });
dbx.authenticateWithCordova(AuthSuccess,AuthFail);
}
function AuthSuccess(accessToken)
{
localStorage.accessToken = accessToken;
}
function AuthFail()
{
alert("Auth Fail");
}

I have found an analog issue right yesterday. This is the way I solved it.
First, I have set var dbx as global. In my index.js I put these lines immediately after app.initialize():
var CLIENT_ID = 'xxxxxxxxxxxxxxx';
var dbxt;
var dbx = new Dropbox({clientId: CLIENT_ID});
Then I check if dbxt is null: if it is, I create a new Dropbox object using accessToken, otherwise I go with the dropbox connection already established:
if (dbxt == null) {
dbx.authenticateWithCordova(function (accessToken) {
dbxt = new Dropbox({accessToken: accessToken});
dbxt.filesUpload({
path: '/mydump.sql',
contents: sql,
mode: 'overwrite',
mute: true
}).then(function (response) {
alert('Your backup has been successfully uploaded to your Dropbox!')
}).catch(function (error) {
alert('Error saving file to your Dropbox!')
console.error(error);
});
}, function (e){
console.log("failed Dropbox authentication");
}
}else{//dbxt already created
dbxt.filesUpload... //and the rest
}
This is just to avoid to create a new connection and get a new access token everytime and I confess I'm not sure this is a good practice: I only know that before to apply this code I got a lot of bad requests responses by Dropbox server:)
When I used the above code, after the first login, I started to see the blank page: that's is the inappbrowser page which Dropbox OAuth2 uses as redirect URI (set to https://www.dropbox.com/1/oauth2/redirect_receiver in your Dropbox app page).
So the problem was how to make this page invisible. I found a dirty trick applying a small tweak to inappbrowser.js script.
Near the bottom of the script, immediately before this line:
strWindowFeatures = strWindowFeatures || "";
I have put this small block:
if (strUrl.indexOf('dropbox') > -1){
strWindowFeatures += "location=no,hidden=yes";
}
I would have expected to can just use 'hidden=yes' but surprisingly if I remoce 'location=no' the blkank page appears again.
Notice 1: you don't have to modify the script inappbrowser.js located at plugins\cordova-plugin-inappbrowser\www\ but the one you find in platforms\android\platform_www\plugins\cordova-plugin-inappbrowser\www\
Notice 2: I have found this workaround right now so I'm not 100% sure it works perfectly.
Notice 3: making the inappbrowser page invisible, depending on the Internet connection, it could look like nothing is happening for a while, so you'll have to add some loader to inform your user that the app is working.
Hope this help.
UPDATE
I've just realized we can tweak directly the dropbox-sdk instead of inappbrowser.
If you are using Dropbox with browserify you have to open dropbox-base.js and look for authenticateWithCordova() method (it should be at line 107. Then change the line
var browser = window.open(url, '_blank');
to
var browser = window.open(url, '_blank', "location=no,hidden=yes");
If you are using Dropbox-sdk.min.js, you have to look for 'window.open' using the search function of your code editor. It will be easy because 'window.open' is used only once. So you'll have to change the following:
i=window.open(n,"_blank"),
to
i=window.open(n,"_blank","location=no,hidden=yes"),
And this seems to work fine (I prefer to be careful before I get excited).
UPDATE 2
Forgive previous update. My previous check:
if (strUrl.indexOf('dropbox') > -1){
strWindowFeatures += "location=no,hidden=yes";
}
is wrong because it makes invisible any inappbrowser window which tries to connect to dropbox so it prevent us from even logging into Dropbox. So we need to change it to
if (strUrl == 'https://www.dropbox.com/1/oauth2/redirect_receiver') {
strWindowFeatures += "location=no,hidden=yes";
}
This way we can do the login correctly and next connections won't show the inappbrowser window, as we want.
So summarizing:
Ignore my first update
Use UPDATE 2 to modify the url check in inappbrowser.js
Forgive me for the confusion...

Retrieve html content of a page several seconds after it's loaded

I'm coding a script in nodejs to automatically retrieve data from an online directory.
Knowing that I had never done this, I chose javascript because it is a language I use every day.
I therefore from the few tips I could find on google use request with cheerios to easily access components of dom of the page.
I found and retrieved all the necessary information, the only missing step is to recover the link to the next page except that the one is generated 4 seconds after loading of page and link contains a hash so that this step Is unavoidable.
What I would like to do is to recover dom of page 4-5 seconds after its loading to be able to recover the link
I looked on the internet, and much advice to use PhantomJS for this manipulation, but I can not get it to work after many attempts with node.
This is my code :
#!/usr/bin/env node
require('babel-register');
import request from 'request'
import cheerio from 'cheerio'
import phantom from 'node-phantom'
phantom.create(function(err,ph) {
return ph.createPage(function(err,page) {
return page.open(url, function(err,status) {
console.log("opened site? ", status);
page.includeJs('http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js', function(err) {
//jQuery Loaded.
//Wait for a bit for AJAX content to load on the page. Here, we are waiting 5 seconds.
setTimeout(function() {
return page.evaluate(function() {
var tt = cheerio.load($this.html())
console.log(tt)
}, function(err,result) {
console.log(result);
ph.exit();
});
}, 5000);
});
});
});
});
but i get this error :
return ph.createPage(function (page) {
^
TypeError: ph.createPage is not a function
Is what I am about to do is the best way to do what I want to do? If not what is the simplest way? If so, where does my error come from?

If You dont have to use phantomjs You can use nightmare to do it.
It is pretty neat library to solve problems like yours, it uses electron as web browser and You can run it with or without showing window (You can also open developer tools like in Google Chrome)
It has only one flaw if You want to run it on server without graphical interface that You must install at least framebuffer.
Nightmare has method like wait(cssSelector) that will wait until some element appears on website.
Your code would be something like:
const Nightmare = require('nightmare');
const nightmare = Nightmare({
show: true, // will show browser window
openDevTools: true // will open dev tools in browser window
});
const url = 'http://hakier.pl';
const selector = '#someElementSelectorWitchWillAppearAfterSomeDelay';
nightmare
.goto(url)
.wait(selector)
.evaluate(selector => {
return {
nextPage: document.querySelector(selector).getAttribute('href')
};
}, selector)
.then(extracted => {
console.log(extracted.nextPage); //Your extracted data from evaluate
});
//this variable will be injected into evaluate callback
//it is required to inject required variables like this,
// because You have different - browser scope inside this
// callback and You will not has access to node.js variables not injected
Happy hacking!

javascript window.open from callback

window.open() called from main thread opens new tab by default.
But, here open new window every time (Opera 16 and Google Chrome 29)
<input type="button" value="Open" onclick="cb1()">
<script type="text/javascript">
function cb1() {
setTimeout(wo, 1000); //simple async
}
function wo()
{
var a = window.open("http://google.com", "w2");
a.focus();
}
</script>
(lol, this is my answer for Open a URL in a new tab (and not a new window) using JavaScript).
How I can open in the tab (by browser default) here?

We ran across this same problem and hunted around SO for an answer. What we found works in our circumstances and the distilled wisdom is as follows:
The problem is related to browser pop-up blockers preventing programmatic window opens. Browsers allow window opens from actual user clicks which occur on the main thread. Similarly, if you call window.open on the main thread it will work, as noted above. According to this answer on Open a URL in a new tab (and not a new window) using JavaScript if you are using an Ajax call and want to open the window on success you need to set async: false which works because that will keep everything on the main thread.
We couldn't control our Ajax call like that, but found another solution that works because of the same reasons. Warning, it is a bit hacky and may not be appropriate for you given your constraints. Courtesy of a comment on a different answer on Open a URL in a new tab (and not a new window) using JavaScript you open the window before calling setTimeout and then update it in the delayed function. There are a couple of ways of doing this. Either keep a reference to the window when you open it, w = window.open... and set w.location or open with a target, window.open('', 'target_name'), in the delayed function open in that target, window.open('your-url', 'target_name'), and rely on the browser keeping the reference.
Of course, if the user has their settings to open links in a new window this isn't going to change that, but that wasn't a problem for the OP.

Like the other posts mentions the best way to do this is to first open the window and then set its location after the callback or asynchronous function
<input type="button" value="Open" onclick="cb1()">
<script type="text/javascript">
function cb1() {
var w = window.open('', 'w2');
setTimeout(function () {
wo(w);
}, 1000); //simple async
}
function wo(w)
{
w.location = "http://google.com";
w.focus();
}
</script>
Alternatively if you are using async await you will also have the same problem. The same solution still applies.
public async openWindow(): Promise<void> {
const w = window.open('', '_blank');
const url = await getUrlAsync();
w.location = url;
}
A further enhancement is to open the window on an initial page that provides some quick feedback either by loading a url or writing some html to that page
public async openWindow(): Promise<void> {
const w = window.open('', '_blank');
w.document.write("<html><head></head><body>Please wait while we redirect you</body></html>");
w.document.close();
const url = await getUrlAsync();
w.location = url;
}
This will prevent a user looking at a blank tab/window for however long it takes to resolve your URL.

This is even 'hackier' but...
If you wrap the window.open in a console.log then it will work.
console.log(window.open('https://someurl', '_blank'))

If a new window is opened as a new tab, or a new instance, depends on the user-settings.

I was working with Nestjs and Vue3. but I solved it using this code write here.
I just took the token that was sent from the back end on the localhost:8080.
So I just slice the token and set it local storage then redirect the user to the next page. which will authorize the user to use this token.
Vue 3.
This way is solved. you can improve it and make it even better.
that's how I did it because I used the OAuth google-passport in the backend instead of firebase
<script>
methods: {
googleLogin() {
const win = window.open(
"http://localhost:3000",
"windowname1",
"width=800, height=600"
);
const validateToken = (token) => {
localStorage.setItem("token", token);
this.$router.push("/GeneralPage");
window.clearInterval(pollTimer);
clearInterval(pollTimer);
win.close();
};
const pollTimer = window.setInterval(async () => {
try {
let url = win.document.URL;
let token = url.slice(22, url.length);
if (token !== "") {
localStorage.setItem("token", token);
this.$router.push("/GeneralPage");
clearInterval(pollTimer);
win.close();
validateToken(token);
return;
}
return;
} catch (e) {}
}, 1000);
},
}
</script>