We Keep Coding

AngleSharp: run all JavaScript before looking for element

I'm currently building an app for web automation using AngleSharp. I've managed to log in to a website, but can't find the element I'm looking for using context.Active.QuerySelectorAll.
I understand that this is likely because some of the JavaScript hasn't run in the HTML I'm searching through, as per this link: Is the HTML shown via 'View Source' different from the HTML shown in (Firebug) developer tools?
How do I force AngleSharp to execute all of the JavaScript before I look for the specific element?
code:
var config = AngleSharp.Configuration.Default.WithDefaultLoader().WithCookies().WithJavaScript().WithCss();
var browsingContext = BrowsingContext.New(config);
await browsingContext.OpenAsync("https://users.premierleague.com/");
await browsingContext.Active.QuerySelector<IHtmlFormElement>("form[action='/accounts/login/']").SubmitAsync(new
{
login = "abc#gmail.com",
password = "password"
});
await browsingContext.OpenAsync("https://fantasy.premierleague.com/a/team/my/");
Everything works fine up until this point, and I can confirm that I am logged in. However, I then can't seem to get a value returned for the following:
var x = browsingContext.Active.QuerySelectorAll("*").Where(m => m.ClassName == "ismjs-link ism-link ism-link--more");
And I know that this element exists, as I've checked numerous times through the "Inspect" functionality available on Google Chrome.
What am I missing/ how do I get the JavaScript to run?
Thanks!

How to print out the results of the following snipet of code?

var ctof = function () {
var celsius = prompt("How many degrees celsius?")
var degC = parseInt(celcius)
var fahrenheit = ((degC * (9 / 5)) + 32)
return fahrenheit
};
document.write(ctof());
The window that appears in my browser (safari) seems to read the line where I prompt the user for input, however it does not write the output of the 'ctof' function onto the browser page. How can I fix this?

The answer to your problem: You mistyped celsius as celcius with two c's.
How to find this kind of mistakes on your own? You can open Chrome DevTools by right clicking on the page and selecting 'inspect'. In Windows you can press F12 and on Mac you can press 'Cmd + Opt + i'.
Now that Chrome DevTools is open, go to the console tab. In here you can see all the errors that are produced by JavaScript code running on that page. When you run your code, fill in the prompt and press enter, you see the following error:
Uncaught ReferenceError: celcius is not defined(…)
p.s: See this page for more awesome shortcuts for the Chrome DevTools.

iframe content doesn't always load

So I have a system that essentially enabled communication between two computers, and uses a WebRTC framework to achieve this:
"The Host": This is the control computer, and clients connect to this. They control the clients window.
"The Client": The is the user on the other end. They are having their window controlled by the server.
What I mean by control, is that the host can:
change CSS on the clients open window.
control the URL of an iframe on the clients open window
There are variations on these but essentially thats the amount of control there is.
When "the client" logs in, the host sends a web address to the client. This web address will then be displayed in an iframe, as such:
$('#iframe_id').attr("src", URL);
there is also the ability to send a new web address to the client, in the form of a message. The same code is used above in order to navigate to that URL.
The problem I am having is that on, roughly 1 in 4 computers the iframe doesn't actually load. It either displays a white screen, or it shows the little "page could not be displayed" icon:
I have been unable to reliably duplicate this bug
I have not seen a clear pattern between computers that can and cannot view the iframe content.
All clients are running google chrome, most on an apple powermac. The only semi-link I have made is that windows computers seem slightly more susceptible to it, but not in a way I can reproduce. Sometimes refreshing the page works...
Are there any known bugs that could possibly cause this to happen? I have read about iframe white flashes but I am confident it isn't that issue. I am confident it isn't a problem with jQuery loading because that produces issues before this and would be easy to spot.
Thanks so much.
Alex
edit: Ok so here is the code that is collecting data from the server. Upon inspection the data being received is correct.
conn.on('data', function(data) {
var data_array = JSON.parse(data);
console.log(data_array);
// initialisation
if(data_array.type=='init' && inititated === false) {
if(data_array.duration > 0) {
set_timeleft(data_array.duration); // how long is the exam? (minutes)
} else {
$('#connection_remainingtime').html('No limits');
}
$('#content_frame').attr("src", data_array.uri); // url to navigate to
//timestarted = data_array.start.replace(/ /g,''); // start time
ob = data_array.ob; // is it open book? Doesnt do anything really... why use it if it isnt open book?
snd = data_array.snd; // is sound allowed?
inititated = true;
}
}
It is definitele trying to make the iframe navigate somewhere as when the client launches the iframe changes - its trying to load something but failing.
EDIT: Update on this issue: It does actually work, just not with google forms. And again it isn't everybody's computers, it is only a few people. If they navigate elsewhere (http://www.bit-tech.net for example) then it works just fine.
** FURTHER UPDATE **: It seems on the ones that fail, there is an 'X-Frames-Origin' issue, in that its set the 'SAMEORIGIN'. I dont understand why some students would get this problem and some wouldn't... surely it depends upon the page you are navigating to, and if one person can get it all should be able to?

So the problem here was that the students were trying to load this behind a proxy server which has an issue with cookies. Although the site does not use cookies, the proxy does, and when the student had blocked "third party cookies" in their settings then the proxy was not allowing the site to load.
Simply allowed cookies and it worked :)

iframes are one of the last things to load in the DOM, so wrap your iframe dependent code in this:
document.getElementById('content_frame').onload = function() {...}
If that doesn't work then it's the document within the iframe. If you own the page inside the iframe then you have options. If not...setTimeout? Or window.onload...?
SNIPPET
conn.on('data', function(data) {
var data_array = JSON.parse(data);
console.log(data_array);
// initialisation
if (data_array.type == 'init' && inititated === false) {
if (data_array.duration > 0) {
set_timeleft(data_array.duration); // how long is the exam? (minutes)
} else {
$('#connection_remainingtime').html('No limits');
}
document.getElementById('content_frame').onload = function() {
$('#content_frame').attr("src", data_array.uri); // url to navigate to
//timestarted = data_array.start.replace(/ /g,''); // start time
ob = data_array.ob; // is it open book? Doesnt do anything really... why use it if it isnt open book?
snd = data_array.snd; // is sound allowed?
inititated = true;
}
}
}

CasperJS- Register on a site and validate the mail sent on Gmail -for both slimer and phantom-

Edit : this is the windows behaviour, with linux it just fails.
First, if you succeeded navigate on gmail with casper (without random waiting time -from 20sec to 5min-), please tell me.
I want to register on our site, then validate my registration automatically with Gmail (an entire register step). Did someone do that before?
I have no problem to register, and I can login on my mailbox (Gmail) but after i have some troubles to navigate and validate my registration in Gmail, and i observe different behaviors between phantomJS and slimerJS.
In phantom it will work (without special commands), but it may take until 5 minutes before pass in the next step (waitForSelector). And with slimerjs it just stays stuck on the mailbox page.
EDIT : A strange thing : if i click manually (slimer) on a link which opens a popup, it stops being blocked and my navigation continues, it's like it can't detect the end of the step itself and can't perform the waitFor after the submit click without another interaction. Is it a refresh/reload problem?
Try that to see yourself :
casper.thenOpen('https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=en', function(){
this.sendKeys("input#Email","your mail");
this.sendKeys("input#Passwd","your password");
this.click("input#signIn.rc-button-submit");
console.log(this.getCurrentUrl());
this.waitForSelector(".aeF",function(){//fail with linux -> timeout
this.test.pass("ok"); //windows -> stuck in slimer, several times in phantom
this.test.assertExists(".T-I.J-J5-Ji.T-I-KE.L3","Gmail Home ok");
console.log("url "+this.getCurrentUrl());
});
And i don't get any timeOut error. In slimerjs it just keeps the page opened.
If i do a waitForPopup instead of a waitForUrl, i have the error (timeout -> did not pop up), so why does a waitForUrl/waitForSelector... stay stuck ? I tried --web-security=no,--ignore-ssl-errors=true commands too (not linked but i tried --output-encoding=ISO 8859-1 too which doesn't work).
Here the differences between phantom and slimer (doc) :
http://docs.slimerjs.org/0.8/differences-with-phantomjs.html
(useless in this issue i think)

Well, we finally found a way to do it : the problem is by default gmail loop on ajax requests, to check some new mails, etc... see Page polls remote url, causing casper to block in step.
Fortunately google proposes a way to avoid that, using the simplified HTML version (you can for example use a special gmail address for your tests using this version) :
That way the script works as it should.
Bonus :
/*
* Click on an element specified by its selector and a part of its text content.
* This method resolves some problem as random space in textNode / more flexible too.
* Need to fix one bug though : when there is a tag in textContent of our selector.
*/
casper.clickSelectorHasText = function (selector, containsText){
var tmp = this.getElementsInfo(selector)
,i
,l
,bool=false
;
for (i=0,l=tmp.length;i<l; i++){
if(tmp[i].text && tmp[i].text.indexOf(containsText)!==-1){
this.clickLabel(tmp[i].text);
bool=true;
break;
}
}
casper.test.assert(bool, "clickSelectorHasText done, text and selector found -> click selector " + selector +" which contains text " + containsText);
};
casper.thenOpen('https://accounts.google.com/ServiceLogin?service=mail&continue=https://mail.google.com/mail/&hl=en', function scrapeCode(){
//log in
this.sendKeys("input#Email","your email");
this.sendKeys("input#Passwd","your password");
this.click("input#signIn.rc-button-submit");
//wait to redirect to our mailbox
this.waitForSelector("form[name='f']",function(){
//check main block
this.test.assertExists("form[name='f']","Gmail Home ok");
this.test.assertSelectorHasText("span", "Your gmail title message");
this.clickSelectorHasText("font", "one string which appears in font tag");
//wait inscription message appears
this.waitForSelector("div.msg",function(){
this.test.assertSelectorHasText("a","the message which activates your account--> in <a>");
});
})
//validate our account
.then(function(){
this.clickLabel("the message which activates your account--> in <a>");
this.waitForPopup(/mail/, function(){
this.test.pass("popup opened");
});
this.withPopup(/mail/, function(){
this.viewport(1400,800);
this.test.pass("With Popup");
//wait something on your website (for me selector .boxValid)
this.waitForSelector(".boxValid", function(){
/*
* Here your code after validation
*/
});
});
})
It might be possible to do it with normal gmail using event, see resource.received.

Chrome back button terminates upload stream unexpectedly

I'm providing users form-based file uploads (one at a time) with ajax updates/progress bars. I want to give them a heads up if they navigate away that any current upload will be canceled. The following code achieves that goal except in Chrome (v.18). If you click the back button in Chrome the file upload stream is immediately terminated, throwing a org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly from the commons.fileupload library.
I couldn't find any information specific to this issue, but does anyone have any idea if this is something I'm just going to have to live with for now? Thanks.
Edit: I should add that Chrome does still shows the "Leave page" dialog, but at that point the upload is already stopped so the answer won't affect it at all. Clicking a link in Chrome does not stop the upload, and behaves as expected.
window.onbeforeunload =
function(event) {
if (upload_in_progress) {
var msg = "You are uploading a file." +
"If you leave this page the upload " +
"will be cancelled.\n\nLeave page?";
var event = event || window.event;
if (event) { event.returnValue = msg; }
return msg;
}
};

In Chrome and Safari functions attached to beforeunload event must always return string value (message displayed to user).