My question is closely related to this post: Splash do not render the whole page
I am not able to configure splash in a way that in renders the dynamic part of the following website:
https://www.wunderground.com/history/daily/ro/slava-cerchez%C4%83/LRTC/date/2018-8-03?cm_ven=localwx_history
The page looks like this
But it should look like this
Here is the splash-script:
function main(splash, args)
assert(splash:go(args.url))
assert(splash:wait(5))
splash:set_viewport_full()
splash.private_mode_enabled = false
splash.indexeddb_enabled = true
splash.html5_media_enabled = true
return {
html = splash:html(),
png = splash:png(),
har = splash:har(),
}
end
I already tried to increase waiting time and to disable private mode. I would be very glad if someone could give a hint how to configure splash correctly so that it renders the java script part.
I had the same issue with the historical weather data on that site. I was unable to figure out how to get the page to render using Splash. However, using a Seleuium driven browser appears to render the page correctly. This isn't as ideal because you'll actually have to render the page, but it works.
Selenium: https://selenium-python.readthedocs.io/installation.html
Related
There is an image open in the Preview application, and it is unsaved.
I have tried this code, by building and running a standalone script in Script Editor:
var preview = Application('Preview')
var preview.documents[0].close({ saving: true, savingIn: '/Volumes/USB Drive/Untitled-Image.png' })
I see this in the logs
app = Application("Preview")
app.documents.at(0).Symbol.toPrimitive()
--> Error -1700: Can't convert types.
app.documents.at(0).Symbol.toPrimitive()
--> Error -1700: Can't convert types.
However, I'm not sure I'm reading the docs correctly, and I'm not sure how to further debug this code. A satisfactory alternative here would be to send a keystroke to save and click the Save button.
What can I do to persist the image in Preview to a file?
One can declare paths speculatively in JavaScript using Path('...'), and then use the save command to save the image to the file that will be created at that path:
Preview = Application('com.apple.Preview');
ImageFilepath = Path('/Users/CK/Pictures/Preview_SavedImage.jpg');
Preview.documents[0].save({ in: ImageFilepath });
This code worked for me.
var preview = Application('Preview')
var photo = preview.documents[0]
photo.close({ saving: 'yes', savingIn: Path("/Path/To Some/file_name.png") })
Upon execution, in the Replies log of Script Editor window, something slightly different from the question sample is logged.
app = Application("Preview")
app.close(app.documents.at(0), {savingIn:Path("/Path/To Some/file_name.png"), saving:"yes"})
I was able to figure this out by studying the scripting dictionary for macOS Mojave 10.14 as well as the Release Notes for JXA; there is a similar example under Passing Event Modifiers section of that.
From the Script Editor scripting dictionaries, this is the entry for the close method in the Standard Suite:
close method : Close a document.
close specifier : the document(s) or window(s) to close.
[saving: "yes"/"no"/"ask"] : Should changes be saved before closing?
[savingIn: File] : The file in which to save the document, if so.
Note the key differences between the working code here and the code in the question:
Specify a Path object when passing the savingIn option to close.
Specify a the saving option as one of "yes", "no", or "ask" (not true).
If you're new to JXA, Automator, and/or Script Editor for Mac, checkout the JXA Cookbook.
If this answer helps you, you're probably having a bad time, so best of luck!
I'm currently building an app for web automation using AngleSharp. I've managed to log in to a website, but can't find the element I'm looking for using context.Active.QuerySelectorAll.
I understand that this is likely because some of the JavaScript hasn't run in the HTML I'm searching through, as per this link: Is the HTML shown via 'View Source' different from the HTML shown in (Firebug) developer tools?
How do I force AngleSharp to execute all of the JavaScript before I look for the specific element?
code:
var config = AngleSharp.Configuration.Default.WithDefaultLoader().WithCookies().WithJavaScript().WithCss();
var browsingContext = BrowsingContext.New(config);
await browsingContext.OpenAsync("https://users.premierleague.com/");
await browsingContext.Active.QuerySelector<IHtmlFormElement>("form[action='/accounts/login/']").SubmitAsync(new
{
login = "abc#gmail.com",
password = "password"
});
await browsingContext.OpenAsync("https://fantasy.premierleague.com/a/team/my/");
Everything works fine up until this point, and I can confirm that I am logged in. However, I then can't seem to get a value returned for the following:
var x = browsingContext.Active.QuerySelectorAll("*").Where(m => m.ClassName == "ismjs-link ism-link ism-link--more");
And I know that this element exists, as I've checked numerous times through the "Inspect" functionality available on Google Chrome.
What am I missing/ how do I get the JavaScript to run?
Thanks!
I am trying to inject some javascript into my web view when it is navigated to a product page URL. The website doesn't reload when navigating to different pages, so to my understanding that means it is using Ajax.
The problem is I need the page to be fully loaded because the purpose of the javascript I am using is to automatically select the size drop down.
I tried to use the navigation delegate but since it's not reloading between pages it only gets called when the web view is first loaded.
What I have done is setup and observer by
webView.addObserver(self, forKeyPath: "URL", options: .new, context: nil)
and check if the current URL is the URL I want to inject the javascript on by
override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) {
if keyPath == "URL" {
guard var currentURL = webView.url?.absoluteString else { return }
if currentURL.lowercased().range(of: "products") != nil {
webView.evaluateJavaScript("document.getElementById('size-options').selectedIndex = 1")
}
}
}
The problem with this is it gets called right when the URL changes but not when the page finishes loading. I have got it to successfully work by adding a delay but that isn't a good solution because if the page doesn't load in time it won't work. Is there any way to know when a page like this finishes loading all its elements? The website I am working with is http://www.supremenewyork.com/mobile/#categories mobile site.
So I have a system that essentially enabled communication between two computers, and uses a WebRTC framework to achieve this:
"The Host": This is the control computer, and clients connect to this. They control the clients window.
"The Client": The is the user on the other end. They are having their window controlled by the server.
What I mean by control, is that the host can:
change CSS on the clients open window.
control the URL of an iframe on the clients open window
There are variations on these but essentially thats the amount of control there is.
When "the client" logs in, the host sends a web address to the client. This web address will then be displayed in an iframe, as such:
$('#iframe_id').attr("src", URL);
there is also the ability to send a new web address to the client, in the form of a message. The same code is used above in order to navigate to that URL.
The problem I am having is that on, roughly 1 in 4 computers the iframe doesn't actually load. It either displays a white screen, or it shows the little "page could not be displayed" icon:
I have been unable to reliably duplicate this bug
I have not seen a clear pattern between computers that can and cannot view the iframe content.
All clients are running google chrome, most on an apple powermac. The only semi-link I have made is that windows computers seem slightly more susceptible to it, but not in a way I can reproduce. Sometimes refreshing the page works...
Are there any known bugs that could possibly cause this to happen? I have read about iframe white flashes but I am confident it isn't that issue. I am confident it isn't a problem with jQuery loading because that produces issues before this and would be easy to spot.
Thanks so much.
Alex
edit: Ok so here is the code that is collecting data from the server. Upon inspection the data being received is correct.
conn.on('data', function(data) {
var data_array = JSON.parse(data);
console.log(data_array);
// initialisation
if(data_array.type=='init' && inititated === false) {
if(data_array.duration > 0) {
set_timeleft(data_array.duration); // how long is the exam? (minutes)
} else {
$('#connection_remainingtime').html('No limits');
}
$('#content_frame').attr("src", data_array.uri); // url to navigate to
//timestarted = data_array.start.replace(/ /g,''); // start time
ob = data_array.ob; // is it open book? Doesnt do anything really... why use it if it isnt open book?
snd = data_array.snd; // is sound allowed?
inititated = true;
}
}
It is definitele trying to make the iframe navigate somewhere as when the client launches the iframe changes - its trying to load something but failing.
EDIT: Update on this issue: It does actually work, just not with google forms. And again it isn't everybody's computers, it is only a few people. If they navigate elsewhere (http://www.bit-tech.net for example) then it works just fine.
** FURTHER UPDATE **: It seems on the ones that fail, there is an 'X-Frames-Origin' issue, in that its set the 'SAMEORIGIN'. I dont understand why some students would get this problem and some wouldn't... surely it depends upon the page you are navigating to, and if one person can get it all should be able to?
So the problem here was that the students were trying to load this behind a proxy server which has an issue with cookies. Although the site does not use cookies, the proxy does, and when the student had blocked "third party cookies" in their settings then the proxy was not allowing the site to load.
Simply allowed cookies and it worked :)
iframes are one of the last things to load in the DOM, so wrap your iframe dependent code in this:
document.getElementById('content_frame').onload = function() {...}
If that doesn't work then it's the document within the iframe. If you own the page inside the iframe then you have options. If not...setTimeout? Or window.onload...?
SNIPPET
conn.on('data', function(data) {
var data_array = JSON.parse(data);
console.log(data_array);
// initialisation
if (data_array.type == 'init' && inititated === false) {
if (data_array.duration > 0) {
set_timeleft(data_array.duration); // how long is the exam? (minutes)
} else {
$('#connection_remainingtime').html('No limits');
}
document.getElementById('content_frame').onload = function() {
$('#content_frame').attr("src", data_array.uri); // url to navigate to
//timestarted = data_array.start.replace(/ /g,''); // start time
ob = data_array.ob; // is it open book? Doesnt do anything really... why use it if it isnt open book?
snd = data_array.snd; // is sound allowed?
inititated = true;
}
}
}
Background
I have a network performance testing tool which after every test,displays all its results in a new results page. Below is my workflow to find the final value.(Details not relevant here but the html elements i am looking for might
1.Go to a section called "Data frame" ( which has a table with timestamp,transmit,recieve and percentage as columns) and get the time stamp when percentage first drops below 99.9
Here is a snippet of the "Data Frame" table
Data frame snippet
2.Go to another section called "Data throughput" ( table with timestamp, throughput as columns) and get the maximum value of throughput before the previously found timestamp.
After every test, i have to do this manually, but there should be someway to automate this.
Question
1.Is it possible to write a script ( maybe in javascript ) to automate my workflow for every new results page?
example algorithm
section = webpage.section_with_title("Data frames")
for each row in section
find first my_timestamp with percent < 99.9
done
section2 = webpage.section_with_title("Data Throughput")
for each row in section2
find max row.throughput if row.timestamp < my_timestamp
don
2.Once i write the script, how do i execute it in the web page?
3.Can i leverage the firefox/chrome developer console in some way ?
Note:I develop mostly in C,C++ and ruby with very basic experience in HTML and CSS. I have barely used javascript, but i have a vague idea that the browsers use it to handle the content of every page.
My thought process was to somehow execute the script through firefox console, and it will provide me with the results... is that difficult?
edit: since i am not familiar with the web development terms, ill put an a different choice of words...
1.i open up the results page
2.i open the console window of firefox/chrome
3.run my script(dont know what language).
4.i should get the result in the console. i do not want to modify the results page. i just need the result displayed to me(in the console, or to a text file).
Yes it's possible to do what you think using Javascript. Please look up XMLHttpRequest
Here is a way for you to start (in Java):
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
public class YourClass
{
private WebDriver webDriver = null;
public void open() throws Exception
{
webDriver = new FirefoxDriver();
}
public void close() throws Exception
{
webDriver.quit();
}
public void doStuff() throws Exception
{
WebElement element = webDriver.findElement(By.id(...));
...
// Alternatively, it might be easier to parse the entire
// page-source instead of searching elements in the DOM
String pageSource = webDriver.getPageSource();
...
}
}
You will need to download the following JAR files and add them in your project's class-path:
selenium-java-2.xx.0.jar (currently xx=39)
selenium-server-standalone-2.xx.0.jar (currently xx=39)