Background
I have a network performance testing tool which after every test,displays all its results in a new results page. Below is my workflow to find the final value.(Details not relevant here but the html elements i am looking for might
1.Go to a section called "Data frame" ( which has a table with timestamp,transmit,recieve and percentage as columns) and get the time stamp when percentage first drops below 99.9
Here is a snippet of the "Data Frame" table
Data frame snippet
2.Go to another section called "Data throughput" ( table with timestamp, throughput as columns) and get the maximum value of throughput before the previously found timestamp.
After every test, i have to do this manually, but there should be someway to automate this.
Question
1.Is it possible to write a script ( maybe in javascript ) to automate my workflow for every new results page?
example algorithm
section = webpage.section_with_title("Data frames")
for each row in section
find first my_timestamp with percent < 99.9
done
section2 = webpage.section_with_title("Data Throughput")
for each row in section2
find max row.throughput if row.timestamp < my_timestamp
don
2.Once i write the script, how do i execute it in the web page?
3.Can i leverage the firefox/chrome developer console in some way ?
Note:I develop mostly in C,C++ and ruby with very basic experience in HTML and CSS. I have barely used javascript, but i have a vague idea that the browsers use it to handle the content of every page.
My thought process was to somehow execute the script through firefox console, and it will provide me with the results... is that difficult?
edit: since i am not familiar with the web development terms, ill put an a different choice of words...
1.i open up the results page
2.i open the console window of firefox/chrome
3.run my script(dont know what language).
4.i should get the result in the console. i do not want to modify the results page. i just need the result displayed to me(in the console, or to a text file).
Yes it's possible to do what you think using Javascript. Please look up XMLHttpRequest
Here is a way for you to start (in Java):
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
public class YourClass
{
private WebDriver webDriver = null;
public void open() throws Exception
{
webDriver = new FirefoxDriver();
}
public void close() throws Exception
{
webDriver.quit();
}
public void doStuff() throws Exception
{
WebElement element = webDriver.findElement(By.id(...));
...
// Alternatively, it might be easier to parse the entire
// page-source instead of searching elements in the DOM
String pageSource = webDriver.getPageSource();
...
}
}
You will need to download the following JAR files and add them in your project's class-path:
selenium-java-2.xx.0.jar (currently xx=39)
selenium-server-standalone-2.xx.0.jar (currently xx=39)
Related
My question is closely related to this post: Splash do not render the whole page
I am not able to configure splash in a way that in renders the dynamic part of the following website:
https://www.wunderground.com/history/daily/ro/slava-cerchez%C4%83/LRTC/date/2018-8-03?cm_ven=localwx_history
The page looks like this
But it should look like this
Here is the splash-script:
function main(splash, args)
assert(splash:go(args.url))
assert(splash:wait(5))
splash:set_viewport_full()
splash.private_mode_enabled = false
splash.indexeddb_enabled = true
splash.html5_media_enabled = true
return {
html = splash:html(),
png = splash:png(),
har = splash:har(),
}
end
I already tried to increase waiting time and to disable private mode. I would be very glad if someone could give a hint how to configure splash correctly so that it renders the java script part.
I had the same issue with the historical weather data on that site. I was unable to figure out how to get the page to render using Splash. However, using a Seleuium driven browser appears to render the page correctly. This isn't as ideal because you'll actually have to render the page, but it works.
Selenium: https://selenium-python.readthedocs.io/installation.html
I have testing environment which is perfectly working with chrome driver in desktop mode. I am using some javascript injections (everything works) f.e.:
public static void ForceFillInput(this Driver driver, string selector, string value)
{
var javaScriptExecutor = (IJavaScriptExecutor)driver.webDriver;
javaScriptExecutor.ExecuteScript($"$(\"{selector}\").val(\"{value}\")");
}
but when i want to run it in headless mode
AddArguments("--headless")
it will just fail on
"$ is not defined"
Can somebody help me how to inject js/jquery into headless solution?
M.
your Javascript snippet used jQuery api. In modern web development, we put Javascript at the end of HTML page to let browser to load javascript at last, so that static resources (like picture/image/text content) can display earlier as possible, withing this way to improve user experience when user open website.
I think your page also put jQuery at the end to load, try add some wait/sleep before ExecuteScript to wait browser complete load jQuery.
It looks like the shorthand for JQuery is not yet created at the time your script is executed.
Use a waiter to wait for JQuery and for the selector to be found:
public static void ForceFillInput(this Driver driver, string selector, string value)
{
string JS_SET_VALUE =
"var e; return !!window.$ && (e = window.$(arguments[0])).length > 0 && (e.val(arguments[1]), true);";
new WebDriverWait(driver, TimeSpan.FromSeconds(60))
.until(ctx => (bool)((IJavaScriptExecutor)ctx).ExecuteScript(JS_SET_VALUE, selector, value));
}
I am looking for a way to confirm the image has actually loaded. Maybe JavaScript is an option? I am using right now WebdriverWait to simply force the wait on all images, then the actual src, then I finally run my tests after...
wait = WebDriverWait(driver, 10)
...
albums = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".albums .album img")))
albumslength = len(albums)-1
while albumslength > 0:
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ('.album:nth-child('+str(albumslength)+') img[src*="album-foo"]'))))
albumslength -= 1
...
This will at some point confirm "album-foo" exists in the <img src="http://www.album-foo/images/blah.jpg" of each album in albums. But it does not confirm that the image completed loading into view. I have searched high and low, anyone have a solution?
I am testing on Chrome and Firefox, so no phantom or IE restrictions, if the ideas grow into including JavaScript or related ideas that might have environment restrictions.
I am not that experienced with selenium, though I will give this a try.
If you want to check whether an image is loaded, I'd say use a boolean. Set the original value of this boolean let's say 'loaded' to false.
Then just call something like this, making sure it returns true the moment the image has loaded.
Boolean loaded = false;
WebDriver driver = new AnyDriverYouWant();
JavascriptExecutor js;
if (driver instanceof JavascriptExecutor) {
js = (JavascriptExecutor)driver;
}
loaded = (Boolean)js.executeScript("image.onload(function() { return true; });");
I wrote a PhantomJS app to crawl over a site I built and check for a JavaScript file to be included. The JavaScript is similar to Google where some inline code loads in another JS file. The app looks for that other JS file which is why I used Phantom.
What's the expected result?
The console output should read through a ton of URLs and then tell if the script is loaded or not.
What's really happening?
The console output will read as expected for about 50 requests and then just start spitting out this error:
2013-02-21T10:01:23 [FATAL] QEventDispatcherUNIXPrivate(): Can not continue without a thread pipe
QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files
This is the block of code that opens a page and searches for the script include:
page.open(url, function (status) {
console.log(YELLOW, url, status, CLEAR);
var found = page.evaluate(function () {
if (document.querySelectorAll("script[src='***']").length) {
return true;
} else { return false; }
});
if (found) {
console.log(GREEN, 'JavaScript found on', url, CLEAR);
} else {
console.log(RED, 'JavaScript not found on', url, CLEAR);
}
self.crawledURLs[url] = true;
self.crawlURLs(self.getAllLinks(page), depth-1);
});
The crawledURLs object is just an object of urls that I've already crawled. The crawlURLs function just goes through the links from the getAllLinks function and calls the open function on all links that have the base domain of the domain that the crawler started on.
Edit
I modified the last block of the code to be as follows, but still have the same issue. I have added page.close() to the file.
if (!found) {
console.log(RED, 'JavaScript not found on', url, CLEAR);
}
self.crawledURLs[url] = true;
var links = self.getAllLinks(page);
page.close();
self.crawlURLs(links, depth-1);
From the documentation:
Due to some technical limitations, the web page object might not be completely garbage collected. This is often encountered when the same object is used over and over again.
The solution is to explicitly call close() of the web page object (i.e. page in many cases) at the right time.
Some included examples, such as follow.js, demonstrate multiple page objects with explicit close.
Open Files Limit.
Even with closing files properly, you might still run into this error.
After scouring the internets I discovered that you need to increase your limit of the number of files a single process is allowed to have open. In my case, I was generating PDFs with hundreds to thousands of pages.
There are different ways to adjust this setting based on the system you are running but here is what worked for me on an Ubuntu server:
Add the following to the end of /etc/security/limits.conf:
# Sets the open file maximum here.
# Generating large PDFs hits the default ceiling (1024) quickly.
* hard nofile 65535
* soft nofile 65535
root hard nofile 65535 # Need these two lines because the wildcards (above)
root soft nofile 65535 # are not applied to the root user as well.
A good reference for the ulimit command can be found here.
I hope that puts some people on the right track.
I had this error come up while running multiple threads in my ruby program.
I was running phantomjs with Capybara-poltergeist and each thread was visiting a page opening up the same CSV file and writing to it.
I was able to fix it by using the Mutex class.
lock = Mutex.new
lock.synchronize do
CSV.open("reservations.csv", "w") do |file|
file << ["Status","Name","Res-Code","LS-Num","Check-in","Check-out","Talk-URL"]
$status.length.times do |i|
file << [$status[i],$guest_name[i],$reservation_code[i],$listing_number[i],$check_in[i],$check_out[i], $talk_url[i]]
end
end
puts "#{user.email} PAGE NUMBER ##{p+1} WRITTEN TO CSV"
end
end
I am trying to use Apple's UIAutomation to write unit tests for an iOS Application that has a server-side component. In order to setup the test server in various states (as well as simulate two clients communicating through my server), I would like to issue HTTP get requests from within my javascript-based test.
Can anyone provide an example of how to either issue HTTP GET requests directly from within UIAutomation javascript tests, or how to invoke a shell script from within my UIAutomation javascript tests?
FWIW, most of the core objects made available by all browsers are missing within the UIAutomation runtime. Try to use XMLHTTPRequest for example and you will get an exception reporting that it cannot find the variable.
Thanks!
Folks,
I was able to work around this by sending HTTP requests to the iOS client to process and return the results in a UIAlertView. Note that all iOS code modifications are wrapped in #if DEBUG conditional compilation directives.
First, setup your client to send out notifications in the event of a device shake. Read this post for more information.
Next, in your iOS main app delegate add this code:
[[NSNotificationCenter defaultCenter] addObserver:self
selector:#selector(deviceShakenShowDebug:)
name:#"DeviceShaken"
object:nil];
Then add a method that looks something like this:
- (void) deviceShakenShowDebug:(id)sender
{
if (!self.textFieldEnterDebugArgs)
{
self.textFieldEnterDebugArgs = [[[UITextField alloc] initWithFrame:CGRectMake(0, 0, 260.0, 25.0)] autorelease];
self.textFieldEnterDebugArgs.accessibilityLabel = #"AlertDebugArgsField";
self.textFieldEnterDebugArgs.isAccessibilityElement = YES;
[self.textFieldEnterDebugArgs setBackgroundColor:[UIColor whiteColor]];
[self.tabBarController.selectedViewController.view addSubview:self.textFieldEnterDebugArgs];
[self.tabBarController.selectedViewController.view bringSubviewToFront:self.textFieldEnterDebugArgs];
}
else
{
if ([self.textFieldEnterDebugArgs.text length] > 0)
{
if ([self.textFieldEnterDebugArgs.text hasPrefix:#"http://"])
{
[self doDebugHttpRequest:self.textFieldEnterDebugArgs.text];
}
}
}
}
- (void)requestDidFinishLoad:(TTURLRequest*)request
{
NSString *response = [[[NSString alloc] initWithData:((TTURLDataResponse*)request.response).data
encoding:NSUTF8StringEncoding] autorelease];
UIAlertView *resultAlert =
[[[UIAlertView alloc] initWithTitle:NSLocalizedString(#"Request Loaded",#"")
message:response
delegate:nil
cancelButtonTitle:NSLocalizedString(#"OK",#"")
otherButtonTitles:nil] autorelease];
resultAlert.accessibilityLabel = #"AlertDebugResult";
[resultAlert show];
}
This code will add a UITextField to the very top view controller after a shake, slapped right above any navigation bar or other UI element. UIAutomation, or you the user, can manually enter a URL into this UITextField. When you shake the device again, if the text begins with "http" it will issue an HTTP request in code (exercise for the reader to implement doDebugHttpRequest).
Then, in my UIAutomation JavaScript file, I have defined the following two functions:
function httpGet(url, delayInSec) {
if (!delayInSec) delay = 1;
var alertDebugResultSeen = false;
var httpResponseValue = null;
UIATarget.onAlert = function onAlert(alert) {
httpResponseValue = alert.staticTexts().toArray()[1].name();
alert.buttons()[0].tap();
alertDebugResultSeen = true;
}
var target = UIATarget.localTarget();
var application = target.frontMostApp();
target.shake(); // bring up the input field
application.mainWindow().textFields()["AlertDebugArgsField"].setValue(url);
target.shake(); // send back to be processed
target.delay(delayInSec);
assertTrue(alertDebugResultSeen);
return httpResponseValue;
}
function httpGetJSON(url, delayInSec) {
var response = httpGet(url, delayInSec);
return eval('(' + response + ')');
}
Now, in my javascript file, I can call
httpGet('http://localhost:3000/do_something')
and it will execute an HTTP request. If I want JSON data back from the server, I call
var jsonResponse = httpGetJSON('http://localhost:3000/do_something')
If I know it is going to be a long-running call, I call
var jsonResponse = httpGetJSON('http://localhost:3000/do_something', 10 /* timeout */)
I've been using this approach successfully now for several weeks.
Try performTaskWithPathArgumentsTimeout
UIATarget.host().performTaskWithPathArgumentsTimeout("/usr/bin/curl", "http://google.com", 30);
Just a small correction. The answer that suggests using UIATarget.host().performTaskWithPathArgumentsTimeout is an easy way to make a request on a URL in iOS 5.0+, but the syntax of the example is incorrect. Here is the correct way to make this call:
UIATarget.host().performTaskWithPathArgumentsTimeout("/usr/bin/curl", ["http://google.com"], 30);
The "[" around the "args" param is important, and the test will die with an exception similar to the following if you forget the brackets:
Error: -[__NSCFString count]: unrecognized selector sent to instance
Here is a fully working example that hits google.com and logs all the output:
var result = UIATarget.host().performTaskWithPathArgumentsTimeout("/usr/bin/curl", ["http://www.google.com"], 30);
UIALogger.logDebug("exitCode: " + result.exitCode);
UIALogger.logDebug("stdout: " + result.stdout);
UIALogger.logDebug("stderr: " + result.stderr);
+1 for creative use of "shake()". However, that's not an option for some projects, especially those that actually use the shake feature.
Think outside the box. Do the fetching with something else (Python, Ruby, node.js, bash+wget, etc). Then, you can use the pre-canned response and auto-generate the ui-test.js on the fly by including that dynamically generated json payload as the "sample data" into the test. Then you simply execute the test.
In my opinion, the test is the test, leave that alone. The test data you are using, if it's that dynamic, it ought to be separated from the test itself. By doing it this way of fetching / generating JSON, and referencing it from the test, you can update that JSON however often you like, either immediately right before every test, or on a set interval like when you know the server has been updated. I'm not sure you would want to generate it while the test is running, that seems like it would create problems. Taking it to the next level, you could get fancy and use functions that calculate what values ought to be based on other values, and expose them as "dynamic properties" of the data, rather than that math being inside the test, but at that point I think the discussion is more of an academic one rather than the practical one of how.
Apple has recently updated UIAutomation to include a new UIAHost element for performing a task on the Host that is running the instance of Instruments that is executing the tests.