I am trying to use CasperJS to scrape a website that has dynamic content and am failing so far. The website uses a couple of js libaries(e.g. Prototype) to use autocomplete to create content. I am therefore trying to insert some values and trigger events to set off the content creation. Inserting and triggering events works fine, but no content is created.
I am new to CasperJS and found that it has problems with relative paths to the libaries. I suspect that my problem arises from the libaries not loading properly in my page environment. I therefore tried to use injectJs to inject them like this
// ... create casper and do some other stuff
casper.then(function() {
this.echo(this.page.injectJs('http://www.my-website.com/path/to/js1'));
this.echo(this.page.injectJs('http://www.my-website.com/path/to/js2'));
});
Unfortunately this returns false for both of the functions. The path to the script is correct, where else could be the root of this problem?
Thank you very much for any help.
I do not believe CasperJS can handle HTTP requests to inject scripts. This is weird because they actually show this in their FAQ on including jQuery. However, that same FAQ says the following:
You can't inject scripts using the HTTP protocol, you actually have to
use a relative/absolute filesystem path to the script resource.
Just to double check, I tried it myself using Mac OS X and your above code. The results of echo were false when using a file over http and true when it was local.
this is how I Inject Jquery in Casperjs script
casper.options.clientScripts = ["jquery-3.2.1.min.js"]
Jquery file should be on same directory where casperjs script exisit
Related
This question already has answers here:
getting the raw source from Firefox with javascript
(3 answers)
Closed 8 years ago.
I'm not using Selenium to automate testing, but to automate saving AJAX pages that inject content, even if they require prior authentication to access.
I tried
tl;dr: I tried multiple tools for downloading sites with AJAX and gave up because they were hard to work with or simply didn't work. I'm resorting to using Selenium after trying out WebHTTrack (whose GUI wasn't able to start up on my Ubuntu machine + was a headache to provide authentication with in interactive-terminal mode), wget (which didn't download any of the scripts of stylesheets included on my page, see the bottom for what I tried with wget)... and then I finally gave up after a promising post on using a Mozilla XULRunner AJAX scraper called Crowbar simply seg-faulted on me. So...
ended up making my own broken thing in NodeJS and Selenium-WebdriverJS
My NodeJS script uses selenium-webdriver npm module which is "officially supported by the main project" to:
provide login information + do necessary button-clicking & typing for authentication
download all JS and CSS referenced on target page
download target page with original JS/CSS file links change to local file paths
Now when I view my test page locally I see double of many page elements because the target site loads HTML snippets into the page each time it's loaded. I use this to download my target page right now:
var $;
var getTarget = function () {
driver.getPageSource().then(function (source) {
$ = cheerio.load(source.toString());
});
};
var targetHtmlDest = 'test.html';
var writeTarget = function () {
fs.writeFile(targetHtmlDest, $.html());
}
driver.get(targetSite)
.then(authenticate)
.then(getRoot)
.then(downloadResources)
.then(writeRoot);
driver.quit();
The problem is that the page source I get is the already modified page source, instead of the original one. Trying to run alert("x");window.stop(); within driver.executeAsyncScript() and driver.executeScript() does nothing.
Perhaps using Curl to get the page (you can pass authentication in the command) will get you the bare source?
Otherwise you may be able to turn off JavaScript on your test browsers to prevent JS actions from firing.
Here it is described how to check if Firebug is installed by checking if an image of firebug exists: http://webdevwonders.com/detecting-firefox-add-ons/
But it seems to be a bit outdated, cause the images he uses there don't exist anymore in firebug.
the firebug chrome.manifest looks like:
content firebug content/firebug/ contentaccessible=yes
...
but in the whole addon I only find one png now, and that is placed in the rootfolder of the addon. But some other content is accessible, for example: chrome://firebug/content/trace.js
Ho
So, in gerneral:
How do I make an image accessible that resides inside a Firefox SDK Addon?
I program an Addon and I want to make an image ok.png available to all javascripts in Firefox.
I added the image in the data folder and added a chrome.manifest
content response-timeout-24-hours data/
content response-timeout-24-hours data/ contentaccessible=yes
But no way to call it via a URL like
chrome://response-timeout-24-hours/data/ok.png
How do the paths belong together? which is relative to which?
I created a Bug report here.
So if you want to make your add-on detectable you need another approach:
you can use a PageMod to attach a content script that would wait for a
message from your web-app and "respond" by sending another message
back to your app. you would know that if you don't receive the
response, your addon is not installed. check out the documentation for
more details:
https://developer.mozilla.org/en-US/Add-ons/SDK/High-Level_APIs/page-mod
I used this to make my add-on detectable.
I cloned an existing magento 1.7.2 installation on the same server with a test subdomain. The frontend seems to work, and I can login to the admin. The admin menu doesn't work however, no dropdowns, and copying url paths doesn't work either. I've searched online, and most answers date back to 2008 and suggest that it's a rights issue. So I've changed the rights of folders and files to 755 and 644, but still no working menus. The cache (var/cache) is empty.
These menus are javascript generated. The following error message is from the console:
Error: TypeError: Element.addClassName is not a function
To be clear - the solution is not in javascript, but it's something on the server. This install works on the same server in another directory with another domain.
Any ideas how to fix this?
The error
Error: TypeError: Element.addClassName is not a function
indicates some javascript on your page can't call the addClassName method.
The addClassName method is added to element via the prototype javascript framework.
That means its very likely your browser can't download the prototype.js file. Since it can't download this file, the addClassName method is never defined, and you get the error you're seeing.
Look at the source code of your admin pages and find the script tag that includes the version of prototpye shipped with your version of Magento.
<script type="text/javascript" src="http://magento.example.com/js/prototype/prototype.js"></script>
Take the URL from this script tag and load it in your browser.
My guess is you'll get a 404 because the file is missing, or a forbidden error because the file has incorrect permissions, or some other web server error that prevents the file from being shown. It's also possible that the link is pointing to an older domain name that's based on a value configured or cached in Magento.
Track down the source of that problem, and you'll be good to go.
Another reason could be that the skin and CSS rules are not correct for your environment.
I've just moved a site from live to local, and the skin/css/media were configured to a subdomain so I looked in the core_config_data table and updated the URLs
Please check if you have set merge js or css to yes, you can update this via db if you cant do it via menu:
SELECT * FROM core_config_data WHERE path LIKE 'dev%'
Change from 1 to 0 merge_css and merge_js
In my case I have changed the permissions of folder and its recurring files and folder and it started working. Try it once.
I need to load a var by getting JSON from a webservice, so my question is where does this code go? I tried to put it in the content script but XHR would fail there.
Any suggestions?
Starting from Chrome 13 content scripts can also perform XHR requests (before only background pages could). So you can put your code wherever you like.
If it doesn't work then you probably didn't specify domain permissions (or trying to connect to non-80 port, to non-http(s) protocol etc).
I'm trying to develop a firefox extension that inserts additional HTTP header fields into outgoing HTTP requests (to interface with an apache extension i'm concurrently developing).
While I understand the individual components of an extension and understand the basic tutorials that are presented on the web, I'm finding it difficult going from the "Hello World" tutorial extensions, into developing a full blown extension.
The sample code I am wanting to adapt for my purposes is presented at the bottom of Setting HTTP request headers.
I am wondering, where in the extension hierarchy should this code be placed and how is such code called/constructed/activated, will it run automatically when the extension is initialised?
Thanks in advance.
For a basic extension, you would place your code in the chrome/content directory of the extension. You would hook this content into Firefox using an overlay. While overlays are usually xul content (buttons, etc) they can be anything. Including a script tag which would load fire off your Javascript code.
That code is an XPCOM component and goes into a components/<some name>.js file.
You should read up on XPCOM components if you want to dig it, but yes, .js files in components are loaded at startup. Such files contain registration code (starts at the var myModule = { line in that example), which tells Firefox whether the component defined in the file is available upon request or should it be instantiated automatically.
In that example you can see the component getting registered to be notified of the application's startup:
catMgr.addCategoryEntry("app-startup", this.myName, this.myProgID, true, true);
and when handling the app-startup notification it registers itself for the http-on-modify-request notification:
os.addObserver(this, "http-on-modify-request", false);