I'm trying to develop a basic Firefox Addon with the "new" WebExtensions system.
I'd like to
Extract some text from a web page (not owned by me)
evaluate it using a remote website
post in the same page the result
The problem is how to make the web request with the addon (point 2). I found that I could use XMLHttpRequest, but as I imagined for security reasons I can't get access to remote paths.
That is because (I guess) the javascript code is run inside the page, even though I had thought that an addon would be... external.
Of course the result would be inside the page, but I assume the addon could work as a proxy to make this request. That said I have no idea how and what should I do.
I don't want to use some strange trick (like removing some security control), I'd like to do it the "right" way.
What I also don't understand is if the addons are bounded to run within the page they are made for.
EDIT: OK, turns out the chrome docs is actually better than the mozilla one. To actually use the XHR to cross-site req you have to put an additional line of code in your manifest.
{...
"permissions": [
"http://random.com/"
],
}..
I'm still not sure if this is the proper way to do what I aim to though.
To actually use the XHR to cross-site req you have to put an additional line of code in your manifest.
{...
"permissions": [
"http://random.com/"
],
}
Related
I've encountered a paywall and I'm trying to bypass it using javascript in the console. I did some research and found a few different approaches, one of which is changing the requestheader in order to make a given website believe that you got there through a twitter link (thus allowing you to view the content for free). The function I use aims to change the referer by listening to the onBeforeSendHeaders event as specified on https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onBeforeSendHeaders. It looks like the following (NOTE: This function is typed and executed directly inside of the devtools console):
function setReferer(x){
x.requestHeaders = x.requestHeaders.filter(function(header){
if(header.name === 'Referer'){
return false
return true
)}
x.requestheaders.push(
{
"name: "Referer",
"value": "https://t.co/" //Twitter website
}
return {requestHeaders: x.requestHeaders};
}
//this example uses chrome browser
chrome.webRequest.onBeforeSendHeaders.addListener(setReferer,
{
urls: ["<all_urls>"],
types: ["main_frame"], },
["requestHeaders", "blocking", "extraHeaders"] //extraHeaders meant to bypass CORS protocol
);
Unfortunately upon refreshing the window, this approach gives me folllowing error:
GET <some_url> net:ERR_BLOCKED_BY_CLIENT
Behind this error is the URL to the source code of the article, which I was able to load and copy into word, so I got the article I was looking for anyway. However I wasn't able to view it inside of the browsers main frame. Note that I am doing this only for the purpose of polishing my coding skills. I am trying to get a better understanding of the more complicated facets of the HTTP protocol, especially the way headers get sent clientside and interpreted serverside. If anyone knows more about the subject or knows / has a resource that he or she wants to share, this would me greatly appreciated!
I try to get the raw response body inside a Web Extension using Firefox 55.0.3.
Only "solutions" I have seen for now:
Repeat the request (I absolutly don't want to repeat the request)
Using Javascript to get innerHTML attribute of HTML tags such as head and body (tell me if I'm wrong, but with a solution like that I will not always have the whole content, for example I will get nothing in case of response without HTML. So it will never be the real raw response and in some case it will simply not work.)
Also, I saw this answer for Chrome (from 2015) using the debugger, but I wasn't able to do it with Firefox. This kind of solutions are interesting, I read Mozilla documentation about devtools but I didn't find a way of using the network tab of webtools interface with Javascript inside a Web Extension.
To give you more details, my goal is to intercept the full request and response from server (header and body). This is not a problem to do it, except for the response body.
Here an example of code to get the request body: (background script)
browser.webRequest.onBeforeRequest.addListener(
function (e) {
console.log(e);
},
{urls: ["http://*/*", "https://*/*"]},
["requestBody"]
)
Here some documentations that I used (there is more, but these links are all official):
Mozilla documentation about Web Extension
Intercept HTTP requests
webRequest
webRequest.onHeadersReceived
webRequest.onBeforeRequest
webRequest.onBeforeSendHeaders
Here some examples of Web Extensions.
Any ideas, solutions or even explainations "why this is not possible" are welcome, thank you in advance for your time !
Cheers++
This is now available, as of Firefox 57:
browser.webRequest.filterResponseData allows you to add a listener via browser.webRequest.onBeforeRequest which receives, and allows you to modify the response.
You can see an example in the Mozilla github webextensions-examples repo
Firefox 57 is going to provide the API browser.webRequest.filterResponseData. This doesn't seem to be documented yet, but you can look through bug 1255894 for details.
Why is this not possible?
For the simple reason that WebRequest was ported over from Chrome extensions, where this is explicitly impossible.
Requests for such functionality (to edit, or just to read) has been around for a very long time (since 2011 and 2015 respectively); they are challenging from both the security perspective and technical perspective, however a principal agreement that read access is a good idea is there.
However, it's simply not yet implemented. Rob W has been doing some work in this direction but it's not done yet.
Perhaps Firefox has a different implementation?
A cursory glance on Mozilla bugtracker doesn't find any bugs on providing this functionality. So, it's not likely that the implementation will diverge anytime soon.
Any workarounds?
Well, only the debugger-level access can touch actual response data.
Since debugger is not implemented in the WebExtension platform, only a devtools.network-using extension can access it - and only while Dev Tools are open for the tab making said request, which is the main limitation of devtools.* APIs.
I want to create a custom profiler for Javascript as a Chrome DevTools Extension. To do so, I'd have to instrument all Javascript code of a website (parse to AST, inject hooks, generate new source). This should've been easily possible using chrome.devtools.inspectedWindow.reload() and its parameter preprocessorScript described here: https://developer.chrome.com/extensions/devtools_inspectedWindow.
Unfortunately, this feature has been removed (https://bugs.chromium.org/p/chromium/issues/detail?id=438626) because nobody was using it.
Do you know of any other way I could achieve the same thing with a Chrome Extension? Is there any other way I can replace an incoming Javascript source with a changed version? This question is very specific to Chrome Extensions (and maybe extensions to other browsers), I'm asking this as a last resort before going a different route (e.g. dedicated app).
Use the Chrome Debugging Protocol.
First, use DOMDebugger.setInstrumentationBreakpoint with eventName: "scriptFirstStatement" as a parameter to add a break-point to the first statement of each script.
Second, in the Debugger Domain, there is an event called scriptParsed. Listen to it and if called, use Debugger.setScriptSource to change the source.
Finally, call Debugger.resume each time after you edited a source file with setScriptSource.
Example in semi-pseudo-code:
// Prevent code being executed
cdp.sendCommand("DOMDebugger.setInstrumentationBreakpoint", {
eventName: "scriptFirstStatement"
});
// Enable Debugger domain to receive its events
cdp.sendCommand("Debugger.enable");
cdp.addListener("message", (event, method, params) => {
// Script is ready to be edited
if (method === "Debugger.scriptParsed") {
cdp.sendCommand("Debugger.setScriptSource", {
scriptId: params.scriptId,
scriptSource: `console.log("edited script ${params.url}");`
}, (err, msg) => {
// After editing, resume code execution.
cdg.sendCommand("Debugger.resume");
});
}
});
The implementation above is not ideal. It should probably listen to the breakpoint event, get to the script using the associated event data, edit the script and then resume. Listening to scriptParsed and then resuming the debugger are two things that shouldn't be together, it could create problems. It makes for a simpler example, though.
On HTTP you can use the chrome.webRequest API to redirect requests for JS code to data URLs containing the processed JavaScript code.
However, this won't work for inline script tags. It also won't work on HTTPS, since the data URLs are considered unsafe. And data URLs are can't be longer than 2MB in Chrome, so you won't be able to redirect to large JS files.
If the exact order of execution of each script isn't important you could cancel the script requests and then later send a message with the script content to the page. This would make it work on HTTPS.
To address both issues you could redirect the HTML page itself to a data URL, in order to gain more control. That has a few negative consequences though:
Can't reload page because URL is fixed to data URL
Need to add or update <base> tag to make sure stylesheet/image URLs go to the correct URL
Breaks ajax requests that require cookies/authentication (not sure if this can be fixed)
No support for localStorage on data URLs
Not sure if this works: in order to fix #1 and #4 you could consider setting up an HTML page within your Chrome extension and then using that as the base page instead of a data URL.
Another idea that may or may not work: Use chrome.debugger to modify the source code.
Hey everyone, I'm working on a widget for Apple's Dashboard and I've run into a problem while trying to get data from my server using jquery's ajax function. Here's my javascript code:
$.getJSON("http://example.com/getData.php?act=data",function(json) {
$("#devMessage").html(json.message)
if(json.version != version) {
$("#latestVersion").css("color","red")
}
$("#latestVersion").html(json.version)
})
And the server responds with this json:
{"message":"Hello World","version":"1.0"}
For some reason though, when I run this the fields on the widget don't change. From debugging, I've learned that the widget doesn't even make the request to the server, so it makes me think that Apple has some kind of external URL block in place. I know this can't be true though, because many widgets phone home to check for updates.
Does anyone have any ideas as to what could be wrong?
EDIT: Also, this code works perfectly fine in Safari.
As requested by Luca, here's the PHP and Javascript code that's running right now:
PHP:
echo $_GET["callback"].'({"message":"Hello World","version":"1.0"});';
Javascript:
function showBack(event)
{
var front = document.getElementById("front");
var back = document.getElementById("back");
if (window.widget) {
widget.prepareForTransition("ToBack");
}
front.style.display = "none";
back.style.display = "block";
stopTime();
if (window.widget) {
setTimeout('widget.performTransition();', 0);
}
$.getJSON('http://nakedsteve.com/data/the-button.php?callback=?',function(json) {
$("#devMessage").html(json.message)
if(json.version != version) {
$("#latestVersion").css("color","red")
}
$("#latestVersion").html(json.version)
})
}
In Dashcode click Widget Attributes then Allow Network Access make sure that option is checked. I've built something that simply refused to work, and this was the solution.
Cross-domain Ajax requests ( Using the XMLHttpRequest / ActiveX object ) are not allowed in the current standard, as per the W3C spec:
This specification does not include
the following features which are being
considered for a future version of
this specification:
Cross-site XMLHttpRequest;
However there's 1 technique of doing ajax requests cross-domain, JSONP, by including a script tag on the page, and with a little server configuration.
jQuery supports this, but instead of responding on your server with this
{"message":"Hello World","version":"1.0"}
you'll want to respond with this:
myCallback({"message":"Hello World","version":"1.0"});
myCallback must be the value in the "callback" parameter you passed in the $.getJSON() function. So if I was using PHP, this would work:
echo $_GET["callback"].'({"message":"Hello World","version":"1.0"});';
Apple has some kind of external URL block in place.
In your Info.plist you need to have the key AllowNetworkAccess set to true.
<key>allowNetworkAccess</key>
<true/>
Your code works in Safari because it is not constrained in the dashboard sever and it is not standards complient in that it DOES allow cross site AJAX. FF IS standards complient in that it DOES NOT allow cross site ajax.
If you are creating a dashboard widget, why don't you use the XMLHttpRequest Setup function in the code library of DashCode. Apple built these in so you don't need to install 3rd party JS libraries. I'm not sure about JSON support but perhaps starting here will lead you in a better direction.
So another solution is to create your own server side web service where you can control the CORS of, the users web browser can't access another site, but if you wrap that other site in your own web service (on the same domain) then it does not cause an issue.
Interesting that it works in Safari. As far as I know to do x-domain ajax requests you need to use the jsonp dataType.
http://docs.jquery.com/Ajax/jQuery.getJSON
http://bob.pythonmac.org/archives/2005/12/05/remote-json-jsonp/
Basically you need to add callback=? to your query string and jquery will automatically replace it with the correct method eg:
$.getJSON("http://example.com/getData.php?act=data&callback=?",function(){ ... });
EDIT: put the callback=? bit at the end of the query string just to be safe.
I understand (I think) that XmlHttpRequest objects adhere to the "same-domain" policy. However, I want to create a simple (POC) local html file that downloads XML from a web server and does something with it (let's start with a simple "alert()").
Is it possible at all? Do I need a special Firefox config option?
The server from which I'm trying to download is not under my control (it's actually a Google API).
My simple attempt is the code from Mozilla's "Using XMLHttpRequest" page. It returns an error I don't really understand from the "send" method.
Disclaimer: I'm mainly a C/C++ developer - never done any serious JS programming, never tried using these APIs.
XMLHttpRequest actually adheres to a much stricter implementation of the same domain policy: while you can set the document.domain property to allow JavaScript served from two sub-domains to talk with each other, you can't do that with XMLHttpRequestObject. In your case, going to a completely different domain, you couldn't do that with JavaScript either.
There are a couple of options. First, you can use a reverse proxy to make it appear that the external site is a sub-domain of your site. Take a look at Apache's mod_proxy, in particular ProxyPassReverse
Another alternative is to return data as a JSON object: <script src="foo"> can retrieve whatever data it wants from wherever it wants. Downside of this is that it's not (easily) repeatable (as in multiple requests from same page).
I also recommend that you Google for "google mashups". Most of these live on the "googlemashops.com" domain, which makes implementation a lot easier. A few live outside that domain, and may give you some ideas.
Edit: rather than use the XMLHttpRequest object directly, I recommend going through a third-party library such as prototype.js
If the XML that you're trying to retrieve is returned by one of Google's JS APIs, then there's no need for XmlHttpRequest(since that can only be used on the same domain as your page anway).
So in the case of using a Google API, such as the Maps one, usually begin by adding a reference to their common API somewhere on your page:
<script type="text/javascript" src="http://www.google.com/jsapi?key=your_google_api_key"></script>
Then add a reference to the specific API(s) you plan to use to your page:
<script type="text/javascript">google.load("maps", "2");</script>
Now you can call the various functions provided by that API:
<script type="text/javascript">
function initialize() {
var map = new google.maps.Map2(document.getElementById("map"));
map.setCenter(new google.maps.LatLng(41.86, 87.68), 13);
}
google.setOnLoadCallback(initialize);
</script>
No XmlHttpRequest required :)
You can use JSONP to do this. I do it here using jQuery and PHP. Basically I use the PHP proxy to wrap the JSON reply so jQuery can handle it. It's under BSD.
Alternatively try using IE8. If you are running from disk and not a website IE8 will ignore all normal domain restrictions and get the data you want.
With jQuery (and presumably XHR, but I try not to use it directly) you can happily do cross-domain requests, provided that you do not specify unusual headers or non-GET methods. If you do want to use these, you must have control of the server to enable OPTIONS requests.
See https://developer.mozilla.org/En/HTTP_access_control for details.