We Keep Coding

Manifest V3 web extension overwrite response body

How would I overwrite the response body for an image with a dynamic value in a Manifest V3 Chrome extension?
This overwrite would happen in the background, as per the Firefox example, (see below) meaning no attaching debuggers or requiring users to press a button every time the page loads to modify the response.
I'm creating a web extension that would store an image in the extension's IndexedDB storage and then override the response body with that image on requests to a certain image. A redirect to a dataurl: I have it working in a Manifest V2 extension in Firefox via the browser.webRequest.onBeforeRequest api with the following code, but browser.webRequest and MV2 are depreciated in Chrome. In MV3, browser.webRequest was replaced with browser.declarativeNetRequest, but it doesn't have the same level of access, as you can only redirect and modify headers, not the body.
Firefox-compatible example:
browser.webRequest.onBeforeRequest.addListener(
(details) => {
const request = browser.webRequest.filterResponseData(details.requestId);
request.onstart = async () => {
request.write(racetrack);
request.disconnect();
};
},
{
urls: ['https://www.example.com/image.png'],
},
['requestBody', 'blocking']
);
The Firefox solution is the only one that worked for me, albeit being exlusive to Firefox. I attempted to write a POC userscript with xhook to modify the content of a DOM image element, but it didn't seem to return the modified image as expected. Previously, I tried using a redirect to a data URI and an external image, but while the redirect worked fine, the website threw an error that it couldn't load the required resources.
I'm guessing I'm going to have to write a content script that injects a Service Worker (unexplored territory for me) into the page and create a page rule that redirects, say /extension-injected-sw.js to either a web-available script, but I'm not too sure about how to pull that off, or if I'd still be able to have the service worker communicate with the extension, or if that would even work at all. Or is there a better way to do this that I'm overlooking?
Thank you for your time!

Modify <meta> tag with JS (chrome extension) on response receiving

I have a Chrome extension that adds a panel to the page in the floating iframe (on extension button click). There's certain JS code that is downloaded from 3rd party host and needs to be executed on that page. Obviously there's XSS issue and extension needs to comply with content security policies for that page.
Previously I had to deal with CSP directives that are passed via request headers, and was able to override those via setting a hook in chrome.webRequest.onHeadersReceived. There I was adding my host URL to content-security-policy headers. It worked. Headers were replaced, new directives applied to the page, all good.
Now I discovered websites that set the CSP directives via <meta> tag, they don't use request headers. For example, app pages in iTunes https://itunes.apple.com/us/app/olympics/id808794344?mt=8 have such. There is also an additional meta tag with name web-experience-app/config/environment (?) that somewhat duplicates the values that are set in content of tag with http-equiv="Content-Security-Policy".
This time I am trying to add my host name into meta tag inside chrome.webNavigation.onCommitted or onCompleted events listeners (JS vanilla via chrome.tabs.executeScript). I also experimented with running the same code from the webrequest's onCompleted listener (at the last step of lifecycle according to https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest).
When I inspect the page after its load - I see the meta tags have changed. But when I click on my extension to start loading iframe and execute JS - console prints the following errors:
Refused to frame 'https://myhost.com' because it violates the following Content Security Policy directive: "frame-src 'self' *.apple.com itmss: itms-appss: itms-bookss: itms-itunesus: itms-messagess: itms-podcasts: itms-watchs: macappstores: musics: apple-musics:".
I.e. my tag update was not effective.
I have several questions: first, do I do it right? Am I doing the update at the proper event? When is the data content from meta tags being read in the page lifecycle? Will it be auto-applied after tag content change?

As of March 2018 Chromium doesn't allow to modify the responseBody of the request. https://bugs.chromium.org/p/chromium/issues/detail?id=487422#c29
"WebRequest API: allow extensions to read response body" is a ticket from 2015. It is not on a path of getting to be resolved and needs some work/help.
--
Firefox has the webRequest filter implementation that allows to modify the response body before the page's meta directives are applied.
https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest/filterResponseData
BUT, my problem is focused on fixing the Chrome extension. Maybe Chrome picks this up one day.
--
In general, the Chrome's extension building framework seems like not a reliable path of building a long term living software; with browser vendors changing the rules frequently, reacting to newly discovered threats, having no up-to-date supported cross-browser standard.
--
In my case, the possible way around this issue can be to throw all the JS code into the extension's source base. Such that there's no 3-rd party to connect to fetch and execute the JS (and conflict with/violate the CSP rules). Haven't explored this yet, as I expected to reuse the code & interactive components I am using in my main browser application.

I've been interested in the same things and here are a few aspects that perhaps could help:
chrome.debugger extension API with Fetch (or Network) domain can be used to modify responseBody: https://chromedevtools.github.io/devtools-protocol/tot/Fetch/
This is an example implementation: https://github.com/mr-yt12/Debugger-API-Fetch-example-Chrome-Extension
However, I'm facing a problem of Fetch.requestPaused not firing on the first page load: Chrome Extension Debugger API, Fetch domain attaches/enables too late for the response body to be intercepted
And I haven't found a solution to this yet, besides first redirecting the request to 'http://google.com/gen_204' and then updating the tab. But this creates flicker and also I'm not sure if it's possible to redirect the request like this with manifest v3.
When using debugger API, Chrome shows a warning at the top, which changes the page's size and doesn't go away (perhaps it goes away 5 seconds after the debugger is detached and also if the user clicks "cancel"). This means that it's mostly only good for personal use or when distributing as a developer mode extension. Using --silent-debugger-extension-api flag with Chrome disables this warning.
I've tried injecting my script at document_start (when the meta tag is not yet created). Then I used mutationObserver (also tried other methods) to wait for the meta tag, to modify it before it's applied or fully created. Somehow it succeeded once or a few times, but could be a coincidence or a wrong interpretation of results. Perhaps it's worth experimenting with it.
Another idea (but I think I didn't make it work, but perhaps it's possible) is to use window.stop() at document_start and then rewrite the html content programmatically. This needs more researching.
It seems that meta tag CSP is applied once the meta tag is created (or while it's being created), and then there is no way to cancel what's been applied. It should be researched more on how to prevent it from applying or modifying it before it's fully created or applied.

Inject content script into iFrame across domains (but with optional permissions)? [duplicate]

Content Script can be injected programatically or permanently by declaring in Extension manifest file. Programatic injection require host permission, which is generally grant by browser or page action.
In my use case, I want to inject gmail, outlook.com and yahoo mail web site without user action. I can do by declaring all of them manifest, but by doing so require all data access to those account. Some use may want to grant only outlook.com, but not gmail. Programatic injection does not work because I need to know when to inject. Using tabs permission is also require another permission.
Is there any good way to optionally inject web site?

You cannot run code on a site without the appropriate permissions. Fortunately, you can add the host permissions to optional_permissions in the manifest file to declare them optional and still allow the extension to use them.
In response to a user gesture, you can use chrome.permission.request to request additional permissions. This API can only be used in extension pages (background page, popup page, options page, ...). As of Chrome 36.0.1957.0, the required user gesture also carries over from content scripts, so if you want to, you could add a click event listener from a content script and use chrome.runtime.sendMessage to send the request to the background page, which in turn calls chrome.permissions.request.
Optional code execution in tabs
After obtaining the host permissions (optional or mandatory), you have to somehow inject the content script (or CSS style) in the matching pages. There are a few options, in order of my preference:
Use the chrome.declarativeContent.RequestContentScript action to insert a content script in the page. Read the documentation if you want to learn how to use this API.
Use the webNavigation API (e.g. chrome.webNavigation.onCommitted) to detect when the user has navigated to the page, then use chrome.tabs.executeScript to insert the content script in the tab (or chrome.tabs.insertCSS to insert styles).
Use the tabs API (chrome.tabs.onUpdated) to detect that a page might have changed, and insert a content script in the page using chrome.tabs.executeScript.
I strongly recommend option 1, because it was specifically designed for this use case. Note: This API was added in Chrome 38, but only worked with optional permissions since Chrome 39. Despite the "WARNING: This action is still experimental and is not supported on stable builds of Chrome." in the documentation, the API is actually supported on stable. Initially the idea was to wait for a review before publishing the API on stable, but that review never came and so now this API has been working fine for almost two years.
The second and third options are similar. The difference between the two is that using the webNavigation API adds an additional permission warning ("Read your browsing history"). For this warning, you get an API that can efficiently filter the navigations, so the number of chrome.tabs.executeScript calls can be minimized.
If you don't want to put this extra permission warning in your permission dialog, then you could blindly try to inject on every tab. If your extension has the permission, then the injection will succeed. Otherwise, it fails. This doesn't sound very efficient, and it is not... ...on the bright side, this method does not require any additional permissions.
By using either of the latter two methods, your content script must be designed in such a way that it can handle multiple insertions (e.g. with a guard). Inserting in frames is also supported (allFrames:true), but only if your extension is allowed to access the tab's URL (or the frame's URL if frameId is set).

I advise against using declarativeContent APIs because they're deprecated and buggy with CSS, as described by the last comment on https://bugs.chromium.org/p/chromium/issues/detail?id=708115.
Use the new content script registration APIs instead. Here's what you need, in two parts:
Programmatic script injection
There's a new contentScripts.register() API which can programmatically register content scripts and they'll be loaded exactly like content_scripts defined in the manifest:
browser.contentScripts.register({
matches: ['https://your-dynamic-domain.example.com/*'],
js: [{file: 'content.js'}]
});
This API is only available in Firefox but there's a Chrome polyfill you can use. If you're using Manifest v3, there's the native chrome.scripting.registerContentScript which does the same thing but slightly differently.
Acquiring new permissions
By using chrome.permissions.request you can add new domains on which you can inject content scripts. An example would be:
// In a content script or options page
document.querySelector('button').addEventListener('click', () => {
chrome.permissions.request({
origins: ['https://your-dynamic-domain.example.com/*']
}, granted => {
if (granted) {
/* Use contentScripts.register */
}
});
});
And you'll have to add optional_permissions in your manifest.json to allow new origins to be requested:
{
"optional_permissions": [
"*://*/*"
]
}
In Manifest v3 this property was renamed to optional_host_permissions.
I also wrote some tools to further simplify this for you and for the end user, such as
webext-domain-permission-toggle and webext-dynamic-content-scripts. They will automatically register your scripts in the next browser launches and allow the user the remove the new permissions and scripts.

Since the existing answer is now a few years old, optional injection is now much easier and is described here. It says that to inject a new file conditionally, you can use the following code:
// The lines I have commented are in the documentation, but the uncommented
// lines are the important part
//chrome.runtime.onMessage.addListener((message, callback) => {
// if (message == “runContentScript”){
chrome.tabs.executeScript({
file: 'contentScript.js'
});
// }
//});
You will need the Active Tab Permission to do this.

How to start two or more custom URL Protocol from Javascript

I have an old html page that creates a script file and executes it using:
fsoObject = new ActiveXObject("Scripting.FileSystemObject")
wshObject = new ActiveXObject("WScript.Shell")
I am trying to modify it and make it usable also from other browsers. If you know the answer stop reading and please answer. If there is no quick answer, here is the description of my attempts. I was successful in doing the job, but only when the script is shorter than 2000 characters. I need help for scripts longer than 2000 characters.
The webpage is for internal use only, so it is easy for me to create a custom URL protocol on each computer that runs a VBScript file from a network drive.
I created my custom URL Protocol that starts a VBScript file like this:
Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\MyUrlProtocol]
"URL Protocol"=""
#="Url:MyUrlProtocol"
"UseOriginalUrlEncoding"=dword:00000001
[HKEY_CLASSES_ROOT\MyUrlProtocol\DefaultIcon]
#="C:\\Windows\\System32\\WScript.exe"
[HKEY_CLASSES_ROOT\MyUrlProtocol\shell]
[HKEY_CLASSES_ROOT\MyUrlProtocol\shell\open]
[HKEY_CLASSES_ROOT\MyUrlProtocol\shell\open\command]
#="C:\\Windows\\System32\\WScript.exe \"X:\\MyUrlProtocol.vbs\" \"%1\""
In MyUrlProtocol.vbs I have this:
MsgBox "The length of the link is " & Len(WScript.Arguments(0)) & " characters"
MsgBox "The content of the link is: " & WScript.Arguments(0)
When I click on click me I see two messages, so everything works well (tested with Chrome and IE in Windows 7.)
It works also when I execute document.getElementById("test").click()
I thought this could be the solution: I would pass the text of the script to the VBS static script, which would create the dynamic script and run it, but with this system I can't pass more than ~2000 characters.
So I tried to split the text of the script in chunks smaller than 2000 characters and simulate several clicks on the link, but only the first one works.
So I tried with xmlhttp.open("GET","MyUrlProtocol:test",false);, but Chrome says Cross origin requests are only supported for HTTP.
Is it possible to pass more than 2000 characters to a VBScript script via a custom URL protocol?
If not, is it possible to call several custom URL protocols in sequence?
If not, is there another way to create a script file and run it from Javascript?
EDIT 1
I found a solution, but in Chrome only works when it likes, so I'm back to square one.
The code below in IE executes the script 4 times (correct), but in Chrome only the first execution runs.
If I change it to delay += 2000, then Chrome usually runs the script 2 times, but sometimes 1 and sometimes 3 or even 4 times.
If I change it to delay += 10000, then it usually runs the script 4 times, but sometimes misses one.
The function is always executed 4 times, both in Chrome and IE. What is weird is that the sr.click() sometimes does nothing and the function execution continues.
<HTML>
<HEAD>
<script>
var delay;
function runScript(text) {
setTimeout(function(){runScript2(text)}, delay);
delay += 100;
}
function runScript2(text) {
var sr = document.getElementById('scriptRunner');
sr.href='intelliclad:'+text;
sr.click();
}
function test(){
delay = 0;
runScript("uno");
runScript("due");
runScript("tre");
runScript("quattro");
}
</script>
</HEAD>
<BODY>
<input type="button" value="Run test" onclick="test()">
scriptRunner
</BODY>
</HMTL>
EDIT 2
I tried with Luke's suggestion of setting the next timeout from inside the call back but nothing changed (IE works always, Chrome whenever it likes).
Here is the new code:
var scripts;
var delay = 2000;
function runScript() {
var sr = document.getElementById('scriptRunner');
sr.href = 'intelliclad:' + scripts.shift();
sr.click();
if(scripts.length)
setTimeout(function() {runScript()}, delay);
}
function test(){
scripts = ["uno", "due", "tre", "quattro"];
runScript();
}
Some background: The page asks for the shape of a panel, which can be just a few parameters [nfaces=1, shape1='square', width1=100] or hundreds of parameters for panels with many faces, many slots, many fasteners, etc. After asking for all the parameters a script for our internal 3D CAD (which can be larger than 20KB) is generated and the CAD is started and asked to execute the script.
I would like to do all on the client side, because the page is served by a Domino web server, which can't even dream of managing such a complex script.

I didn't read your whole post...have an answer:
I too wish that custom url protocols can handle long urls. They simply do not. IE is even worse as some OSs only accept 800 chars.
So, here's the solution:
For long urls, only pass a single use token. The vbscript uses the token
and does a url get to your web server to get all of the data.
This is the only way I've been able to successfully pass lots of data around. If you ever find a clearer solution, please remember to post it here.
Update:
Note that this is the best way I have found to deal with the url protocol limitations. I too wish this was not necessary. This does work and works well.
You mentioned Dominos, so possibly you need something in a POS environment... I create a web based POS system, so we could face a lot of the same issues.
Suppose you want a custom url to print a pdf to the default printer without the annoying popup window. We need to do this thousands of times a day...
When building the web page, add the print button which when pressed calls the custom url: myproto://printpdf?id=12345&tocken=onetimetoken
this will execute your vbscript on the local desktop
in your vbscript, parse the arguments and react. In this case, your command is printpdf and the id is 123456 and you have a onetime tocken key.
have the vb script to an https get to: https://mydomain.com/APIs/printpdf.whatever?id=12345&key=onetimetoken
check the credentials based on the ip address and token, if all aligns, then return the contents of the pdf (you may want to convert the pdf to a byte array string)
now the vbscript has the pdf, assemble it and write it to a temp folder then execute a silent pdf print command (I use Sumatra PDF http://blog.kowalczyk.info/software/sumatrapdf/free-pdf-reader.html)
mission accomplished.
Since I do know what you what to do in your custom url and the general workflow, I can only describe how I've solved the sort url issue.
Using this technique, the possibilities are limitless. You have full control over the local computer running the web browser, you have a onetime use token which grants access to a web API with can return any sort of information you program.
You could write a custom url protocol to turn on the pizza oven if you wanted :)
If you are not able to create the server side code which is listening for vbscript's get request then this would not work.
You might be able to pass the data from the browser to the vbscript using the clipboard.
Update 2:
Since in this case the data is on the client (one single form can define hundreds of parameters), the server API doesn't know what to answer to the vb script request. So the workflow described above must be preceded by these two steps:
The onkeypress event executes a submit to send the current parameters to the server
The server replies with the refreshed form, adding to the body onload a call to a function which uses another submit to call the custom url, as described on point 1 listed above.
Update 3:
stenci, what you've added (in Update 2) will work. I would do it like this:
user presses a button saying I'm done editing the form
ajax post the form to the server
the server saves the data and attaches unique key to the datastore
the server returns the key to ajax callback function
now the client has a single use key and invokes the url schema passing the key
vbscript does an https get to the server and passes the key
server returns the data to the vbscript
It is a bit long winded. Once coded it will work like a charm.
The only other alternative I can see is to copy the form data to the clipboard using something like: http://zeroclipboard.org/
and then in vbscript see if you can read the clipboard like: Use clipboard from VBScript

How about creating an iFrame for each instance?
Something like this:
function runScript(text) {
var iframe = document.createElement('iframe');
iframe.src = 'intelliclad:'+text;
document.body.appendChild(iframe);
}
function test(){
runScript("uno");
runScript("due");
runScript("tre");
runScript("quattro");
}
You can then use css styling to make these iframes transparent / hidden.

You might not like this answer, but I've used this method in the past and it works.
Instead of relying on ActiveX, consider using a Java Applet, and JNI.
Basically, you have to make sure the native scripts you want to run are available on your client machine, along with a JNI wrapper.
The applet will have to be at least self signed, for the browser to allow it to load and access a native library. Once the JNI libraries are loaded, you can easily call methods from the page / applet.
As a consequence of using Java, you could possibly use the same applet for windows as well as linux clients, provided of course you have native libraries present on the respective clients.
This series of articles talks about precisely your problem : http://www.javaworld.com/article/2076775/java-security/escape-the-sandbox--access-native-methods-from-an-applet.html
P.S the article is really old, but the concept remains unchanged.

Is there a way to mitigate downloading of resources (images/css and js files) with Javascript?

I have a html page on my localhost - get_description.html.
The snippet below is part of the code:
<input type="text" id="url"/>
<button id="get_description_button">Get description</button>
<iframe id="description_container" src="#"/>
When the button is clicked the src of the iframe is set to the url entered in the textbox. The pages fetched this way are very big with lots of linked files. What I am interested in the page is a block of text contained in a <div id="description"> element.
Is there a way to mitigate downloading of resources linked in the page that loads into the iframe?
I don't want to use curl because the data is only available to logged in users and the steps to take with curl to get the content is too complicated. The iframe is simple as I use this on a box which sends the right cookies to identify the request as coming from a logged in user, but the problem is that it is very wasteful to get nearly 1 MB of data to keep 1 KB of it and throw out the rest.
Edit
If the proposed method just works in Firefox it is fine, so I added Firefox tag. Also, it is possible that the answer actually is from the realm of Firefox add-on techniques, so I added that tag as well.
The problem is not that I cannot get at what I'm looking for, rather, the problem is the easy iframe method is wasteful.
I know that Firefox does allow loading only the text of a page. If you open a page and press Ctrl+U you are taken to 'view page source' window, There links behave as normal and are clickable, if you click on a link in source view, the source of the new page is loaded into the view source window, without the linked resources being downloaded, exactly what I'm trying to get. But I don't know how to access this behaviour.
Another example is the Adblock add-on. It somehow kills elements before they get loaded. With plain Javascript this is not possible. Because it only is triggered too late to intervene in good time.

The Same Origin Policy forbids any web page to access contents of any other web page in a different domain so basically you cannot do that.
However it seems that with some browsers it is allowed to access web pages content if you are trying to access it from a local web page which seems to be your case.
Safari, IE 6/7/8 are browser that allow a local web page to do so via XMLHttpRequest (source: Google Browser Security Handbook) so you may want to choose to use one of those browsers to do what you need (note that future versions of those browsers may not allow to do so anymore).
A part from this solution I only see two possibities:
If the web pages you need to fetch content from are somehow controlled by you, you can create a simpler interface to let other web pages to get the content you need (for example allowing JSONP requests).
If the web pages you need to fetch content from are not controlled by you the only solution I see is to fetch content server side logging in from the server directly (I know that you don't want to do so, but I don't see any other possibility if the previous I mentioned are not practicable)
Hope it helps.

Actually I've seen Cross Domain jQuery .load request before, here: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/
The author claims that codes like these found on that page
$('#container').load('http://google.com'); // SERIOUSLY!
$.ajax({
url: 'http://news.bbc.co.uk',
type: 'GET',
success: function(res) {
var headline = $(res.responseText).find('a.tsh').text();
alert(headline);
}
});
// Works with $.get too!
would work. (The BBC code might not work because of the recent redesign, but you get the idea)
Apparently it is using YQL wrapped into a jQuery plugin to do the trick. Now I cannot say I fully understand what he is doing there but it appears to work, and fits the bill. Once you load the data I suppose it is a simple matter of filtering out the data that you need.
If you prefer something that works at the browser level, may I suggest Mozilla's Jetpack framework for lightweight extensions. I've not yet read the documentations in its entirety but it should contain the APIs needed for this to work.

There are various ways to go about this in AJAX, I'm going to show the jQuery way for brevity as one option, though you could do this in vanilla JavaScript as well.
Instead of an <iframe> you can just use a container, let's say a <div> like this:
<div id="description_container"></div>
Then to load it:
$(function() {
$("#get_description_button").click(function() {
$("#description_container").load($("input").val() + " #description");
});
});
This uses the .load() method which takes a string in this format: .load("url selector"), then takes that element in the page and places it's content inside the container you're loading, in this case #description_container.
This is just the jQuery route, mainly to illustrate that yes, you can do what you want, but you don't have to do it exactly like this, just showing the concept is getting what you want from an AJAX request, rather than in an <iframe>.

Your description sounds like you are fetching pages from the same domain (you said that you need to be logged in and have session credentials) so have you tried to use async request via XMLHttpRequest? It might complain if the html on a page is particularly messed up but you chould still be able to get raw text via .responseText and extract what you need with a regex.