We Keep Coding

playwright disable caching of webpage so I can fetch new elements after scrolling down

I'm using playwright (Firefox browser) to scrape some websites. Many of the websites load more content when I scroll down the page. The problem is that the new content loaded is not being picked up by the await page.$$("") methods.
But If I do a document.querySelectorAll("") on the console after the scroll, then I am able to get the newer content as well.
I see that puppeteer has a setting page.setCacheEnabled(enabled) that allows for disabling cache, but I can't find a similar thing in playwright.

You are quite correct that there is no method like setCacheEnabled in playwright. One workaround is to set up a route for all requests:
page.route('**', route => route.continue());
You can see here that:
Enabling routing disables http cache.
Which should accomplish the same thing.

We had the same issue where Playwright 'remembers' the state of the browser from one session to another. The workaround that worked for us was to delete the test-data-dir directory before running the script.
playwright creates this directory once you run it. So, we have this script defined in our package.json
"cleanup:chrome": "rm -rf ./path/to/test-data-dir/* || true",
And then run playwright as usual

How to Change Firefox Proxy Settings Programmatically?

I'm launching Firefox via command line and I'd like to launch a specific Firefox Profile with a proxy. According to this answer on Stackoverflow, Firefox proxy settings are stored in pref.js in the Firefox Profile folder and it is necessary to edit this file to launch FF with a proxy.
I've edited the file as follows:
user_pref("network.proxy.ftp", "1.0.0.1");
user_pref("network.proxy.ftp_port", 00000);
user_pref("network.proxy.gopher", "1.0.0.1");
user_pref("network.proxy.gopher_port", 00000);
user_pref("network.proxy.http", "1.0.0.1");
user_pref("network.proxy.http_port", 22222);
user_pref("network.proxy.no_proxies_on", "localhost, 1.0.0.1");
user_pref("network.proxy.socks", "1.0.0.1");
user_pref("network.proxy.socks_port", 00000);
user_pref("network.proxy.ssl", "1.0.0.1");
user_pref("network.proxy.ssl_port", 00000);
user_pref("network.proxy.type", 1);
Note: the IP address and port used above are for demonstration purposes.
However, I'm encountering two problems:
1) Firefox completely ignores these settings and launches FF without any proxy at all
2) When Firefox exits the text modification is reverted/deleted
Note: When I edited the text file above, Firefox was not running. I know there's a disclaimer at the top of prefs.js:
If you make changes to this file while the application is running, the
changes will be overwritten when the application exits.
But there were no live instances of Firefox running at the time I edited the above file.
Manually creating different FF Profiles (as suggested by another user) with different proxies is not an option as everything needs to be done programmatically, without manual intervention.
Does Firefox still support linking proxy via pref.js? If not, what is the current working solution to launch Firefox via command line with a proxy in Java?
Thanks

A proxy-autoconfig file is what you are looking for.
Docs here.
Define a file name.pac, that contains the javascript function
function FindProxyForURL(url, host)
Inside the file you can use any javscript you'd like to decide what proxy to use. Set the path to your .pac file in the firefox settings, under auto-config proxy. Remember to use a file url.
To setup automatic file switching, simply configure firefox to point towards a single file, and overwrite the file programmatically every time you want it to change. You could keep copies of all options, and simply copy an option file into the target file right before running.
An example of a super simple pac file is this:
function FindProxyForURL (url, host) {
return 'PROXY proxy.example.com:8080; DIRECT';
}
It will always return the identical proxy for all endpoints.
Passwords are not explicitly supported by the pac standard, but there are different ways to approach this. Firefox will prompt you for a login if it thinks it needs one, and you could also embed the password into the url (username:password#proxy.example.com). Additionally, a tool like proxy login automator could allow you to use passwords and to dynamically set the proxy without having to fight with firefox.

How to add "open link in app mode" to right-click menu in chrome?

app mode: chrome window without navigation panel(address+tab bars). Run this in terminal
google-chrome --app=http://stackoverflow.com/
I want to open a website in app mode directly from chrome. Is there an extension that adds such option? If not how do I write a small extension that does just that? I never wrote a chrome extension but I have some experience with html and javascript. Thanks
Edit: Main issue is chrome.windows.create has no "app" option for CreateType. I guess we can't do anything about it.

There is a way using chrome.management API.
chrome.management.generateAppForLink("http://stackoverflow.com/", "Stack Overflow", function(info) {
chrome.management.setLaunchType(info.id, "OPEN_AS_WINDOW", function() {
chrome.management.launchApp(info.id);
})
});
Note that the above code requires a user gesture (which is undocumented). For examples, see Invoking activeTab. Activating a context menu should be sufficient as a gesture.
However, this will create an app in the app launcher permanently. On the plus side, it will not create duplicates for the same URL/Title.
You can call chrome.management.uninstall(id), but it will require a confirmation from the user.

HTML transient modal window

We have a legacy web application. At various places it opens a window with the help of Privilege Manager on Firefox to get the needed result.
Some of these windows open a Java applet or a PDF document.
The client machines are updating Firefox and Privilege Manager is gone.
What is the easiest way around it?
The problems are :
There must be only one instance of the pop-up at anyone time. This could be done by selecting appropriate window name on window.open() call.
If the window is opened again (by means of user action), it should not reload but just focus to bring it to the foreground (I have seen I can keep a reference to the window on JavaScript to do that)
It basically really must be transient/modal so that the client cannot leave the current page or reload or any other kind of interaction with the parent window (except opening/refocusing the child window) without closing the child window first. I have no idea how to do that.
Do anyone has an idea how to do that?
The client is only Firefox (it works in a special kiosk configuration) on Linux.
I read somewhere that I could somehow write an extension but I am basically clueless about extensions and its API.
Edit1:
Example of (simplified) legacy code. Not really sure if all the permissions were required, but this is it: This function opens a window that stays over the parent window and prevents any interaction from the user with the parent window.
function fWindowOpen(url, name) {
netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead");
netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserWrite");
netscape.security.PrivilegeManager
.enablePrivilege("CapabilityPreferencesAccess");
netscape.security.PrivilegeManager
.enablePrivilege("UniversalPreferencesWrite");
netscape.security.PrivilegeManager
.enablePrivilege("UniversalPreferencesRead");
netscape.security.PrivilegeManager.enablePrivilege("UniversalFileRead");
netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
window.open(
url,
name,
"screenX=70,dependent=yes,menubar=0,toolbar=0,width=900,height=700,modal=1,dialog=1"
);
}
function fnCapture(){
fWindowOpen("/path/to/document_or_japplet/page","_blank");
}
HTML:
<button value="Capture" property="btnCapture" onclick="javascript:fnCapture();"/>
Edit2: Solution
On a typical extension, on the xul code, define this javascript code:
var dialogExt = {
listener: function(evt) {
// Do work with parameters read through evt.target.getAttribute("attribute_name")
window.openDialog(evt.target.getAttribute("url"), evt.target.getAttribute("name"), evt.target.getAttribute("features"));
}
}
// from examples
document.addEventListener("dialogExtEvent", function(e){ dialogExt.listener(e); }, false, true);
Then, on the web page:
var element = document.createElement("dialogExtElement");
element.setAttribute("url", url);
element.setAttribute("name", name);
element.setAttribute("features", features);
document.documentElement.appendChild(element);
var evt = document.createEvent("Events");
evt.initEvent("dialogExtEvent", true, false);
element.dispatchEvent(evt);
Now, maybe I am missing some security checks to let the code work if it originates from the same host, and how to handle a reference to the document that requested the dialog as means of interaction between the dialog window and it's opener.

The Privilege Manager was deprecated in Firefox 12 and removed in Firefox 17 (briefly restored).
You might want to look into Window.showModalDialog(). However, it is deprecated and is expected to go away within the year, or in 2016 if you go with an extended service release (ESR) of Firefox 38. It may be a temporary solution while you develop an extension.
In order to accomplish the same tasks, you will need to write an extension and ask the user to install it (from Bypassing Security Restrictions and Signing Code, the old information about Privilege Manager):
Sites that require additional permissions should now ask Firefox users to install an extension, which can interact with non-privileged pages if needed.
It is possible to write such an extension using any of the three different extension types:
XUL overlay
Restartless/Bootstrap
Add-on SDK
For the first two types, you would use window.open(). The modal option is in "Features requiring privileges". You will probably also want to look at Window.openDialog().
For the Add-on SDK, you would normally use the open() function in the SDK's window/utils module. Here, again, you will probably want to look at openDialog().
It appears you may be opening content that is supplied from the web in these modal windows. It is unlikely that you will get an extension approved to be hosted on AMO which opens content in such windows which in not included in the add-on release. This does not mean you can not develop the extension and have it installed on your kiosk clients without hosting it on AMO. However, there are additional restrictions in development for Firefox this year which will make this significantly more difficult, see: "Introducing Extension Signing: A Safer Add-on Experience".

You should be able to get similiar window.open behavior, including support for the modal option from the sdk's window/utils module.
You will have to install the onclick listener with a content script, send a message to the addon-main through its port and then open that window from the addon main.

How to make a 'protocol' of my own and a Desktop application to use it for a Browser? [duplicate]

How do i set up a custom protocol handler in chrome? Something like:
myprotocol://testfile
I would need this to send a request to http://example.com?query=testfile, then send the httpresponse to my extension.

The following method registers an application to a URI Scheme. So, you can use mycustproto: in your HTML code to trigger a local application. It works on a Google Chrome Version 51.0.2704.79 m (64-bit).
I mainly used this method for printing document silently without the print dialog popping up. The result is pretty good and is a seamless solution to integrate the external application with the browser.
HTML code (simple):
Click Me
HTML code (alternative):
<input id="DealerName" />
<button id="PrintBtn"></button>
$('#PrintBtn').on('click', function(event){
event.preventDefault();
window.location.href = 'mycustproto:dealer ' + $('#DealerName').val();
});
URI Scheme will look like this:
You can create the URI Scheme manually in registry, or run the "mycustproto.reg" file (see below).
HKEY_CURRENT_USER\Software\Classes
mycustproto
(Default) = "URL:MyCustProto Protocol"
URL Protocol = ""
DefaultIcon
(Default) = "myprogram.exe,1"
shell
open
command
(Default) = "C:\Program Files\MyProgram\myprogram.exe" "%1"
mycustproto.reg example:
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Classes\mycustproto]
"URL Protocol"="\"\""
#="\"URL:MyCustProto Protocol\""
[HKEY_CURRENT_USER\Software\Classes\mycustproto\DefaultIcon]
#="\"mycustproto.exe,1\""
[HKEY_CURRENT_USER\Software\Classes\mycustproto\shell]
[HKEY_CURRENT_USER\Software\Classes\mycustproto\shell\open]
[HKEY_CURRENT_USER\Software\Classes\mycustproto\shell\open\command]
#="\"C:\\Program Files\\MyProgram\\myprogram.exe\" \"%1\""
C# console application - myprogram.exe:
using System;
using System.Collections.Generic;
using System.Text;
namespace myprogram
{
class Program
{
static string ProcessInput(string s)
{
// TODO Verify and validate the input
// string as appropriate for your application.
return s;
}
static void Main(string[] args)
{
Console.WriteLine("Raw command-line: \n\t" + Environment.CommandLine);
Console.WriteLine("\n\nArguments:\n");
foreach (string s in args)
{
Console.WriteLine("\t" + ProcessInput(s));
}
Console.WriteLine("\nPress any key to continue...");
Console.ReadKey();
}
}
}
Try to run the program first to make sure the program has been placed in the correct path:
cmd> "C:\Program Files\MyProgram\myprogram.exe" "mycustproto:Hello World"
Click the link on your HTML page:
You will see a warning window popup for the first time.
To reset the external protocol handler setting in Chrome:
If you have ever accepted the custom protocol in Chrome and would like to reset the setting, do this (currently, there is no UI in Chrome to change the setting):
Edit "Local State" this file under this path:
C:\Users\Username\AppData\Local\Google\Chrome\User Data\
or Simply go to:
%USERPROFILE%\AppData\Local\Google\Chrome\User Data\
Then, search for this string: protocol_handler
You will see the custom protocol from there.
Note: Please close your Google Chrome before editing the file. Otherwise, the change you have made will be overwritten by Chrome.
Reference:
https://msdn.microsoft.com/en-us/library/aa767914(v=vs.85).aspx

Chrome 13 now supports the navigator.registerProtocolHandler API. For example,
navigator.registerProtocolHandler(
'web+custom', 'http://example.com/rph?q=%s', 'My App');
Note that your protocol name has to start with web+, with a few exceptions for common ones (like mailto, etc). For more details, see: http://updates.html5rocks.com/2011/06/Registering-a-custom-protocol-handler

This question is old now, but there's been a recent update to Chrome (at least where packaged apps are concerned)...
http://developer.chrome.com/apps/manifest/url_handlers
and
https://github.com/GoogleChrome/chrome-extensions-samples/blob/e716678b67fd30a5876a552b9665e9f847d6d84b/apps/samples/url-handler/README.md
It allows you to register a handler for a URL (as long as you own it). Sadly no myprotocol:// but at least you can do http://myprotocol.mysite.com and can create a webpage there that points people to the app in the app store.

This is how I did it. Your app would need to install a few reg keys on installation, then in any browser you can just link to foo:\anythingHere.txt and it will open your app and pass it that value.
This is not my code, just something I found on the web when searching the same question. Just change all "foo" in the text below to the protocol name you want and change the path to your exe as well.
(put this in to a text file as save as foo.reg on your desktop, then double click it to install the keys)
-----Below this line goes into the .reg file (NOT including this line)------
REGEDIT4
[HKEY_CLASSES_ROOT\foo]
#="URL:foo Protocol"
"URL Protocol"=""
[HKEY_CLASSES_ROOT\foo\shell]
[HKEY_CLASSES_ROOT\foo\shell\open]
[HKEY_CLASSES_ROOT\foo\shell\open\command]
#="\"C:\\Program Files (x86)\\Notepad++\\notepad++.exe\" \"%1\""

Not sure whether this is the right place for my answer, but as I found very few helpful threads and this was one of them, I am posting my solution here.
Problem: I wanted Linux Mint 19.2 Cinnamon to open Evolution when clicking on mailto links in Chromium. Gmail was registered as default handler in chrome://settings/handlers and I could not choose any other handler.
Solution:
Use the xdg-settings in the console
xdg-settings set default-url-scheme-handler mailto org.gnome.Evolution.desktop
Solution was found here https://alt.os.linux.ubuntu.narkive.com/U3Gy7inF/kubuntu-mailto-links-in-chrome-doesn-t-open-evolution and adapted for my case.

I've found the solution by Jun Hsieh and MuffinMan generally works when it comes to clicking links on pages in Chrome or pasting into the URL bar, but it doesn't seem to work in a specific case of passing the string on the command line.
For example, both of the following commands open a blank Chrome window which then does nothing.
"c:\Program Files (x86)\Google\Chrome\Application\chrome.exe" "foo://C:/test.txt"
"c:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --new-window "foo://C:/test.txt"
For comparison, feeding Chrome an http or https URL with either of these commands causes the web page to be opened.
This became apparent because one of our customers reported that clicking links for our product from a PDF being displayed within Adobe Reader fails to invoke our product when Chrome is the default browser. (It works fine with MSIE and Firefox as default, but not when either Chrome or Edge are default.)
I'm guessing that instead of just telling Windows to invoke the URL and letting Windows figure things out, the Adobe product is finding the default browser, which is Chrome in this case, and then passing the URL on the command line.
I'd be interested if anyone knows of Chrome security or other settings which might be relevant here so that Chrome will fully handle a protocol handler, even if it's provided via the command line. I've been looking but so far haven't found anything.
I've been testing this against Chrome 88.0.4324.182.

open
C:\Users\<Username>\AppData\Local\Google\Chrome\User Data\Default
open Preferences then search for excluded_schemes you will find it in 'protocol_handler' delete this excluded scheme(s) to reset chrome to open url with default application