How to use fakebrowser driver and control it through Selenium Python - javascript

Summery of the question
The question is about how I can incorporate javascript modules in python.
I need to operate the fakebrowser's driver in python the same way it does in javascript so it can keep all the manipulated values of the browser fingerprint & system.
Full question
I have created an automated script for a website using selenium & python. What my code does is that it manages multiple accounts but after some time the website can detect that the accounts are being used from the same devices even after using various methods to make it undetectable (Proxies, User-Agent, Driver Options, and a ton more).
So the best module I found that can get my job done is fakebrowser because it can bypass most of the known ways websites use to detect automation. Fakebrowser can build a chromium driver but the problem I am facing is that it is using javascript and I don't want to rewrite all my code in javascript while I have already written it in Python (because I know only the basics in Javascript).
I have tried a lot of other solutions, below are only two of them but nothing I have tried so far have worked.
Attempt #1
I have tried using the chromium build which fakebrowser creates.
driver = webdriver.Chrome(executable_path=r"C:\Users\Hidden\.cache\puppeteer\chrome\win64-1069273\chrome-win\chrome.exe")
driver.get('https://pixelscan.com/')
It still leaks most of the information about my system & browser fingerprint because most of the modifications to the browser are done when you open the chrome build using fakebrowser's module.
Attempt #2
I have tried to open the driver using javascript and then start controlling it using selenium python which I found in here:
https://stackoverflow.com/questions/8344776/can-selenium-interact-with-an-existing-browser-session
Fakebrowser Javascript Code
const {FakeBrowser} = require('fakebrowser');
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
!(async () => {
const createBrowserAndGoto = async (userDataDir, url) => {
const builder = new FakeBrowser.Builder()
.vanillaLaunchOptions({
headless: false,
args: ['--remote-debugging-port=9223']
})
.userDataDir(userDataDir);
const fakeBrowser = await builder.launch();
const page = await fakeBrowser.vanillaBrowser.newPage();
await page.goto(url);
await sleep(1000000000);
await fakeBrowser.shutdown();
};
createBrowserAndGoto('./fakeBrowserUserData1', 'http://pixelscan.net/').then(e => e);
})();
Selenium Python Code
from selenium import webdriver
from selenium.webdriver.chromium.options import ChromiumOptions
class Driver:
def __init__(self):
self.options = ChromiumOptions()
self.options.add_experimental_option("debuggerAddress", "127.0.0.1:9223")
def launch(self):
driver = webdriver.Chrome(r"C:\Users\User\.cache\puppeteer\chrome\win64-1069273\chrome-win\chrome.exe", chrome_options=self.options)
driver.get('http://pixelscan.net/')
Driver().launch()
Observation
Instead of connecting to the already open browser, it creates a new one but it's not using the same values as the fakebrowser driver does. So it's still leaking most of the information of my system & browser.
I need a way to control the same one that is already open or maybe I can use the driver variable created in JavaScript and pass it to the python code but I am not sure if that's even possible

The fakebrowser package has two versions:
The basic version is puppeteer based which uses JavaScript hooks to modify properties and provides a simple api to make your web bot undetectable.
In the advanced version fakechrome recompiled Chromium to complete the simulation more robostly.
This usecase
Now you being able able to open the driver using javascript, but using Selenium-Python client when you initiate a new browsing session you still won't be able to attach it to the previous session.
This issue was discussed at length within the thread Allow webdriver to attach to a running browser.
#simon.m.stewart, WebDriver creator took a final call on this requirement as NotFeasible and further added:
I'm going to make a call on this one: it's a browser specific feature,
and not something that we can implement in a general way. With IE,
it's possible to iterate over the open windows in the OS and find the
right IE process to attach to.
Firefox and Chrome, OTOH, need to be started in a specific mode and
configuration, which means that just attaching to a running instance
isn't technically possible.
Closing as "not feasible" here as this is a browser specific feature.
Conclusion
It's still not possible to attach webdriver to a running browser instance.

As #undetected-selenium allready in his answer mentioned, this seems not to be possible.
However, if, your goal is to use the bowser for different profiles, you cantry using Selenium-Profiles.
It bypasses bot-protections like cloudfare and currently allows the following properties for the profile:
{
"options": {
"browser": {
"sandbox": true,
"window_size": {"x":1024,"y":648},
"headless": false,
"load_images": true,
"incognito": true,
"app": false,
"gpu": false,
"proxy": null,
"proxy_method": null
},
"extensions": {
"extension_paths": [],
"auth_proxy": {"host":"host","port":9000,"username":"user", "password":"password"}
},
"option_args": ["--my-arg1", "..."],
"capabilities": [],
"adb": false,
"adb_package": "com.android.chrome",
"use_running_app": true
},
"cdp": {
"browser": {
"pointer_as_touch": false,
"darkmode": false,
"mobile": true
},
"touch": true,
"maxtouchpoints": 5,
"cdp_args": [],
"emulation": {"mobile":true,"width": 384, "height": 700, "deviceScaleFactor": 10,
"screenOrientation": {"type": "portrait-primary", "angle": 0}},
"useragent": {
"platform": "Linux aarch64",
"acceptLanguage":"en-US",
"userAgent": "Mozilla/5.0 (Linux; Android 11; HD1913) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Mobile Safari/537.36",
"userAgentMetadata": {
"brands": [{"brand": "Google Chrome", "version": "105"}, {"brand": "Not)A;Brand", "version": "8"},
{"brand": "Chromium", "version": "105"}],
"fullVersionList": [{"brand": "Google Chrome", "version": "105.0.5195.136"},
{"brand": "Not)A;Brand", "version": "8.0.0.0"},
{"brand": "Chromium", "version": "105.0.5195.136"}],
"fullVersion": "105.0.5195.136",
"platform": "Android",
"platformVersion": "11.0.0",
"architecture": "",
"model": "HD1913",
"mobile": true,
"bitness": "",
"wow64": false}
}}
}
A example profile for Windows would be:
"options": {
"browser": {
"gpu": false,
"window_size": {"x":1024,"y":648}}
},
"cdp": {
"touch": true,
"maxtouchpoints": 10,
"emulation": {"mobile":false,"width": 1024, "height": 648, "deviceScaleFactor": 1,
"screenOrientation": {"type": "portraitPrimary", "angle": 0}},
"useragent": {
"platform": "Win32",
"acceptLanguage":"en-US",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
"userAgentMetadata": {
"brands": [{"brand":"Google Chrome","version":"107"},
{"brand":"Chromium","version":"107"},
{"brand":"Not=A?Brand","version":"24"}],
"fullVersionList": [{"brand":"Google Chrome","version":"107.0.5304.88"},
{"brand":"Chromium","version":"107.0.5304.88"},
{"brand":"Not=A?Brand","version":"24.0.0.0"}],
"fullVersion": "107.0.5304.88",
"platform": "Windows",
"platformVersion": "10.0.0",
"architecture": "x86",
"model": "",
"mobile": false,
"bitness": "64",
"wow64": false}
}
}
}
and of course it also supports settiing, deleting and getting cookies :)

Related

Separate/pull capabilities from another JS file

Ok so I'm trying to put the capabilities declaration for my Appium tests into another file and just reference it but I'm having no luck so far with using classes, functions, arrays and even global variables. I have also tried pulling it from a csv file but since JS is asynchronous that didn't work either.
I am not very familiar with JS and honestly stumped so if anyone could point me in the right direction that would be greatly appeciated!
Here's the code that I'm trying to seperate/make a reference for:
const opts = {
port: 4723,
desiredCapabilities: {
platformName: "Android",
platformVersion: "8.1.0",
deviceName: "Nexus 6P",
app: "C:/Users/reina.reinhart/KonyWorkspace/temp/dcpApp/build/luaandroid/dist/luavmandroid.apk",
automationName: "UiAutomator2",
noReset: true
}
}

Unable to use Contacts API in Firefox OS app

I have gone through this and tried everything mentioned there, but I always getting error only.
I have tested in simulator "Firefox OS 2.2".
My manifest contains two permissions like:
"permissions": {
"desktop-notification": {
"description": "Needed for creating system notifications."
},
"contacts": {
"description": "Needed for reading contacts"
}
},
What have I done wrong?
Got it, just added access property to the manifest. Its working now.
"contacts": {
"access": "readcreate"
}

Error retrieving a new session from the selenium server

I'm attempting to setup Nightwatch.js for the first time. I'm following the following tutorial: https://github.com/dwyl/learn-nightwatch
Unfortunately I've hit a roadblock, and I'm in need of help resolving it.
Error retrieving a new session from the selenium server.
Connection refused! Is selenium server started?
nightwatch.conf.js
module.exports = {
"src_folders": [
"test"// Where you are storing your Nightwatch e2e/UAT tests
],
"output_folder": "./reports", // reports (test outcome) output by nightwatch
"selenium": {
"start_process": true, // tells nightwatch to start/stop the selenium process
"server_path": "./node_modules/nightwatch/bin/selenium.jar",
"host": "127.0.0.1",
"port": 4444, // standard selenium port
"cli_args": {
"webdriver.chrome.driver" : "./node_modules/nightwatch/bin/chromedriver"
}
},
"test_settings": {
"default": {
"screenshots": {
"enabled": true, // if you want to keep screenshots
"path": './screenshots' // save screenshots here
},
"globals": {
"waitForConditionTimeout": 5000 // sometimes internet is slow so wait.
},
"desiredCapabilities": { // use Chrome as the default browser for tests
"browserName": "chrome"
}
},
"chrome": {
"desiredCapabilities": {
"browserName": "chrome",
"javascriptEnabled": true // set to false to test progressive enhancement
}
}
}
}
guinea-pig.js
module.exports = { // addapted from: https://git.io/vodU0
'Guinea Pig Assert Title': function(browser) {
browser
.url('https://saucelabs.com/test/guinea-pig')
.waitForElementVisible('body')
.assert.title('I am a page title - Sauce Labs')
.saveScreenshot('ginea-pig-test.png')
.end();
}
};
Based on the configuration setup. I kept it as basic as possible. I cannot pinpoint the source where it would suggest another selenium server has started. Any ideas?
EDIT: TIMEOUT ERROR
In your nightwatch.json file, within "selenium"
Make sure your server path is correct.
Make sure your webdriver.chrome.driver path is correct.
Those are specific to your machine. If those do not refer to the correct file in the correct location, you'll get problems starting the selenium server.
After that, you want to make certain that the version of the selenium server you have works with the version of chrome driver you have and that that will work with the version of the Chrome browser you have.
But as Krishnan Mahadevanindicated, without the whole error message, we can't be of much more help.
The solution involved deleting my instance of Chrome (although it was the most recent version) and simply reinstalling the browser again.
I encourage all facing the same problems to first look at QualiT's response above as it's the more conventional troubleshooting strategy.
I had got the same issue when I used vue-cli init on my project. after I updated to Java 9, this problem was resolved.

Configuring multiple capabilities with promises

This is a follow-up to the Set firefox profile with protractor topic.
According to the setFirefoxProfile howto, it is possible to set a firefox profile with a special "helper" js code which uses firefox-profile and q libraries to make an encoded firefox profile on the fly.
This worked for me until I've tried to use multiple browsers and configuring multiCapabilities:
exports.config = {
seleniumAddress: 'http://localhost:4444/wd/hub',
multiCapabilities: [
{
browserName: 'chrome',
specs: [
'footer.disabledCookies.spec.js'
],
chromeOptions: {
prefs: {
'profile.default_content_settings.cookies': 2
}
}
},
...
// other capabilities here
...
helper.getFirefoxProfile()
},
...
}
With this setup I'm getting an error (full traceback here):
Spec patterns did not match any files.
As I understand, this means that the setup with firefox profile is missing specs key. In other words, it cannot find any tests to run.
I've tried to include specs into the capabilities dictionary inside the helper itself, but the error persists.
How to fix the error and set firefox profile if using multiCapabilities?
As a workaround, I've created a separate protractor configuration file with only firefox configured (using capabilities) and set grunt to run protractor twice - one for this "firefox with a profile" config and the other one for all other browsers.
Right now, protractor can only accept promise as capabilities if we are NOT using multicapabilities. The reason for this is because multiCapabilities runs each task in a new process, so the promise (function) cannot be passed (single capabilities work because we're not forking).
Alternatively we could resolve capabilities in the launcher, before passing the resolved capabilities into the new processes; however, this will break the ability to set up proxies (https://github.com/angular/protractor/pull/1040), which relies on capability promises to be resolved after driverProvider setup.
I can't think of an easy way of doing this (without large refactoring), but it is definitely doable.
I created an issue for Protractor (https://github.com/angular/protractor/issues/1594). Please follow that and/or comment on it if this is something you need or you have other ideas to implement it.
For now you would need to use the workaround you mentioned in your original question.
UPDATE
https://github.com/angular/protractor/pull/1629 supports this. Starting in protractor 1.6 (or if you sync to master) you can pass in a function to config.getMultiCapabilities like onPrepare and onCleanup. This function can return a promise to multiCapabilties (i.e. array of capabilities).
See https://github.com/angular/protractor/blob/master/spec/getCapabilitiesConf.js for an example.
Following the pull request sent by #hankduan, here is how have I used getMultiCapabilities() to combine different capabilities where one of them is a promise (needed for firefox-profile to be set):
"use strict";
var FirefoxProfile = require("firefox-profile");
var q = require("q");
exports.config = {
seleniumAddress: "http://127.0.0.1:4444/wd/hub",
getMultiCapabilities: function() {
var deferred = q.defer();
var multiCapabilities = [
{
browserName: "chrome",
specs: [
"footer.disabledCookies.spec.js"
],
chromeOptions: {
prefs: {
"profile.default_content_settings.cookies": 2
}
}
},
{
browserName: "chrome",
specs: [
"*.spec.js"
],
exclude: [
"footer.disabledCookies.spec.js",
"footer.disabledJavascript.spec.js",
"footer.disabledFlash.spec.js"
]
},
{
browserName: "chrome",
specs: [
"footer.disabledFlash.spec.js"
],
chromeOptions: {
args: [
"--disable-internal-flash",
"--disable-bundled-ppapi-flash",
"--disable-plugins-discovery"
]
}
}
];
// Wait for a server to be ready or get capabilities asynchronously.
setTimeout(function() {
var firefoxProfile = new FirefoxProfile();
firefoxProfile.setPreference("javascript.enabled", false);
firefoxProfile.encoded(function (encodedProfile) {
var capabilities = {
"browserName": "firefox",
"firefox_profile": encodedProfile,
"specs": [
"footer.disabledJavascript.spec.js"
]
};
multiCapabilities.push(capabilities);
deferred.resolve(multiCapabilities);
});
}, 1000);
return deferred.promise;
},
...
};
Hope this would help somebody in the future.

Deployd - Data retrieved via AngularJS CORS

I am currently reading through "Pro AngularJS" by Adam Freeman. In going through the examples, he has the reader create a sports store app using Angular (of course) with a Deployd server resource. The Deployd resource is set up to return JSON data that is to be populated into the model. I am using NodeJS to run my server. It is currently setup on port 5000 (http://localhost:5000/sportsstore/app.html). The Deployd resource is running on port 5500 (http://localhost:5500/products). When hitting Deployd, the response is as follows:
[
{ "name": "Kayak", "description": "A boat for one person", "category": "Watersports", "price": 275, "id": "a1c999fc248b2959" },
{ "name": "Lifejacket", "description": "Protective and fashionable", "category": "Watersports", "price": 48.95, "id": "61303717cfad182e" },
{ "name": "Soccer Ball", "description": "FIFA-approved size and weight", "category": "Soccer", "price": 19.5, "id": "0fb5f67bdcbd992f" },
{ "name": "Corner Flags", "description": "Give your playing field a professional touch", "category": "Soccer", "price": 34.95, "id": "24385d315dd388b4" },
{ "name": "Stadium", "description": "Flat-packed 35,000-seat stadium", "category": "Soccer", "price": 79500, "id": "500fb6805905a856" },
{ "name": "Thinking Cap", "description": "Improve your brain efficiency by 75%", "category": "Chess", "price": 16, "id": "637d8a1f42e6fa1c" },
{ "name": "Unsteady Chair", "description": "Secretly give your opponent a disadvantage", "category": "Chess", "price": 29.95, "id": "73393312ec7dfab7" },
{ "name": "Human Chess Board", "description": "A fun game for the family", "category": "Chess", "price": 75, "id": "7871d02a662b0915" },
{ "name": "Bling-Bling King", "description": "Gold plated, diamon-studded King", "category": "Chess", "price": 1200, "id": "b59a3389a0e248bd" }
]
I am attempting to retrieve this data through use of $http.get:
$http.get("http://localhost:5500/products")
.success(function (data) { ... })
.error(function (error) { ... });
However, this keeps returning an error:
XMLHttpRequest cannot load http://localhost:5500/products. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:5000' is therefore not allowed access.
Research shows that there are/were some issues with Angular and CORS, and that the headers had to be configured to run cross-domain requests. As a result, I added the following to my app.config:
$http.defaults.useXDomain = true;
delete $http.defaults.headers.common['X-Requested-With']; // this isn't needed anymore, but was put here as a just-in-case
Despite having these settings added, I am still getting the error. The Deployd documentation says that it is automatically configured for CORS (Cross-Origin Requests) and will send the appropriate header information as long as the request did not contain invalid custom headers. I'm pretty sure my request does not contain invalid custom headers:
Accept: application/json, text/plain, */*
Cache-Control: max-age=0
Origin: http://localhost:5000
Referer: http://localhost:5000/sportsstore/app.html
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36
My Question: Is there some other configuration I need to put in place in order to have Deployd configured to allow the CORS request to process? The book does not specify any of the special Angular header settings or anything else.
Bic, upgrade your version of deployd to version 0.6.10. This did the trick for me. I was now able to process a get request. It doesn't seem like it's an error with AngularJS code nor Adam Freeman's book.
In the book, he does mention that he includes the deployd program with the source code download on http://www.apress.com/9781430264484. That's version 0.6.9. I'm sure it works fine with it. It'll be easier than to try to find the 0.6.10 version.. which is what I did. Should you want that version, here it is:
https://www.versioneye.com/nodejs/deployd/0.6.10
It's not an installer so you'll have to paste it in your deployd directory, replacing the node_modules.
Just run "npm update" in your deployd installation folder and it will make sure you're updated to the latest version 0.6.10. This resolved the issue for me after reading javaauthority's answers (Thanks for that :)).
You can place your files (app.html, sportStore.js, ...) under the public folder of your deployd project and use the following URL
http://localhost:5500/app.html
What I did to solve that issue was starting Chrome with the "--disable-web-security" flag. But first go to Task Manager and close every chrome process you have there.
While I provided the correct answer and others have improved it, what if you're not using the Deployd web server mentioned in the book? I'm using Wildfly (JBOSS 8.X) and I had to make the CORS issue work for it. I created a simple Java class called CorsFilter. The imports should be fairly easy to find if you're running Wildfly.
Hope this helps other people who are having similar issues with different web servers.
Please note: responseContext.getHeaders().add("Access-Control-Allow-Origin", "*");
Notice the * in the above line? This would allow ANY requests to succeed. This is usually fine for local development work, but tighter security controls should be implemented for Production/Staging environments. For example, you could only accept request from a certain IP address.
import org.jboss.resteasy.annotations.interception.HeaderDecoratorPrecedence;
import org.jboss.resteasy.annotations.interception.ServerInterceptor;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.annotation.WebFilter;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerResponseContext;
import javax.ws.rs.container.ContainerResponseFilter;
import javax.ws.rs.ext.Provider;
import java.io.IOException;
/**
* Class to .
* User: Java Authority
* Date: 12/6/2014
* Time: 12:38 PM
*/
#Provider
#ServerInterceptor
#HeaderDecoratorPrecedence
#WebFilter
public class CorsFilter implements ContainerResponseFilter {
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
HttpServletRequest r = (HttpServletRequest)request;
HttpServletResponse s = (HttpServletResponse)response;
chain.doFilter(request, response);
}
#Override
public void filter(ContainerRequestContext requestContext, ContainerResponseContext responseContext) throws IOException {
responseContext.getHeaders().add("Access-Control-Allow-Origin", "*");
responseContext.getHeaders().add("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
}
}
I used fiddler to modify the responses from deployd. This link will show how to add a custom rule to fiddler to add Access-Control-Allow-Origin: * header to all responses. It works great.
http://www.leggetter.co.uk/2010/03/19/using-fiddler-to-help-develop-cross-domain-capable-javascript-web-applications.html
Angular also sends custom headers that DPD rejects, put this in your code for dev to remove these and DPD will work:
delete $http.defaults.headers.common['X-Requested-With'];

Categories