How to use authenticated proxy in selenium chromedriver?
Each Answer to this Q is separated by one/two green lines.
After searching for many hours I am starting to think this is impossible.
I need to run Chrome through selenium using different authenticated (not public) proxy’s for each run.
PROXY_IP = "<some IP address>"
UID = "<the user id>"
PWD = "<the password">
options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=%s:%[email protected]%s" % (UID,PWD,PROXY_IP))
driver = webdriver.Chrome(executable_path=".\\driver\\chromedriver.exe",
chrome_options=options)
driver.get("<site URL>")
Chrome will fire-up and display the error:
This webpage is not available
ERR_NO_SUPPORTED_PROXIES
If I use a public proxy requiring no authentication like this…
PROXY_IP = "<public proxy IP address>"
options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=%s" % PROXY_IP)
driver = webdriver.Chrome(executable_path=".\\driver\\chromedriver.exe",
chrome_options=options)
driver.get("<site URL>")
…it runs just fine and displays the site while using the proxy.
I also tried a variant with http://
in front of the user ID:
options.add_argument("--proxy-server=http://%s:%[email protected]%s" % (UID,PWD,PROXY_IP))
The fact that I have searched far and wide and haven’t found a solution leads me to believe none might exist.
I did find this but I can’t make sense out of it:
selenium chromedriver authentication proxy
Not sure what browswermob-proxy
is or is supposed to do or how to implement and test in Python. I hate piling up band-aid solutions unless they are absolutely necessary.
EDIT (08NOV21):
I have been away from using Selenium for many years. Because of this I now lack the context (and time, sorry) to go through the newer answers being provided and mark one as the solution to this problem. Does SO have a mechanism one could use to effectively delegate this function to someone who might be a current practitioner with expertise in this domain?
To use proxies with auth in python selenium you can use seleniumwire.
Fistly, install it with pip install selenium-wire
Then import webdriver from seleniumwire instead selenium
from seleniumwire import webdriver
options = {
'proxy': {
'http': 'http://username:[email protected]:port',
'https': 'https://username:[email protected]:port',
'no_proxy': 'localhost,127.0.0.1' # excludes
}
}
browser = webdriver.Chrome(path_to_driver, seleniumwire_options=options)
Now you can use your browser instance exact the same way as selenium: browser.get('https://api.ipify.org')
and so on…
I have checked for most of the solutions on the web and for none of them authentication via chrome/firefox desired capabilities is working. Check this link: https://github.com/webdriverio/webdriverio/issues/324. Finally the temporary solution is to whitelist your IP address with the proxy provider.
This is the best solution I found and is the ONLY one that worked – all other answers on this question are outdated. It basically generates an auth extension for Chrome on the fly. Simply use the function as defined in the script as follows:
driver = proxy_chrome(host, port, username, password)
driver.get("http://www.icanhazip.com/")
driver.get("https://www.reddit.com/")
print('Terminated without issues.')
Note that this doesn’t work with the –headless option. However, on Linux, you can simply use the x virtual frame buffer to simulate that. It’s as easy as this in python:
import xvfbwrapper
x = xvfbwrapper.Xvfb()
x.start()
after trying many solutions that didn’t actually work properly, i finally managed to set the authenticated proxy using the suggested extension from previous answers.
what you need to do is to enter this link:
http://crxextractor.com/
and paste this url:
https://www.crx4chrome.com/crx/1446/
It will let you download the extention as a .crx file without installing it.
than i used this code:
proxy = {'address': 'pr.oxylabs.io:7777',
'username': 'USERNAME',
'password': 'PASSWORD'}
capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
'httpProxy': proxy['address'],
'ftpProxy': proxy['address'],
'sslProxy': proxy['address'],
'noProxy': '',
'class': "org.openqa.selenium.Proxy",
'autodetect': False,
'socksUsername': proxy['username'],
'socksPassword': proxy['password']}
options = webdriver.ChromeOptions()
options.add_extension("./extension_2_0_0_0.crx")
driver = webdriver.Chrome(executable_path=CHROME_PATH, desired_capabilities=capabilities, chrome_options=options)
I could not find any solution for chrome. We can not add extensions with headless
option.
I am using Heroku with chrome-buildpack. There are following options
- Use
xvfb
instead of headless options and install extension - Use local proxy forwarder that forwards traffic to authenticated proxy; we can use Squid, mitProxy, or something like proxy-login-automator
Instead of these workaround I switched to Firefox where i was able to fill Username and Password on Proxy authentication Pop-up. Like given below. Following code is for Ruby using Capybara. You should be able to do something like this on your platform
page.driver.browser.switch_to.alert.send_keys('proxy-username' + Selenium::WebDriver::Keys::KEYS[:tab] + 'my-password')
page.driver.browser.switch_to.alert.accept
I tryed lot of time to do the same.
Chrome is using only proxy of the OS where it is installed. You can check it by going to options-> find: proxy -> change proxy settings
So without additional addons and configuring this addons you cannot do this.
Or you can change your OS proxy settings — this is much easier.
Also you can use phantomjs — it have the same engine(WebKit) as chrome.
using something like this:
String PROXY = proxyIP + ":" + proxyPort;
String proxyAuth= proxyUser + ":" + proxyPass;
OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
proxy.HttpProxy = PROXY;
proxy.FtpProxy = PROXY;
proxy.SslProxy = PROXY;
proxy.SocksProxy = PROXY;
var serviceJS = PhantomJSDriverService.CreateDefaultService(phantomPath);
serviceJS.AddArguments("--proxy=" + PROXY, "--proxy-type=http", "--proxy-auth=" + proxyAuth);
THis is a temporary solution might work in initial state:
Code is in Python:
Download the plugin first from chrome plugin store : Proxy-Auto-Auth_v2.0.crx
options = webdriver.ChromeOptions()
options.add_extension("./Proxy-Auto-Auth_v2.0.crx")) #this will provide you a window to enter user name and proxy
driver = webdriver.Remote(command_executor=selenium_server,desired_capabilities=options.to_capabilities())
or
driver = webdriver.Chrome(chrome_options=options)
I have been searching extensively. This is what it works for me
PROXY_HOST = 'FILL IN' # rotating proxy or host
PROXY_PORT = 8080 # port
PROXY_USER = 'FILL IN' # username
PROXY_PASS = 'FILL IN' # password
http_proxies = { 'https' : 'http://' + PROXY_USER + ':' + PROXY_PASS + '@' + PROXY_HOST + ':' + str(PROXY_PORT) }
manifest_json = """
{
"version": "1.0.0",
"manifest_version": 2,
"name": "Chrome Proxy",
"permissions": [
"proxy",
"tabs",
"unlimitedStorage",
"storage",
"<all_urls>",
"webRequest",
"webRequestBlocking"
],
"background": {
"scripts": ["background.js"]
},
"minimum_chrome_version":"22.0.0"
}
"""
background_js = """
var config = {
mode: "fixed_servers",
rules: {
singleProxy: {
scheme: "http",
host: "%s",
port: parseInt(%s)
},
bypassList: ["localhost"]
}
};
chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});
function callbackFn(details) {
return {
authCredentials: {
username: "%s",
password: "%s"
}
};
}
chrome.webRequest.onAuthRequired.addListener(
callbackFn,
{urls: ["<all_urls>"]},
['blocking']
);
""" % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)
def get_chromedriver(self, use_proxy=False):
driver_path="FILL IN"
path = os.path.dirname(os.path.abspath(__file__))
chrome_options = webdriver.ChromeOptions()
if use_proxy:
pluginfile="proxy_auth_plugin.zip"
with zipfile.ZipFile(pluginfile, 'w') as zp:
zp.writestr("manifest.json", libshared.manifest_json)
zp.writestr("background.js", libshared.background_js)
chrome_options.add_extension(pluginfile)
driver = webdriver.Chrome(
executable_path=driver_path,
chrome_options=chrome_options)
return driver
1ยบ If you always use the same IP you can speak with your proxy provider to access proxy with your IP = no user or password needed. In my experience that’s easier.
If that’s not possible other solution is:
pip install undetectable chromedriver and seleniumwire if you don’t have them
selenium-wire 4.5.6
selenium 4.1.0
undetected-chromedriver 2.2.1
from seleniumwire.undetected_chromedriver.v2 import Chrome
options = {
'proxy': {
'http': 'http://user:[email protected]:port',
'https': 'https://user:[email protected]:port',
'no_proxy': 'localhost,127.0.0.1'
}
}
driver = Chrome(seleniumwire_options=options)
or you can also try another solution:
import undetected_chromedriver.v2 as uc
IP = XXXX
SOCKS5_PORT = XXXX
HTTPS_PORT = XXXX
options = uc.ChromeOptions()
options.add_argument(f'--proxy-server=socks5://{IP}:{SOCKS5_PORT}')
#you can substitue the last line for this one:
options.add_argument(f'--proxy-server=https://{IP}:{HTTPS_PORT}')
driver = uc.Chrome(executable_path=EXECUTABLE_PATH, options=options)