Selenium爬虫代理代码示例
2019-06-19 17:41
1651 查看
from selenium import webdriver
import string
import zipfile
# 代理服务器 proxyHost = "t.16yun.cn" proxyPort = "31111" # 代理隧道验证信息 proxyUser = "username" proxyPass = "password" def create_proxy_auth_extension(proxy_host, proxy_port, proxy_username, proxy_password, scheme='http', plugin_path=None): if plugin_path is None: plugin_path = r'C:/{}_{}@t.16yun.zip'.format(proxy_username, proxy_password) manifest_json = """ { "version": "1.0.0", "manifest_version": 2, "name": "16YUN Proxy", "permissions": [ "proxy", "tabs", "unlimitedStorage", "storage", "", "webRequest", "webRequestBlocking" ], "background": { "scripts": ["background.js"] }, "minimum_chrome_version":"22.0.0" } """ background_js = string.Template( """ var config = { mode: "fixed_servers", rules: { singleProxy: { scheme: "${scheme}", host: "${host}", port: parseInt(${port}) }, bypassList: ["foobar.com"] } }; chrome.proxy.settings.set({value: config, scope: "regular"}, function() {}); function callbackFn(details) { return { authCredentials: { username: "${username}", password: "${password}" } }; } chrome.webRequest.onAuthRequired.addListener( callbackFn, {urls: [""]}, ['blocking'] ); """ ).substitute( host=proxy_host, port=proxy_port, username=proxy_username, password=proxy_password, scheme=scheme, ) with zipfile.ZipFile(plugin_path, 'w') as zp: zp.writestr("manifest.json", manifest_json) zp.writestr("background.js", background_js) return plugin_path proxy_auth_plugin_path = create_proxy_auth_extension( proxy_host=proxyHost, proxy_port=proxyPort, proxy_username=proxyUser, proxy_password=proxyPass) option = webdriver.ChromeOptions() option.add_argument("--start-maximized") # 如报错 chrome-extensions # option.add_argument("--disable-extensions") option.add_extension(proxy_auth_plugin_path) driver = webdriver.Chrome(chrome_options=option) driver.get("http://httpbin.org/ip")
相关文章推荐
- C#多线程爬虫抓取免费代理IP的示例代码
- PHP爬虫程序中添加代理信息代码示例
- Python爬虫获取整个站点中的所有外部链接代码示例
- Selenium_python 示例代码(1)
- 基于node下的http小爬虫的示例代码
- python selenium 爬虫,使用代理 ,和 python 爬取代理网站,保存ip
- Python Selenium Cookie 绕过验证码实现登录示例代码
- Python程序中添加代理信息代码示例
- Powershell 简单爬虫示例 附,爬知乎示例 加代理
- 爬虫技术 -- 进阶学习(七)简单爬虫抓取示例(附c#代码)
- Koa代理Http请求的示例代码
- selenium python浏览器多窗口处理代码示例
- Python selenium抓取微博内容的示例代码
- selenium+java破解极验滑动验证码的示例代码
- Selenium 2 (WebDreiver) 代码示例:弹出窗口测试
- jdk动态代理示例以及代码原理分析
- 用python做自动化测试--web 自动化测试(2)-Selenium 3.x启动IE, Firefox,Chrome代码示例
- 爬虫学习笔记1--代码示例
- python爬虫简单的添加代理进行访问的实现代码
- Python爬虫爬取新浪微博内容示例【基于代理IP】