scrapy怎么设置带有密码的代理ip base64.encodestring不能用 python3.5,base64库里面的encodestring()被换成了什么?
2018-01-08 19:26
639 查看
自己写爬虫时买的代理ip有密码,在网上查了都是下面这种:
1、在Scrapy工程下新建"middlewares.py":
import base64
# Start your middleware class
class ProxyMiddleware(object):
# overwrite process request
def process_request(self, request, spider):
# Set the location of the proxy
request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"
# Use the following lines if your proxy requires authentication
proxy_user_pass = "USERNAME:PASSWORD"
# setup basic authentication for the proxy
encoded_user_pass = base64.encodestring(proxy_user_pass) #我用的python3.5,这个地方报错 TypeError: expected bytes-like object, not str
request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
2、在项目配置文件里setting.py添加:
DOWNLOADER_MIDDLEWARES = {
... #此处省略其他中间件配置
'ScrapyRedisTest.middlewares.ProxyMiddleware': 100,
}
================================解决方法如下:(只改动标红的两行,其他的都不变)================================================
import base64
# Start your middleware class
class ProxyMiddleware(object):
# overwrite process request
def process_request(self, request, spider):
# Set the location of the proxy
request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"
# Use the following lines if your proxy requires authentication
proxy_user_pass = "USERNAME:PASSWORD"
# setup basic authentication for the proxy
encoded_user_pass = base64.b64encode(proxy_user_pass.encode('utf-8'))
request.headers['Proxy-Authorization'] = 'Basic ' + str(encoded_user_pass, encoding="utf-8")
2、在项目配置文件里setting.py添加:
DOWNLOADER_MIDDLEWARES = {
... #此处省略其他中间件配置
'ScrapyRedisTest.middlewares.ProxyMiddleware': 100,
}
1、在Scrapy工程下新建"middlewares.py":
import base64
# Start your middleware class
class ProxyMiddleware(object):
# overwrite process request
def process_request(self, request, spider):
# Set the location of the proxy
request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"
# Use the following lines if your proxy requires authentication
proxy_user_pass = "USERNAME:PASSWORD"
# setup basic authentication for the proxy
encoded_user_pass = base64.encodestring(proxy_user_pass) #我用的python3.5,这个地方报错 TypeError: expected bytes-like object, not str
request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
2、在项目配置文件里setting.py添加:
DOWNLOADER_MIDDLEWARES = {
... #此处省略其他中间件配置
'ScrapyRedisTest.middlewares.ProxyMiddleware': 100,
}
================================解决方法如下:(只改动标红的两行,其他的都不变)================================================
import base64
# Start your middleware class
class ProxyMiddleware(object):
# overwrite process request
def process_request(self, request, spider):
# Set the location of the proxy
request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"
# Use the following lines if your proxy requires authentication
proxy_user_pass = "USERNAME:PASSWORD"
# setup basic authentication for the proxy
encoded_user_pass = base64.b64encode(proxy_user_pass.encode('utf-8'))
request.headers['Proxy-Authorization'] = 'Basic ' + str(encoded_user_pass, encoding="utf-8")
2、在项目配置文件里setting.py添加:
DOWNLOADER_MIDDLEWARES = {
... #此处省略其他中间件配置
'ScrapyRedisTest.middlewares.ProxyMiddleware': 100,
}
相关文章推荐
- python3.5里面 urlopen 代理不能识别proxies关键字的问题
- Python 设置 IP 代理 访问网页 ( 用户名密码验证代理 )
- Python-Scrapy 个人兴趣教程(二):没错,从代理IP开始
- Python-Scrapy 个人兴趣教程(二):没错,从代理IP开始
- Python爬虫设置代理IP的方法(爬虫技巧)
- 第三百四十八节,Python分布式爬虫打造搜索引擎Scrapy精讲—通过自定义中间件全局随机更换代理IP
- Python爬虫设置代理IP和伪装成浏览器的方法
- Linux 设置代理时, 密码出现特殊字符怎么办?
- scrapy使用代理ip的时候下载器中间件的配置文件设置
- 1. 怎么设置可以使得虚拟机里面既可以访问主机也可以访问局域网而且是静态ip
- python爬虫的代理IP设置
- 干货|Python爬虫如何设置代理IP
- win8图片密码是什么怎么设置?win8图片密码设置图解
- Python爬虫设置代理IP爬取知乎图片
- Python实现爬虫设置代理IP和伪装成浏览器的方法分享
- Python爬虫常用小技巧之设置代理IP
- Python爬虫技巧---设置代理IP
- Python网络爬虫Scrapy框架研究 以及 代理设置
- python设置代理ip爬取知乎图片