scrapy [boto] ERROR: Caught exception reading instance data URLError: <urlopen error [Errno 10051] >

执行过程中出现错误：

2015-09-09 11:13:26 [boto] DEBUG: Retrieving credentials from metadata server.
2015-09-09 11:13:27 [boto] ERROR: Caught exception reading instance data
Traceback (most recent call last):
File "D:\anzhuang\Anaconda\lib\site-packages\boto\utils.py", line 210, in retry_url
r = opener.open(req, timeout=timeout)
File "D:\anzhuang\Anaconda\lib\urllib2.py", line 431, in open
response = self._open(req, data)
File "D:\anzhuang\Anaconda\lib\urllib2.py", line 449, in _open
'_open', req)
File "D:\anzhuang\Anaconda\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "D:\anzhuang\Anaconda\lib\urllib2.py", line 1227, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "D:\anzhuang\Anaconda\lib\urllib2.py", line 1197, in do_open
raise URLError(err)
URLError: <urlopen error timed out>
2015-09-09 11:13:27 [boto] ERROR: Unable to read instance data, giving up

在setting.py中禁用s3 download就可以了

解决方法：

DOWNLOAD_HANDLERS = {'S3': None,}

在stackoverflow上有些解释,有时间可以看看这篇错误分析错误

下面是我贴的部分内容。

That particular error message is being generated by boto (boto 2.38.0 py27_0), which is used to connect to Amazon S3. Scrapy doesn’t have this enabled by default.

EDIT: In reply to the comments, this appears to be a bug with Scrapy when boto is present (bug here).

In response “how to disable the Download handler”, add the following to your settings.py file:

DOWNLOAD_HANDLERS = {
'S3':None,
}

Your settings.py file should be in the root of your Scrapy project folder, (one level deeper than your scrapy.cfg file).

If you’ve already got DOWNLOAD_HANDLERS in your settings.py file, just add a new entry for ‘s3’ with a None value.

EDIT 2: I’d highly recommend looking at setting up virtual environments for your projects. Look into virtualenv, and it’s usage. I’d make this recommendation regardless of packages used for this project, but doubly so with your extreme number of packages.