Python3网络爬虫开发实战（崔庆才）笔记——ProxyPool的代码问题：AttributeError: 'int' object has no attribute 'item及相关问题的处理

博主在阅读崔庆才著的《Python3网络爬虫实战》时，深深被其爬虫的高超技术所吸引。当阅读到代理池部分的时候，在代码实践时遇到的一些问题：

AttributeError: 'int' object has no attribute 'item

笔者百思不得其解，终于在经过http://www.pianshen.com/article/3497197559/的启发之后发现了问题的本源。

ZADD的帮助文档如下：

class Redis(object)  def zadd(self, name, mapping, nx=False, xx=False, ch=False, incr=False)
Inferred type: (self: Redis, name: Any, mapping: {__len__, items, iteritems}, nx: bool, xx: bool, ch: bool, incr: bool) -> Any
Set any number of element-name, score pairs to the key name. Pairs are specified as a dict of element-names keys to score values.

nx forces ZADD to only create new elements and not to update scores for elements that already exist.

xx forces ZADD to only update scores of elements that already exist. New elements will not be added.

ch modifies the return value to be the numbers of elements changed. Changed elements include new elements that were added and elements whose scores changed.

incr modifies ZADD to behave like ZINCRBY. In this mode only a single element/score pair can be specified and the score is the amount the existing score will be incremented by. When using this mode the return value of ZADD will be the new score of the element.

The return value of ZADD varies based on the mode specified. With no options, ZADD returns the number of new elements added to the sorted set.

可以看到ZADD函数的第三个参数为Mapping类型，而非单独的proxy与score。笔者猜测，可能由于Redis库版本问题而产生了代码不兼容的问题。

解决方法如下：

• 将db.py文件中add、decrease、max函数进行如下修改（主要为增加mapping参数与去除无用return，并应用ZADD函数替换ZINCRBY函数）：

def add(self, proxy, score=INITIAL_SCORE):
"""
添加代理，设置分数为最高
:param proxy: 代理
:param score: 分数
:return: 添加结果
"""
if not re.match('\d+\.\d+\.\d+\.\d+\:\d+', proxy):
print('代理不符合规范', proxy, '丢弃')
return
if not self.db.zscore(REDIS_KEY, proxy):
mapping={
proxy:score,
}
return self.db.zadd(REDIS_KEY, mapping)

def decrease(self, proxy):
"""
代理值减一分，小于最小值则删除
:param proxy: 代理
:return: 修改后的代理分数
"""
score = self.db.zscore(REDIS_KEY, proxy)
if score and score > MIN_SCORE:
print('代理', proxy, '当前分数', score, '减1，变为',score -1)
score = score -1
mapping={
proxy:score,
}
self.db.zadd(REDIS_KEY,mapping)
else:
print('代理', proxy, '当前分数', score, '移除')
self.db.zrem(REDIS_KEY, proxy)

def max(self, proxy):
"""
将代理设置为MAX_SCORE
:param proxy: 代理
:return: 设置结果
"""
print('代理', proxy, '可用，设置为', MAX_SCORE)
mapping = {
proxy:MAX_SCORE,
}
self.db.zadd(REDIS_KEY, mapping)

关于更多Redis函数的具体操作与参数说明，建议阅读：http://redisdoc.com/sorted_set/zadd.html