python multiprocessing.Pool - UnpickleableError: Cannot pickle <type 'thread.lock'> objects
2012-08-22 11:35
971 查看
相关代码:
max_threads_num为进程中的线程个数,db为数据库操作相关的类,i为数字,seeds为db执行select后返回的元组。
修改第3行为第5行,及相关代码之后运行时报错。
错误信息:
开始以为错误是由于传递的最后一个参数由i改为seeds引起,google后发现元组、列表、字符串类型都是pickleable的。
错误是传递的db类型变为thread.lock造成的,而之前的db没有进行数据库操作故没有连接数据库。
于是,在db进行数据库操作后断开数据库连接后再传入Pool就好了。
参考:
http://stackoverflow.com/questions/7865430/multiprocessing-pool-picklingerror-cant-pickle-type-thread-lock-attribu
原文:/article/6987037.html
pool = multiprocessing.Pool(processes=pool_size, initializer=start_process) #process_cralwer = pool.apply_async(dianshang_current.main, args=(max_threads_num, db, i)) process_cralwer = pool.apply_async(dianshang_current.main, args=(max_threads_num, db, seeds))
max_threads_num为进程中的线程个数,db为数据库操作相关的类,i为数字,seeds为db执行select后返回的元组。
修改第3行为第5行,及相关代码之后运行时报错。
错误信息:
Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/multiprocessing/pool.py", line 226, in _handle_tasks put(task) UnpickleableError: Cannot pickle <type 'thread.lock'> objects
开始以为错误是由于传递的最后一个参数由i改为seeds引起,google后发现元组、列表、字符串类型都是pickleable的。
错误是传递的db类型变为thread.lock造成的,而之前的db没有进行数据库操作故没有连接数据库。
于是,在db进行数据库操作后断开数据库连接后再传入Pool就好了。
参考:
http://stackoverflow.com/questions/7865430/multiprocessing-pool-picklingerror-cant-pickle-type-thread-lock-attribu
原文:/article/6987037.html
相关文章推荐
- python multiprocessing.Pool - PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
- PicklingError: Can't pickle <type 'generator'>: it's not found as __builtin_
- python报TypeError: cannot convert the series to <class 'float'>
- 调查 PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
- python3 PicklingError: Can't pickle <function <lambda> at......> attribute lookup <lambda> on __main
- python中multiprocessing.pool函数介绍<转>
- configure: error: Cannot find OpenSSL's <evp.h>
- Python字符串与数字拼接报错解决TypeError: cannot concatenate 'str' and 'int' objects
- 运行 python 脚本错误:urllib2.URLErroe:<urlopen error unknown url type : https>
- :-1: error: cannot open C:\Users\<username>\AppData\Local\Temp\main.obj.5136.0.jom for write
- Uncaught TypeError: Cannot assign to read only property 'exports' of object '#<Object>'
- 使用Gevent程序正常运行出现这个异常KeyError(155880780,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored解决办法
- python报错:<IndentationError: unindent does not match any outer indentation level>
- pickle.PicklingError: Can't pickle <function <lambda>
- 【Python】 URLError: <urlopen error timed out> 错误
- python调用py方法,报错【ValueError: no such test method in <class 'mytestcase.MyTestCase'>: runTest】
- datanode----nitialization failed for Block pool <registering> (Datanode Uuid unassigned) service to
- PYTHON 3 爬虫出现<urlopen error no host given> 问题
- 解决php编译报错configure: error: Cannot find OpenSSL's <evp.h>
- python2,socket多进程的错误pickle.PicklingError: Can't pickle <built-in method recvfrom_into of _socket.soc