python 多线程处理实验

最近要做个东西，没优化之前。跑一次要11个小时。跑的时候看cpu，内存都有富裕，就考虑用python 多线程来做。

多线程要是能省时间，也是省在等待IO 的时候，让机器做点其他的事。否则如果只是计算循环1+1=2 ,用多线程也不会提高效率，因为cpu很忙。每起一个线程，就会占一块内存，线程也不能起多了，起多了，分不到CPU时间，也没有那么多内存。

多线程实验如下：

1. 用循环来做。主线程循环50次，每循环一次，执行一次处理数据计算，休息5秒（模拟读取文件）

2. 启线程来做。做个线程列表，每循环一次，就新起一个线程，存到列表。列表存满5个或3个时，就等着。检查是否有完成的线程，如果有就清除出列表。

最后比较1与2用的时间差

实验1的代码

import threading
import time

# Define a function for the thread
def print_time(name,delay):
#print "%s worker started\n"%name
for i in range(1,10000):
for j in range(1,1000):
data = i*j

#print "%swork finshed\n"%name
return

count =0
startTime = time.time()
while count <30:
count +=1
print "current record :",count
print_time("hello",1)
time.sleep(5)

endTime = time.time()

timeDiff = endTime-startTime
print "all work is done cost:",timeDiff

实验1 执行时间约178秒

实验2：启线程的方式

1 import threading
2 import time
3
4
5 # Define a function for the thread
6 def print_time(name,delay):
7     #print "%s worker started\n"%name
8     for i in range(1,10000):
9         for j in range(1,1000):
10             data = i*j
11
12     #print "%swork finshed\n"%name
13     return
14
15
16
17 class myThread(threading.Thread):
18     def __init__(self,threadID,name,counter):
19         threading.Thread.__init__(self)
20         self.threadID = threadID
21         self.name = name
22         self.counter = counter
23     def run(self):
24         print "Starting %s\n"%self.name
25         print_time(self.name,self.counter)
26         print "Exiting %s\n"%self.name
27
28 """
29 threadLock = threading.Lock()
30 threads = []
31 thread1 = myThread(1,"Thread-1",2)
32 thread2 = myThread(2,"Thread-2",2)
33 thread1.start()
34 thread2.start()
35 threads.append(thread1)
36 threads.append(thread2)
37 for t in threads:
38     t.join()
39 print "Exiting Main Thread"
40 """
41
42 def readfile():
43     print "read file...."
44     time.sleep(5)
45
46 threadLock = threading.Lock()
47 threads = []
48 count =0
49 startTime = time.time()
50 while count <30:
51     print "current record :",count
52     name="Thread"+str(count)
53     thread = myThread(count,name,2)
54     if len(threads) <5:
55         thread.start()
56         threads.append(thread)
57         count +=1
58         readfile()
59
60         #thread.join()
61     if len(threads)>=5:
62         for t in threads:
63             if not t.isAlive():
64                 print t.name,"need remove"
65                 threads.remove(t)
66
67
68 endTime = time.time()
69
70 timeDiff = endTime-startTime
71 print "all work is done cost:",timeDiff

实验2跑完后，大概用了151秒，基本上计算利用了主线程休息的时间。

如果主线程没有sleep ,只是count+=1 ，那么用多线程反而跑的更慢。