python threading模块进行多线程编程

​

提高程序运行效率的常见方法包括多进程和多线程两种，前面已经介绍了python中的多进程编程，今天来看下多线程在python中的实现。

在使用python的多线程之前，首先要理解GIL这个概念。GIL是Global Interpreter Lock的缩写，称之为全局解释器锁，是python在开发之初为了保证数据安全而设计的，每一个python进程只有一个GIL， 同一时刻，只有拿到GIL的线程可以运行，这就使得python中的多线程无法实现真正意义上的并发。所以多线程在python中的应用场景受到了限制，只适用于处理文件IO，网络IO密集型的任务。

在python中，通过内置模块threading实现多线程处理，基本用法和多进程类似，示意如下

import threading
import urllib.request




def download_html(pathway):
print('Start download kgml')
url = 'http://rest.kegg.jp/get/{}/kgml'.format(pathway)
out = './{}.kgml'.format(pathway)
f = urllib.request.urlopen(url)
with open(out, 'w') as fp:
fp.write(f.read().decode('utf8'))




if __name__ == '__main__':
pathway = 'hsa00010'
p = threading.Thread(target = download_html, args = (pathway, ))
p.start()
p.join()
print('Finish download kgml')

通过Thread类来定义一个线程，start方法用于启动线程，join方法用于阻塞线程。上述代码展示了用一个单独的线程来下载pathway对应的kgml文件。如果有多个pathway对应的文件要下载，用多线程的写法如下

pathways = ['hsa00010', 'hsa00020', 'hsa00030', 'hsa00040', 'hsa00051', 'hsa00052', 'hsa00053']
thread_list = []
for pathway in pathways:
p = threading.Thread(target = download_html, args = (pathway, ))
thread_list.append(p)
p.start()


for thread in thread_list:
thread.join()

尽管多线程并不是真正意义上的并发，但是也有对应的方法来控制同时运行的最大线程数，代码如下

import threading
import urllib.request


def download_html(pathway, semaphore):
semaphore.acquire()
print('Start download kgml')
url = 'http://rest.kegg.jp/get/{}/kgml'.format(pathway)
out = './{}.kgml'.format(pathway)
f = urllib.request.urlopen(url)
with open(out, 'w') as fp:
fp.write(f.read().decode('utf8'))
semaphore.release()


if __name__ == '__main__':
pathways = ['hsa00010', 'hsa00020', 'hsa00030', 'hsa00040', 'hsa00051', 'hsa00052', 'hsa00053']


thread_list = []
semaphore = threading.BoundedSemaphore(3)
for pathway in pathways:
p = threading.Thread(target = download_html, args = (pathway, semaphore ))
p.start()
thread_list.append(p)
for thread in thread_list:
thread.join()


print('Finish download kgml')

多线程中变量是共享的，如果每个子进程都对同一个变量进行修改，就会出现预期之外的错误， 专业点的说法叫做产生了脏数据，示例如下

import threading
import urllib.request


# 存钱
def append_money():
global total
for cnt in range(1000000):
total += cnt
print('total money : {}'.format(total))


# 花钱
def remove_money():
global total
for cnt in range(1000000):
total -= cnt
print('total money : {}'.format(total))




if __name__ == '__main__':
total = 100
print('total money : {}'.format(total))


p1 = threading.Thread(target = append_money, args = ())
p1.start()


p2 = threading.Thread(target = remove_money, args = ())
p2.start()


p1.join()
p2.join()


print('total money : {}'.format(total))

多次运行上述代码， 每次的结果会不一样

C:\Users\Administrator\Desktop>python test.py
total money : 100
total money : 340552501975
total money : -33525835564
total money : -33525835564


C:\Users\Administrator\Desktop>python test.py
total money : 100
total money : -55696821900
total money : -197689058903
total money : -197689058903


C:\Users\Administrator\Desktop>python test.py
total money : 100
total money : -260664176670
total money : -245691977911
total money : -245691977911

多个进程同时对一个变量进行修改，就是会存在脏数据的隐患，为此，我们需要对线程加锁，保证每次只有一个线程对变量进行修改，代码如下

import threading
import urllib.request


def append_money(lock):
lock.acquire()
global total
for cnt in range(1000000):
total += cnt
print('total money : {}'.format(total))
lock.release()




def remove_money(lock):
lock.acquire()
global total
for cnt in range(1000000):
total -= cnt
print('total money : {}'.format(total))
lock.release()




if __name__ == '__main__':
total = 100
print('total money : {}'.format(total))
lock = threading.Lock()
p1 = threading.Thread(target = append_money, args = (lock, ))
p1.start()


p2 = threading.Thread(target = remove_money, args = (lock, ))
p2.start()


p1.join()
p2.join()


print('total money : {}'.format(total))

添加了锁之后，就可以保证多次运行的结果都和预期保持一致了

C:\Users\Administrator\Desktop>python test.py
total money : 100
total money : 499999500100
total money : 100
total money : 100


C:\Users\Administrator\Desktop>python test.py
total money : 100
total money : 499999500100
total money : 100
total money : 100


C:\Users\Administrator\Desktop>python test.py
total money : 100
total money : 499999500100
total money : 100
total money : 100

实际开发中，主要采用python的多线程来完成多个url下载的任务，这种任务属于网路IO密集型，用多线程可以提高速度。如果涉及到多个线程修改同一个变量的情况，通过给线程加锁的方式来保证结果的准确性。

·end·

—如

一个只分享干货的

生信公众号