Python自学:使用多进程处理 multiprocessing

1. 使用多进程执行函数

以下代码没有使用多进程。

python 复制代码
import time

start = time.perf_counter()

def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleep...')

do_something()
do_something()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1 second...

Done Sleep...

Sleeping 1 second...

Done Sleep...

Finished in 2.03 second(s)

以下代码使用了多进程。

python 复制代码
import time
import multiprocessing


def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleep...')


if __name__ == '__main__':

    start = time.perf_counter()

    p1 = multiprocessing.Process(target=do_something)
    p2 = multiprocessing.Process(target=do_something)

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1 second...

Sleeping 1 second...

Done Sleep...

Done Sleep...

Finished in 1.07 second(s)

2. 使用loop创建多个进程,并在函数中传入参数。

python 复制代码
import time
import multiprocessing


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    print('Done Sleep...')


if __name__ == '__main__':

    start = time.perf_counter()

    processes = []

    for _ in range(10):
        p = multiprocessing.Process(target=do_something, args=[1.5])
        p.start()
        processes.append(p)

    for process in processes:
        process.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Finished in 1.62 second(s)

3. 使用进程池实现多进程

python 复制代码
import time
import concurrent.futures


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleep...{seconds}'


if __name__ == '__main__':

    start = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)

        for result in results:
            print(result)

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 5 second(s)...

Sleeping 4 second(s)...

Sleeping 3 second(s)...

Sleeping 2 second(s)...

Sleeping 1 second(s)...

Done Sleep...5

Done Sleep...4

Done Sleep...3

Done Sleep...2

Done Sleep...1

Finished in 5.14 second(s)

4. 使用多进程处理图片

以下代码展示了没有使用多进程处理图片

python 复制代码
import time
from PIL import Image, ImageFilter

img_names = [
    'photo-1516117172878-fd2c41f4a759.jpg',
    'photo-1532009324734-20a7a5813719.jpg',
    'photo-1524429656589-6633a470097c.jpg',
    'photo-1530224264768-7ff8c1789d79.jpg',
    'photo-1564135624576-c5c88640f235.jpg',
    'photo-1541698444083-023c97d3f4b6.jpg',
    'photo-1522364723953-452d3431c267.jpg',
    'photo-1493976040374-85c8e12f0c0e.jpg',
    'photo-1504198453319-5ce911bafcde.jpg',
    'photo-1530122037265-a5f1f91d3b99.jpg',
    'photo-1516972810927-80185027ca84.jpg',
    'photo-1550439062-609e1531270e.jpg',
    'photo-1549692520-acc6669e2f0c.jpg'
]

t1 = time.perf_counter()

size = (1200, 1200)

for img_name in img_names:
    img = Image.open(img_name)

    img = img.filter(ImageFilter.GaussianBlur(15))

    img.thumbnail(size)

    img.save(f'processed/{img_name}')
    print(f'{img_name} was processed...')

t2 = time.perf_counter()

print(f'Finished in {t2-t1} seconds')

输出为:

photo-1516117172878-fd2c41f4a759.jpg was processed...

photo-1532009324734-20a7a5813719.jpg was processed...

photo-1524429656589-6633a470097c.jpg was processed...

photo-1530224264768-7ff8c1789d79.jpg was processed...

photo-1564135624576-c5c88640f235.jpg was processed...

photo-1541698444083-023c97d3f4b6.jpg was processed...

photo-1522364723953-452d3431c267.jpg was processed...

photo-1493976040374-85c8e12f0c0e.jpg was processed...

photo-1504198453319-5ce911bafcde.jpg was processed...

photo-1530122037265-a5f1f91d3b99.jpg was processed...

photo-1516972810927-80185027ca84.jpg was processed...

photo-1550439062-609e1531270e.jpg was processed...

photo-1549692520-acc6669e2f0c.jpg was processed...

Finished in 13.196055100299418 seconds

使用多进程的方式处理图片

python 复制代码
import time
import concurrent.futures
from PIL import Image, ImageFilter

img_names = [
    'photo-1516117172878-fd2c41f4a759.jpg',
    'photo-1532009324734-20a7a5813719.jpg',
    'photo-1524429656589-6633a470097c.jpg',
    'photo-1530224264768-7ff8c1789d79.jpg',
    'photo-1564135624576-c5c88640f235.jpg',
    'photo-1541698444083-023c97d3f4b6.jpg',
    'photo-1522364723953-452d3431c267.jpg',
    'photo-1493976040374-85c8e12f0c0e.jpg',
    'photo-1504198453319-5ce911bafcde.jpg',
    'photo-1530122037265-a5f1f91d3b99.jpg',
    'photo-1516972810927-80185027ca84.jpg',
    'photo-1550439062-609e1531270e.jpg',
    'photo-1549692520-acc6669e2f0c.jpg'
]


def process_image(img_name):
    
    img = Image.open(img_name)

    img = img.filter(ImageFilter.GaussianBlur(15))

    img.thumbnail((1200, 1200))

    img.save(f'processed/{img_name}')
    print(f'{img_name} was processed...')

if __name__ == '__main__':
    t1 = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(process_image, img_names)

    t2 = time.perf_counter()

    print(f'Finished in {t2-t1} seconds')

输出为:

photo-1516117172878-fd2c41f4a759.jpg was processed...

photo-1516972810927-80185027ca84.jpg was processed...

photo-1524429656589-6633a470097c.jpg was processed...

photo-1522364723953-452d3431c267.jpg was processed...

photo-1532009324734-20a7a5813719.jpg was processed...

photo-1530122037265-a5f1f91d3b99.jpg was processed...

photo-1530224264768-7ff8c1789d79.jpg was processed...

photo-1564135624576-c5c88640f235.jpg was processed...

photo-1550439062-609e1531270e.jpg was processed...

photo-1541698444083-023c97d3f4b6.jpg was processed...

photo-1549692520-acc6669e2f0c.jpg was processed...

photo-1504198453319-5ce911bafcde.jpg was processed...

photo-1493976040374-85c8e12f0c0e.jpg was processed...

Finished in 2.651644399855286 seconds

我们可以看到,处理时间缩短为原来的1/5,大大提高了图片处理的速度。

相关推荐
Niuguangshuo11 分钟前
Python 设计模式:访问者模式
python·设计模式·访问者模式
Jamesvalley14 分钟前
【Django】新增字段后兼容旧接口 This field is required
后端·python·django
JAVA学习通40 分钟前
JAVA多线程(8.0)
java·开发语言
Luck_ff081043 分钟前
【Python爬虫详解】第四篇:使用解析库提取网页数据——BeautifuSoup
开发语言·爬虫·python
学渣676561 小时前
什么时候使用Python 虚拟环境(venv)而不用conda
开发语言·python·conda
想睡hhh1 小时前
c++STL——stack、queue、priority_queue的模拟实现
开发语言·c++·stl
悲喜自渡7211 小时前
线性代数(一些别的应该关注的点)
python·线性代数·机器学习
小鹿鹿啊1 小时前
C语言编程--14.电话号码的字母组合
c语言·开发语言·算法
Sunlight_7771 小时前
第六章 QT基础:6、QT的Qt 时钟编程
开发语言·qt·命令模式
cloues break.1 小时前
C++初阶----模板初阶
java·开发语言·c++