Python自学:使用多进程处理 multiprocessing

1. 使用多进程执行函数

以下代码没有使用多进程。

python 复制代码
import time

start = time.perf_counter()

def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleep...')

do_something()
do_something()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1 second...

Done Sleep...

Sleeping 1 second...

Done Sleep...

Finished in 2.03 second(s)

以下代码使用了多进程。

python 复制代码
import time
import multiprocessing


def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleep...')


if __name__ == '__main__':

    start = time.perf_counter()

    p1 = multiprocessing.Process(target=do_something)
    p2 = multiprocessing.Process(target=do_something)

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1 second...

Sleeping 1 second...

Done Sleep...

Done Sleep...

Finished in 1.07 second(s)

2. 使用loop创建多个进程,并在函数中传入参数。

python 复制代码
import time
import multiprocessing


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    print('Done Sleep...')


if __name__ == '__main__':

    start = time.perf_counter()

    processes = []

    for _ in range(10):
        p = multiprocessing.Process(target=do_something, args=[1.5])
        p.start()
        processes.append(p)

    for process in processes:
        process.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Finished in 1.62 second(s)

3. 使用进程池实现多进程

python 复制代码
import time
import concurrent.futures


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleep...{seconds}'


if __name__ == '__main__':

    start = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)

        for result in results:
            print(result)

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 5 second(s)...

Sleeping 4 second(s)...

Sleeping 3 second(s)...

Sleeping 2 second(s)...

Sleeping 1 second(s)...

Done Sleep...5

Done Sleep...4

Done Sleep...3

Done Sleep...2

Done Sleep...1

Finished in 5.14 second(s)

4. 使用多进程处理图片

以下代码展示了没有使用多进程处理图片

python 复制代码
import time
from PIL import Image, ImageFilter

img_names = [
    'photo-1516117172878-fd2c41f4a759.jpg',
    'photo-1532009324734-20a7a5813719.jpg',
    'photo-1524429656589-6633a470097c.jpg',
    'photo-1530224264768-7ff8c1789d79.jpg',
    'photo-1564135624576-c5c88640f235.jpg',
    'photo-1541698444083-023c97d3f4b6.jpg',
    'photo-1522364723953-452d3431c267.jpg',
    'photo-1493976040374-85c8e12f0c0e.jpg',
    'photo-1504198453319-5ce911bafcde.jpg',
    'photo-1530122037265-a5f1f91d3b99.jpg',
    'photo-1516972810927-80185027ca84.jpg',
    'photo-1550439062-609e1531270e.jpg',
    'photo-1549692520-acc6669e2f0c.jpg'
]

t1 = time.perf_counter()

size = (1200, 1200)

for img_name in img_names:
    img = Image.open(img_name)

    img = img.filter(ImageFilter.GaussianBlur(15))

    img.thumbnail(size)

    img.save(f'processed/{img_name}')
    print(f'{img_name} was processed...')

t2 = time.perf_counter()

print(f'Finished in {t2-t1} seconds')

输出为:

photo-1516117172878-fd2c41f4a759.jpg was processed...

photo-1532009324734-20a7a5813719.jpg was processed...

photo-1524429656589-6633a470097c.jpg was processed...

photo-1530224264768-7ff8c1789d79.jpg was processed...

photo-1564135624576-c5c88640f235.jpg was processed...

photo-1541698444083-023c97d3f4b6.jpg was processed...

photo-1522364723953-452d3431c267.jpg was processed...

photo-1493976040374-85c8e12f0c0e.jpg was processed...

photo-1504198453319-5ce911bafcde.jpg was processed...

photo-1530122037265-a5f1f91d3b99.jpg was processed...

photo-1516972810927-80185027ca84.jpg was processed...

photo-1550439062-609e1531270e.jpg was processed...

photo-1549692520-acc6669e2f0c.jpg was processed...

Finished in 13.196055100299418 seconds

使用多进程的方式处理图片

python 复制代码
import time
import concurrent.futures
from PIL import Image, ImageFilter

img_names = [
    'photo-1516117172878-fd2c41f4a759.jpg',
    'photo-1532009324734-20a7a5813719.jpg',
    'photo-1524429656589-6633a470097c.jpg',
    'photo-1530224264768-7ff8c1789d79.jpg',
    'photo-1564135624576-c5c88640f235.jpg',
    'photo-1541698444083-023c97d3f4b6.jpg',
    'photo-1522364723953-452d3431c267.jpg',
    'photo-1493976040374-85c8e12f0c0e.jpg',
    'photo-1504198453319-5ce911bafcde.jpg',
    'photo-1530122037265-a5f1f91d3b99.jpg',
    'photo-1516972810927-80185027ca84.jpg',
    'photo-1550439062-609e1531270e.jpg',
    'photo-1549692520-acc6669e2f0c.jpg'
]


def process_image(img_name):
    
    img = Image.open(img_name)

    img = img.filter(ImageFilter.GaussianBlur(15))

    img.thumbnail((1200, 1200))

    img.save(f'processed/{img_name}')
    print(f'{img_name} was processed...')

if __name__ == '__main__':
    t1 = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(process_image, img_names)

    t2 = time.perf_counter()

    print(f'Finished in {t2-t1} seconds')

输出为:

photo-1516117172878-fd2c41f4a759.jpg was processed...

photo-1516972810927-80185027ca84.jpg was processed...

photo-1524429656589-6633a470097c.jpg was processed...

photo-1522364723953-452d3431c267.jpg was processed...

photo-1532009324734-20a7a5813719.jpg was processed...

photo-1530122037265-a5f1f91d3b99.jpg was processed...

photo-1530224264768-7ff8c1789d79.jpg was processed...

photo-1564135624576-c5c88640f235.jpg was processed...

photo-1550439062-609e1531270e.jpg was processed...

photo-1541698444083-023c97d3f4b6.jpg was processed...

photo-1549692520-acc6669e2f0c.jpg was processed...

photo-1504198453319-5ce911bafcde.jpg was processed...

photo-1493976040374-85c8e12f0c0e.jpg was processed...

Finished in 2.651644399855286 seconds

我们可以看到,处理时间缩短为原来的1/5,大大提高了图片处理的速度。

相关推荐
凛铄linshuo2 分钟前
爬虫简单实操2——以贴吧为例爬取“某吧”前10页的网页代码
爬虫·python·学习
牛客企业服务5 分钟前
2025年AI面试推荐榜单,数字化招聘转型优选
人工智能·python·算法·面试·职场和发展·金融·求职招聘
charlie11451419116 分钟前
深入理解Qt的SetWindowsFlags函数
开发语言·c++·qt·原理分析
胡斌附体17 分钟前
linux测试端口是否可被外部访问
linux·运维·服务器·python·测试·端口测试·临时服务器
likeGhee1 小时前
python缓存装饰器实现方案
开发语言·python·缓存
whoarethenext1 小时前
使用 C++/Faiss 加速海量 MFCC 特征的相似性搜索
开发语言·c++·faiss
项目題供诗1 小时前
黑马python(二十五)
开发语言·python
读书点滴1 小时前
笨方法学python -练习14
java·前端·python
慌糖1 小时前
RabbitMQ:消息队列的轻量级王者
开发语言·javascript·ecmascript
笑衬人心。1 小时前
Ubuntu 22.04 修改默认 Python 版本为 Python3 笔记
笔记·python·ubuntu