Python自学:使用多进程处理 multiprocessing

1. 使用多进程执行函数

以下代码没有使用多进程。

python 复制代码
import time

start = time.perf_counter()

def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleep...')

do_something()
do_something()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1 second...

Done Sleep...

Sleeping 1 second...

Done Sleep...

Finished in 2.03 second(s)

以下代码使用了多进程。

python 复制代码
import time
import multiprocessing


def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleep...')


if __name__ == '__main__':

    start = time.perf_counter()

    p1 = multiprocessing.Process(target=do_something)
    p2 = multiprocessing.Process(target=do_something)

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1 second...

Sleeping 1 second...

Done Sleep...

Done Sleep...

Finished in 1.07 second(s)

2. 使用loop创建多个进程,并在函数中传入参数。

python 复制代码
import time
import multiprocessing


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    print('Done Sleep...')


if __name__ == '__main__':

    start = time.perf_counter()

    processes = []

    for _ in range(10):
        p = multiprocessing.Process(target=do_something, args=[1.5])
        p.start()
        processes.append(p)

    for process in processes:
        process.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Finished in 1.62 second(s)

3. 使用进程池实现多进程

python 复制代码
import time
import concurrent.futures


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleep...{seconds}'


if __name__ == '__main__':

    start = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)

        for result in results:
            print(result)

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 5 second(s)...

Sleeping 4 second(s)...

Sleeping 3 second(s)...

Sleeping 2 second(s)...

Sleeping 1 second(s)...

Done Sleep...5

Done Sleep...4

Done Sleep...3

Done Sleep...2

Done Sleep...1

Finished in 5.14 second(s)

4. 使用多进程处理图片

以下代码展示了没有使用多进程处理图片

python 复制代码
import time
from PIL import Image, ImageFilter

img_names = [
    'photo-1516117172878-fd2c41f4a759.jpg',
    'photo-1532009324734-20a7a5813719.jpg',
    'photo-1524429656589-6633a470097c.jpg',
    'photo-1530224264768-7ff8c1789d79.jpg',
    'photo-1564135624576-c5c88640f235.jpg',
    'photo-1541698444083-023c97d3f4b6.jpg',
    'photo-1522364723953-452d3431c267.jpg',
    'photo-1493976040374-85c8e12f0c0e.jpg',
    'photo-1504198453319-5ce911bafcde.jpg',
    'photo-1530122037265-a5f1f91d3b99.jpg',
    'photo-1516972810927-80185027ca84.jpg',
    'photo-1550439062-609e1531270e.jpg',
    'photo-1549692520-acc6669e2f0c.jpg'
]

t1 = time.perf_counter()

size = (1200, 1200)

for img_name in img_names:
    img = Image.open(img_name)

    img = img.filter(ImageFilter.GaussianBlur(15))

    img.thumbnail(size)

    img.save(f'processed/{img_name}')
    print(f'{img_name} was processed...')

t2 = time.perf_counter()

print(f'Finished in {t2-t1} seconds')

输出为:

photo-1516117172878-fd2c41f4a759.jpg was processed...

photo-1532009324734-20a7a5813719.jpg was processed...

photo-1524429656589-6633a470097c.jpg was processed...

photo-1530224264768-7ff8c1789d79.jpg was processed...

photo-1564135624576-c5c88640f235.jpg was processed...

photo-1541698444083-023c97d3f4b6.jpg was processed...

photo-1522364723953-452d3431c267.jpg was processed...

photo-1493976040374-85c8e12f0c0e.jpg was processed...

photo-1504198453319-5ce911bafcde.jpg was processed...

photo-1530122037265-a5f1f91d3b99.jpg was processed...

photo-1516972810927-80185027ca84.jpg was processed...

photo-1550439062-609e1531270e.jpg was processed...

photo-1549692520-acc6669e2f0c.jpg was processed...

Finished in 13.196055100299418 seconds

使用多进程的方式处理图片

python 复制代码
import time
import concurrent.futures
from PIL import Image, ImageFilter

img_names = [
    'photo-1516117172878-fd2c41f4a759.jpg',
    'photo-1532009324734-20a7a5813719.jpg',
    'photo-1524429656589-6633a470097c.jpg',
    'photo-1530224264768-7ff8c1789d79.jpg',
    'photo-1564135624576-c5c88640f235.jpg',
    'photo-1541698444083-023c97d3f4b6.jpg',
    'photo-1522364723953-452d3431c267.jpg',
    'photo-1493976040374-85c8e12f0c0e.jpg',
    'photo-1504198453319-5ce911bafcde.jpg',
    'photo-1530122037265-a5f1f91d3b99.jpg',
    'photo-1516972810927-80185027ca84.jpg',
    'photo-1550439062-609e1531270e.jpg',
    'photo-1549692520-acc6669e2f0c.jpg'
]


def process_image(img_name):
    
    img = Image.open(img_name)

    img = img.filter(ImageFilter.GaussianBlur(15))

    img.thumbnail((1200, 1200))

    img.save(f'processed/{img_name}')
    print(f'{img_name} was processed...')

if __name__ == '__main__':
    t1 = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(process_image, img_names)

    t2 = time.perf_counter()

    print(f'Finished in {t2-t1} seconds')

输出为:

photo-1516117172878-fd2c41f4a759.jpg was processed...

photo-1516972810927-80185027ca84.jpg was processed...

photo-1524429656589-6633a470097c.jpg was processed...

photo-1522364723953-452d3431c267.jpg was processed...

photo-1532009324734-20a7a5813719.jpg was processed...

photo-1530122037265-a5f1f91d3b99.jpg was processed...

photo-1530224264768-7ff8c1789d79.jpg was processed...

photo-1564135624576-c5c88640f235.jpg was processed...

photo-1550439062-609e1531270e.jpg was processed...

photo-1541698444083-023c97d3f4b6.jpg was processed...

photo-1549692520-acc6669e2f0c.jpg was processed...

photo-1504198453319-5ce911bafcde.jpg was processed...

photo-1493976040374-85c8e12f0c0e.jpg was processed...

Finished in 2.651644399855286 seconds

我们可以看到,处理时间缩短为原来的1/5,大大提高了图片处理的速度。

相关推荐
褚翾澜几秒前
Ruby语言的代码重构
开发语言·后端·golang
utmhikari1 小时前
【日常随笔】万字长文,如何用pyside6开发一个python桌面工具
前端·python·pyqt
小杨4044 小时前
python入门系列十四(多进程)
人工智能·python·pycharm
用户277844910499318 小时前
借助DeepSeek智能生成测试用例:从提示词到Excel表格的全流程实践
人工智能·python
JavaEdge在掘金21 小时前
ssl.SSLCertVerificationError报错解决方案
python
我不会编程55521 小时前
Python Cookbook-5.1 对字典排序
开发语言·数据结构·python
李少兄21 小时前
Unirest:优雅的Java HTTP客户端库
java·开发语言·http
老歌老听老掉牙1 天前
平面旋转与交线投影夹角计算
python·线性代数·平面·sympy
满怀10151 天前
Python入门(7):模块
python
无名之逆1 天前
Rust 开发提效神器:lombok-macros 宏库
服务器·开发语言·前端·数据库·后端·python·rust