Python自学:使用多进程处理 multiprocessing

1. 使用多进程执行函数

以下代码没有使用多进程。

python 复制代码
import time

start = time.perf_counter()

def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleep...')

do_something()
do_something()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1 second...

Done Sleep...

Sleeping 1 second...

Done Sleep...

Finished in 2.03 second(s)

以下代码使用了多进程。

python 复制代码
import time
import multiprocessing


def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleep...')


if __name__ == '__main__':

    start = time.perf_counter()

    p1 = multiprocessing.Process(target=do_something)
    p2 = multiprocessing.Process(target=do_something)

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1 second...

Sleeping 1 second...

Done Sleep...

Done Sleep...

Finished in 1.07 second(s)

2. 使用loop创建多个进程,并在函数中传入参数。

python 复制代码
import time
import multiprocessing


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    print('Done Sleep...')


if __name__ == '__main__':

    start = time.perf_counter()

    processes = []

    for _ in range(10):
        p = multiprocessing.Process(target=do_something, args=[1.5])
        p.start()
        processes.append(p)

    for process in processes:
        process.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Sleeping 1.5 second(s)...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Done Sleep...

Finished in 1.62 second(s)

3. 使用进程池实现多进程

python 复制代码
import time
import concurrent.futures


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleep...{seconds}'


if __name__ == '__main__':

    start = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)

        for result in results:
            print(result)

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

输出为:

Sleeping 5 second(s)...

Sleeping 4 second(s)...

Sleeping 3 second(s)...

Sleeping 2 second(s)...

Sleeping 1 second(s)...

Done Sleep...5

Done Sleep...4

Done Sleep...3

Done Sleep...2

Done Sleep...1

Finished in 5.14 second(s)

4. 使用多进程处理图片

以下代码展示了没有使用多进程处理图片

python 复制代码
import time
from PIL import Image, ImageFilter

img_names = [
    'photo-1516117172878-fd2c41f4a759.jpg',
    'photo-1532009324734-20a7a5813719.jpg',
    'photo-1524429656589-6633a470097c.jpg',
    'photo-1530224264768-7ff8c1789d79.jpg',
    'photo-1564135624576-c5c88640f235.jpg',
    'photo-1541698444083-023c97d3f4b6.jpg',
    'photo-1522364723953-452d3431c267.jpg',
    'photo-1493976040374-85c8e12f0c0e.jpg',
    'photo-1504198453319-5ce911bafcde.jpg',
    'photo-1530122037265-a5f1f91d3b99.jpg',
    'photo-1516972810927-80185027ca84.jpg',
    'photo-1550439062-609e1531270e.jpg',
    'photo-1549692520-acc6669e2f0c.jpg'
]

t1 = time.perf_counter()

size = (1200, 1200)

for img_name in img_names:
    img = Image.open(img_name)

    img = img.filter(ImageFilter.GaussianBlur(15))

    img.thumbnail(size)

    img.save(f'processed/{img_name}')
    print(f'{img_name} was processed...')

t2 = time.perf_counter()

print(f'Finished in {t2-t1} seconds')

输出为:

photo-1516117172878-fd2c41f4a759.jpg was processed...

photo-1532009324734-20a7a5813719.jpg was processed...

photo-1524429656589-6633a470097c.jpg was processed...

photo-1530224264768-7ff8c1789d79.jpg was processed...

photo-1564135624576-c5c88640f235.jpg was processed...

photo-1541698444083-023c97d3f4b6.jpg was processed...

photo-1522364723953-452d3431c267.jpg was processed...

photo-1493976040374-85c8e12f0c0e.jpg was processed...

photo-1504198453319-5ce911bafcde.jpg was processed...

photo-1530122037265-a5f1f91d3b99.jpg was processed...

photo-1516972810927-80185027ca84.jpg was processed...

photo-1550439062-609e1531270e.jpg was processed...

photo-1549692520-acc6669e2f0c.jpg was processed...

Finished in 13.196055100299418 seconds

使用多进程的方式处理图片

python 复制代码
import time
import concurrent.futures
from PIL import Image, ImageFilter

img_names = [
    'photo-1516117172878-fd2c41f4a759.jpg',
    'photo-1532009324734-20a7a5813719.jpg',
    'photo-1524429656589-6633a470097c.jpg',
    'photo-1530224264768-7ff8c1789d79.jpg',
    'photo-1564135624576-c5c88640f235.jpg',
    'photo-1541698444083-023c97d3f4b6.jpg',
    'photo-1522364723953-452d3431c267.jpg',
    'photo-1493976040374-85c8e12f0c0e.jpg',
    'photo-1504198453319-5ce911bafcde.jpg',
    'photo-1530122037265-a5f1f91d3b99.jpg',
    'photo-1516972810927-80185027ca84.jpg',
    'photo-1550439062-609e1531270e.jpg',
    'photo-1549692520-acc6669e2f0c.jpg'
]


def process_image(img_name):
    
    img = Image.open(img_name)

    img = img.filter(ImageFilter.GaussianBlur(15))

    img.thumbnail((1200, 1200))

    img.save(f'processed/{img_name}')
    print(f'{img_name} was processed...')

if __name__ == '__main__':
    t1 = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(process_image, img_names)

    t2 = time.perf_counter()

    print(f'Finished in {t2-t1} seconds')

输出为:

photo-1516117172878-fd2c41f4a759.jpg was processed...

photo-1516972810927-80185027ca84.jpg was processed...

photo-1524429656589-6633a470097c.jpg was processed...

photo-1522364723953-452d3431c267.jpg was processed...

photo-1532009324734-20a7a5813719.jpg was processed...

photo-1530122037265-a5f1f91d3b99.jpg was processed...

photo-1530224264768-7ff8c1789d79.jpg was processed...

photo-1564135624576-c5c88640f235.jpg was processed...

photo-1550439062-609e1531270e.jpg was processed...

photo-1541698444083-023c97d3f4b6.jpg was processed...

photo-1549692520-acc6669e2f0c.jpg was processed...

photo-1504198453319-5ce911bafcde.jpg was processed...

photo-1493976040374-85c8e12f0c0e.jpg was processed...

Finished in 2.651644399855286 seconds

我们可以看到,处理时间缩短为原来的1/5,大大提高了图片处理的速度。

相关推荐
聊无生2 分钟前
JavaSrcipt 函数高级
开发语言·前端·javascript
小安同学iter11 分钟前
Java进阶五 -IO流
java·开发语言·intellij-idea
Hello World and You11 分钟前
R ggplot2 绘图细节 geom_text展示不全 y轴坐标细节 x轴标题
开发语言·r语言
sukalot15 分钟前
windows C#-异步文件访问
开发语言·c#
搏博25 分钟前
Python3.9.13与深度学习框架TensorFlow的完整详细安装教程
python·tensorflow
熬夜学编程的小王34 分钟前
【C++篇】从基础到进阶:全面掌握C++ List容器的使用
开发语言·c++·list·双向链表·迭代器失效
胜天半月子35 分钟前
Python | 结合动态加载importlib模块来理解inspect模块的使用
python·importlib·inspect
Jurio.35 分钟前
Conda 管理项目环境
人工智能·python·深度学习·conda·virtualenv·pip
悄悄敲敲敲36 分钟前
C++:智能指针
开发语言·c++
B站计算机毕业设计超人39 分钟前
计算机毕业设计SparkStreaming+Kafka新能源汽车推荐系统 汽车数据分析可视化大屏 新能源汽车推荐系统 汽车爬虫 汽车大数据 机器学习
数据仓库·爬虫·python·数据分析·kafka·数据可视化·推荐算法