“GIL”如何影響帶有i/o系結任務的Pythonasyncio`run_in

關于Python ayncio 的這個代碼示例： run_in_executor

import asyncio
import concurrent.futures

def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open('/dev/urandom', 'rb') as f:
        return f.read(100)

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    ## Options:

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(
        None, blocking_io)
    print('default thread pool', result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

asyncio.run(main())

該示例（在注釋中）建議使用運行 i/o 系結函式ThreadPoolExecutor，使用ProcessPoolExecutor. 我想通過三個問題來驗證我對這背后原因的理解：

這些建議并不是真正的建議，否則事件回圈會阻塞。因此，我們將失去事件編程的主要好處，對嗎？
作為單獨的執行緒運行 io/bound 任務，需要以下假設：i/o 呼叫將釋放 GIL，對嗎？因為除此之外，作業系統將無法在事件回圈和這個新的單獨執行緒之間進行背景關系切換。
如果第 2 點的答案是肯定的，那么如何確定 i/o 呼叫是否釋放了 GIL？

uj5u.com熱心網友回復：

這些建議并不是真正的建議，否則事件回圈會阻塞。因此，我們將失去事件編程的主要好處，對嗎？

如果您在協程中呼叫阻塞（I/O 和 CPU 阻塞）函式而不等待執行程式，則事件回圈將阻塞。在這方面，是的，您不應該允許這種情況發生。

我的建議是，對于每種型別的阻塞代碼，它都是一種執行器：將 ProcessPoolExecutor 用于 CPU 系結的東西，將 ThreadPoolExecutor 用于 I/O 系結的東西。

作為單獨的執行緒運行 io/bound 任務，需要以下假設：i/o 呼叫將釋放 GIL，對嗎？因為除此之外，作業系統將無法在事件回圈和這個新的單獨執行緒之間進行背景關系切換。

當談到多執行緒時，Python 將在很短的時間內在執行緒之間切換而不釋放 GIL。但是如果一個或多個執行緒有 I/O（或 C 代碼），那么 GIL 將被釋放，允許解釋器花更多時間在需要它的執行緒上。

底線是：

您可以在 executor 中運行任何阻塞代碼，它不會阻塞事件回圈。您獲得并發性，但可能會或可能不會獲得性能。
例如，如果您在 ThreadPoolExecutor 中運行受 CPU 限制的代碼，由于 GIL，您將不會從并發中獲得性能優勢。要獲得 CPU 密集型內容的性能，您應該使用 ProcessPoolExecutor。
但是 I/O-bound 可以在 ThreadPoolExecutor 中運行并且您可以獲得性能。這里沒有必要使用更重的 ProcessPoolExecutor。

我寫了一個例子來演示它是如何作業的：

import sys
import asyncio
import time
import concurrent.futures
import requests
from contextlib import contextmanager

process_pool = concurrent.futures.ProcessPoolExecutor(2)
thread_pool = concurrent.futures.ThreadPoolExecutor(2)


def io_bound():
    for i in range(3):
        requests.get("https://httpbin.org/delay/0.4")  # I/O blocking
        print(f"I/O bound {i}")
        sys.stdout.flush()


def cpu_bound():
    for i in range(3):
        sum(i * i for i in range(10 ** 7))  # CPU blocking
        print(f"CPU bound {i}")
        sys.stdout.flush()


async def run_as_is(func):
    func()


async def run_in_process(func):
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(process_pool, func)


async def run_in_thread(func):
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(thread_pool, func)


@contextmanager
def print_time():
    start = time.time()
    yield
    finished = time.time() - start
    print(f"Finished in {round(finished, 1)}\n")


async def main():
    print("Wrong due to blocking code in coroutine,")
    print(
        "you get neither performance, nor concurrency (which breaks async nature of the code)"
    )
    print("don't allow this to happen")
    with print_time():
        await asyncio.gather(run_as_is(cpu_bound), run_as_is(io_bound))

    print("CPU bound works concurrently with threads,")
    print("but you gain no performance due to GIL")
    with print_time():
        await asyncio.gather(run_in_thread(cpu_bound), run_in_thread(cpu_bound))

    print("To get perfromance for CPU-bound,")
    print("use process executor")
    with print_time():
        await asyncio.gather(run_in_process(cpu_bound), run_in_process(cpu_bound))

    print("I/O bound will gain benefit from processes as well...")
    with print_time():
        await asyncio.gather(run_in_process(io_bound), run_in_process(io_bound))

    print(
        "... but there's no need in processes since you can use lighter threads for I/O"
    )
    with print_time():
        await asyncio.gather(run_in_thread(io_bound), run_in_thread(io_bound))

    print("Long story short,")
    print("Use processes for CPU bound due to GIL")
    print(
        "and use threads for I/O bound since you benefit from concurrency regardless of GIL"
    )
    with print_time():
        await asyncio.gather(run_in_thread(io_bound), run_in_process(cpu_bound))


if __name__ == "__main__":
    asyncio.run(main())

輸出：

Wrong due to blocking code in coroutine,
you get neither performance, nor concurrency (which breaks async nature of the code)
don't allow this to happen
CPU bound 0
CPU bound 1
CPU bound 2
I/O bound 0
I/O bound 1
I/O bound 2
Finished in 5.3

CPU bound works concurrently with threads,
but you gain no performance due to GIL
CPU bound 0
CPU bound 0
CPU bound 1
CPU bound 1
CPU bound 2
CPU bound 2
Finished in 4.6

To get perfromance for CPU-bound,
use process executor
CPU bound 0
CPU bound 0
CPU bound 1
CPU bound 1
CPU bound 2
CPU bound 2
Finished in 2.5

I/O bound will gain benefit from processes as well...
I/O bound 0
I/O bound 0
I/O bound 1
I/O bound 1
I/O bound 2
I/O bound 2
Finished in 3.3

... but there's no need in processes since you can use lighter threads for I/O
I/O bound 0
I/O bound 0
I/O bound 1
I/O bound 1
I/O bound 2
I/O bound 2
Finished in 3.1

Long story short,
Use processes for CPU bound due to GIL
and use threads for I/O bound since you benefit from concurrency regardless of GIL
CPU bound 0
I/O bound 0
CPU bound 1
I/O bound 1
CPU bound 2
I/O bound 2
Finished in 2.9

轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/391052.html

標籤：Python 多线程异步等待蟒蛇异步

上一篇：SwiftUI：初始值設定項中沒有標簽/字串但帶有ButtonStyle的按鈕

下一篇：返回列表

“GIL”如何影響帶有i/o系結任務的Pythonasyncio`run_in_executor`？