如何將串列拆分為多個批次并將每個批次分配給執行緒作業者

我想將一個更大的串列（2000 個專案）分成多個批次，每個批次將在不同的執行緒上并行運行。

問題的文本是這樣的：

我們需要計算包含 2000 多個專案的串列的平方和。例如：如果串列是 [1,2,3,4,5] -> 結果將是 55

確保實施每種方法并為其計時： 15.1。嘗試使用單執行緒方法計算從 0 到 2000 的專案串列的平方和。 15.2。嘗試使用多執行緒方法，將專案串列拆分為需要并行計算的批次專案

這是我到現在為止的想法：

def divide_list(l, n):

    # looping till length l
    for i in range(0, len(l), n):
        yield l[i : i   n]


def task():
    # didn't know what to write here


if __name__ == "__main__":

    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

    # Single thread

    t1_start = perf_counter()

    squares_single = [x**2 for x in range(2001)]
    print(sum(squares_single))

    t1_stop = perf_counter()

    print("Elapsed time using a single thread in seconds:", t1_stop - t1_start)

    # Multithreading

    t2_start = perf_counter()

    squares_threading = [x**2 for x in range(2001)]
    # print("Square threading: ", squares_threading)
    x = list(divide_list(squares_threading, 200))

    sum = 0
    n_threads = len(x)
    with ThreadPoolExecutor(n_threads) as executor:
        for list in x:
            for result in executor.map(task, list):
                sum  = [x**2 for x in list]
        print(sum)
    t2_stop = perf_counter()

    print("Elapsed time using multithreading in seconds:", t2_stop - t2_start)

uj5u.com熱心網友回復：

我是 ThreadPoolExecutor 的新手。在閱讀了https://docs.python.org/3/library/concurrent.futures.html之后，我取出了使用 submit() 方法（而不是映射）的 ThreadPoolExecutor 示例。我的結果如下。我已經洗掉了您的一些代碼（如 perf_counter），以使我的代碼以更簡單的方式作業。

import sys
import concurrent.futures
import logging

def sum_squares( *args, **kwargs ):
  sum = 0 
  try:
    list,start,end = args
  except Exception as e:
    print(e,args)
    raise e

  for x in list[start:end]:
    sum  = x**2 

  return sum

def divide_list(l, n):

    # looping till length l
    for i in range(0, len(l), n):
        yield l[i : i   n]


def task():
    # didn't know what to write here
    pass

if __name__ == "__main__":
  format = "%(asctime)s: %(message)s"
  logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

  # Single thread
  l = list(range(2001))

  squares_single = [x**2 for x in l]
  print(sum(squares_single))
  print( sum_squares( l, 0, len(l)))  # let's make sure the sum_squares function matches your results
  # Multithreading

  sum = 0
  n_threads = 10
  futures = []
  
  with concurrent.futures.ThreadPoolExecutor(n_threads) as executor:
      sum = 0 
      for start in range(0, len(l), n_threads):
          futures.append(executor.submit( sum_squares,  l, start, start   n_threads  ) )

      for future in concurrent.futures.as_completed( futures ):
        sum  = future.result() 

      print(sum)

我有三個列印在那里，結果是：

2668667000
2668667000
2668667000

標籤：Python多线程线程池python-多线程

上一篇：如何使用Boto3從S3存盤桶中獲取檔案名串列

下一篇：在腳本中實作多執行緒/并行處理