我試圖通過在引數串列中呼叫一些函式來處理大量的作業負載：
import multiprocessing as mp
print("Number of processors: ", mp.cpu_count())
pool = mp.Pool(mp.cpu_count())
try:
results = pool.map_async(consume_one, [list-of-arguments]).get()
except:
print(e)
finally:
pool.close()
對于 的每次呼叫consume_one()，我們從 '[list-of-arguments]' 傳遞一個值，并在這個函式中記錄函式的開始和結束時間consume_one()。
觀察值如下：
Completed processing for ... in 0:03:34.283025
Completed processing for ... in 0:04:24.109049
Completed processing for ... in 0:04:58.464374
Completed processing for ... in 0:05:11.830404
Completed processing for ... in 0:08:32.234539
Completed processing for ... in 0:09:09.725937
Completed processing for ... in 0:09:10.968685
Completed processing for ... in 0:09:51.642501
Completed processing for ... in 0:10:58.076675
Completed processing for ... in 0:12:30.905190
Completed processing for ... in 0:14:01.051716
正如我們在日志中看到的那樣，對同一函式的所有后續呼叫都花費了越來越長的時間，但這并不是因為這些呼叫的引數，而且所有呼叫的引數都不太相同。
我的問題是：
為什么會發生這種情況？
我該如何除錯呢？
uj5u.com熱心網友回復：
Q1：
“為什么會發生這種情況？”
A1 :
累積處理效率低下的增長
Q2：
“我該如何除錯？”
A2 :
最好了解你失去大部分處理效率的組成部分。首先原則上，接下來在實際技術中使用，如何完全減少或更好地消除這種昂貴的開銷熱點。
首先找出處理效率低下的個別根本原因：
Python解釋器行程產生附加開銷成本的成本（在某些O / S上，我重復一個完整的副本 - 即可能是
[GB]你的硬體上的一些移動RAM到RAM的物理記憶體I / O瓶頸- 檢查(a) CPU-主板-DRAM 在從/到物理 RAM 的途中有多少個 I/O 通道，以及(b)是否有這么多復制的行程記憶體分配實際上沒有打開 O/S虛擬記憶體管理進入交換模式，即因為（行程次復制）的范圍- 記憶體不適合您的計算平臺的物理 RAM 占用空間，O/S 虛擬記憶體管理器開始“模擬”這種缺失的 RAM，以“回應式”按需滿足記憶體訪問需求（參見 LRU-policy 等) 通過將大塊行程保留的物理 RAM 資料交換出來（一旦 LRU 策略這樣說），到磁盤上（這里可以看到~ 1E5更大的資料訪問延遲，還可以想象交通堵塞帶來的另一個附加成本，如果不是直接阻塞, 任何其他物理 RAM 到 CPU 資料流, 在將大量交換的[GB]-s 塊移入/移出如此緩慢的交換空間磁盤存盤期間. 然后將此成本乘以~ 2X，作為移動另一個資料塊的成本，從緩慢的交換空間磁盤存盤回到現在“釋放的”-RAM，以供下一個允許依次使用它的行程（至少對于一些相當數量的-時間，因為 O/S 虛擬記憶體管理器認為它是公平的），基本上與先前將 RAM 資料塊移出的慢動作成本相同（以便為該資料塊留出空間移入） . 一種“SOKOBAN”問題，當移動所有數以萬計的[GB]-s 但僅通過一對 CPU 到物理 RAM 記憶體 I/O 通道時，因此經常等待1E5任何下一個檔案系統的速度慢一倍的磁盤存盤-block 到 { 讀取 | 在漫長的等待佇列中寫} - 它是如此殘酷，因為我們在這里談論由此產生的端到端計算策略效率，或高效計算科學中的交換雜耍性能反模式）Python Interpreter 行程到行程資料傳輸的成本（SER/xfer/DES 發生在傳遞的呼叫簽名引數和回傳的結果中） -這是一個示例，這種性能反模式可能主要是避免，節省不必要的 SER/xfer/DES 資料流的所有附加成本
~ 75 [TB]（！TB 的物理 RAM 記憶體 -I/O、CPU-pickle.dumps()、O/S-pipe-xfers、CPU- pickle.loads() 和另一個物理 RAM 記憶體 I/O——所有這些都浪費在~ 75 [TB]資料流上，只是由于錯誤使用了基于“外部”迭代器的“slim”-SLOC 串列理解而毫無意義地注入，等等在.map_async()上面使用的 - 方法中操作的一個“隱藏” ）代碼效率低下的成本 不重用快取行中昂貴的預取資料
costs of hardware thermal throttling the "marketing"-promoted CPU frequencies, once the cores start to do some heavy work and get warm ( throttling can on some hardware be deferred, if a job can be shifted onto another, cooler, CPU-core - yet at a cost of losing the core-local cached data, so the code has to re-fetch it again at the most expensive price in TimeDOMAIN, just as it was moved onto a cooler, but working at a higher frequency, CPU-core ). In the situation here, your code has spawned immense amount of processes ( not mentioning the O/S-process-scheduler add-on costs now ), there are no cooler CPU-cores left and your hardware resorts to thermal throttling - i.e. working on lower and lower CPU-core frequency
costs of explicitly uncontrolled garbage collections are a chapter of its own importance
For detailed argumentation on pros and cons & overhead testing templates you might like this collection of multiprocessing.Pool() & joblib.Parallel() related
If you are interested not only in debugging the problem, but also in solving the end-to-end process-performance, try to re-factor the code so as to principally prevent this from happening - if you are sure you want that many processes (while these are most probably memory-starved, waiting for not yet fetched data from slow & overloaded/blocked physical-RAM-I/O channels), be sure to move from a __main__-Python Interpreter hosted "outer"-item-iterator to a block-based work. There you command the worker processes to iterate "internally" (avoiding all the awfully many repetitions of SER/xfer/DES add-on costs) over their commanded, disjunct block-of-list, partitioned by the __main__-Python Interpreter. As the Python Interpreter process-instantiation works in the known manner (copying a state-full copy of the __main__-Python Interpreter - previous attempts to reduce the amount of data copied by O/S-provided "forking" have been documented to cause problems or can even lead to deadlocking situations, so a due care is to be paid here, if the code has to be robust enough, if it is going into production or providing some kind of critical intelligence), the list, per-se will be already present in the worker processes, that can iterate over their "private"-part (one of the disjunct blocks of this list) just by commanded index-ranges, i.e. without expensively passing any other parameter to the .Pool()-processes, but the respective index-range ~ as trivial as ( startHere, stopHere )-tuples, that map / cover the whole list. The costs of returning results depend on what these consist of. Tools exist for doing this efficiently, be it in block-transfer between processes on list-block completed, or in compressed file-I/O storage. Details matter, as always, if performance is the goal. As the items take ~ 3 minutes in the as-is state of the consume_one() processing, there is plenty of opportunities to speed this up, in the name of the overall processing efficiency and performance.
Block-based "internal"-iteration can keep calling the as-is consume_one(), if that could not get faster by some performance improving tools - be it numpy-vectorised (using internally high performance multicore, cache re-use efficient bare-metal optimised libraries) or numba-JIT-compiled accelerated processing (as the re-use count of the JIT/LLVM-compiled code here works in your direction, towards improving the overall performance )
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/450256.html
標籤：python-3.x 多线程 并行处理 多处理 工作库