我正在使用多執行緒來運行消耗記憶體的行程。為了避免記憶體交換，我想在記憶體超過一定數量時殺死所有多執行緒行程。
Python 是 3.9，我使用的是 Jupyter 筆記本。
以下是當記憶體使用量超過 50Gb 時終止行程的示例代碼。
from multiprocessing import Pool, cpu_count
def my_func():
a = MemoryConsumingObject()
memory_usage_giga = os.popen('free -g').readlines()[1].split()[1:][1]
if memory_usage_giga > 50:
# I want quit multi-threading process! How do I quit?
pass
return a
my_list = list(np.arange(1,1000))
with Pool(cpu_count()) as p:
result = p.imap(my_func, my_list)
uj5u.com熱心網友回復：
您可以使用 anEvent在作業人員之間進行通信來做到這一點。您無法輕松終止池（在這種情況下，作業人員無法直接與主行程通信），但您可以阻止作業人員在滿足某些條件時進行記憶體分配。在下面的腳本中，我通過生成一個亂數來模擬何時停止作業。一旦滿足停止條件，作業人員仍會被呼叫來處理要完成的作業，但如果滿足停止條件，他們會停止而不實際執行任何操作。
from os import getpid
from multiprocessing import Pool, cpu_count, Event
from random import random
event = Event()
def my_func(arg):
pid = getpid()
if event.is_set():
print('%s received %s, but aborting as stop condition was seen' % (pid, arg))
return None
r = random()
if r < 0.3:
print('%s received %s, %.2f stop condition met, aborting and letting everyone else know' % (pid, arg, r))
event.set()
return
print("%s received: %s, %.2f stop condition not met, do work" % (getpid(), arg, r))
# do the actual work here
return r
my_list = list(range(10))
def main():
with Pool(cpu_count()) as p:
result = p.map(my_func, my_list)
p.close()
p.join()
print(list(result))
if __name__ == '__main__':
main()
一個示例運行列印：
18786 received: 0, 0.67 stop condition not met, do work
18786 received: 1, 0.38 stop condition not met, do work
18786 received: 2, 0.51 stop condition not met, do work
18786 received: 3, 0.34 stop condition not met, do work
18786 received: 4, 0.72 stop condition not met, do work
18786 received: 5, 0.82 stop condition not met, do work
18786 received 6, 0.00 stop condition met, aborting and letting everyone else know
18786 received 7, but aborting as stop condition was seen
18786 received 8, but aborting as stop condition was seen
18786 received 9, but aborting as stop condition was seen
[0.6733142995176965, 0.3830673860039788, 0.5053762644489409, 0.3437097578584267, 0.7211013474170365, 0.816546904830295, None, None, None, None]
請注意，行程可能并不總是按照您預期的順序運行 - 畢竟它是并發處理。在我的情況下，CPU 計數只有 2，因此一個作業行程正在獲取所有內容。
轉載請註明出處，本文鏈接：https://www.uj5u.com/gongcheng/354214.html