多處理查詢檔案中的新行

我正在嘗試使用多處理來加快時間。目標是；行程將查詢到文本檔案中定義的域。執行時；多行程只是做同樣的事情：每個行程從第一行查詢，而不是每個行程的新行。所以主要目標；new lines每個行程查詢from source中列出的域.txt。這是使用的代碼：

class diginfo:
    expected_response = 101
    control_domain = 'd2f99r5bkcyeqq.cloudfront.net'
    payloads = { "Host": control_domain, "Upgrade": "websocket", "DNT":  "1", "Accept-Language": "*", "Accept": "*/*", "Accept-Encoding": "*", "Connection": "keep-alive, upgrade", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36" }
    file_hosts = ""
    result_success = []
    num_file = 1
    columns = defaultdict(list)
    txtfiles= []
    hostpath = 'host'

def engines(counts, terminate, reach):
    for domain in domainlist:
        try:
            r = requests.get("http://"   domain, headers=headers, timeout=0.7, allow_redirects=False)
            if r.status_code == expected_response:
                print("Success"   domain)
                print(domain, file=open("RelateCFront.txt", "a"))
                result_success.append(str(domain))
            elif r.status_code != expected_response:
                print("Failed"   domain   str(r.status_code))

    print(" Loaded : "    str(len(diginfo.result_success)))
    if len(diginfo.result_success) >= 0:
        print(" Successfull Result : ")
    for result in diginfo.result_success:
        print("  "   result)
    print("")
    while not terminate.is_set():
        reach.set()
        break

def fromtext():
        global headers, domainlist
        files = os.listdir(diginfo.hostpath)
        for f in files:
            if fnmatch.fnmatch(f, '*.txt'):
                print( str(diginfo.num_file),str(f))
                num_file=diginfo.num_file 1
                diginfo.txtfiles.append(str(f))

        fileselector = input("Choose Target Files : ")
        print("Target Chosen : "   diginfo.txtfiles[int(fileselector)-1])
        file_hosts = str(diginfo.hostpath)  "/"  str(diginfo.txtfiles[int(fileselector)-1])

        with open(file_hosts) as f:
            parseddom = f.read().split()
            
        domainlist = list(set(parseddom))
        domainlist = list(filter(None, parseddom))

        terminate = Event()
        reach = Event()
        for counts in range(cpu_count()):
            p = Process(target=engines, args=(counts, terminate, reach))
            p.start()
        reach.wait()
        terminate.set()
        sleep(3)
        exit()

fromtext()

這是我所做的：

for domain in domainlist:
    p = Process(target=engines, args=(domainlist, terminate, reach))
    p.start()

它似乎不會回應并導致 0 結果和無限行程。我不能傳遞counts引數，因為它只接受 3 個引數。Terminate并Reach用于在達到要求后發出信號。

有問題的代碼

有問題的螢屏截圖

uj5u.com熱心網友回復：

您需要分成domainlist多個cpu_count()部分，并將每個部分傳遞給不同的行程。您還錯誤地使用了事件：目前，無論其他行程是否仍在作業，它都會在任何行程完成后 3 秒退出。您應該使用 aBarrier代替，或者只呼叫join()以下中的每個行程fromtext()：

def engines(domainsublist):
    for domain in domainsublist:
        ...

def fromtext():
    ...
    num_cpus = cpu_count()
    processes = []
    for process_num in range(num_cpus):
        section = domainlist[process_num::num_cpus]
        p = Process(target=engines, args=(section,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()

最后，你有一些競爭條件engines()：當你寫到 RelateCFront.txt 和當你追加到diginfo.result_success. SO上有很多很好的解決方案；我不會嘗試在這里修復它們。

標籤：Python python-3.x 多处理

上一篇：我的代碼通過驗證，但運行速度很慢

下一篇：如何僅從python函式中的串列理解中獲取值？