該程式的想法是檢查subdomains.txt檔案中存在的域/子域（通過 http/https 協議）。
我通過使用HEAD requests域/子域并接收回應狀態代碼來做到這一點。如果狀態代碼可用，則域或子域處于活動狀態。（load_url_http功能）
為了加快程式的速度，我使用了并發數為 200 的 concurrent.futures.ThreadPoolExecutor 但是，即使將執行緒數增加到 300 后，程式的速度也沒有太大的提高。
我希望對我的程式進行改進，以便能夠一次發送數千個請求。以下是我的部分源代碼：
python-request-multil.py
import time
import requests
import concurrent.futures
def load_url_http(protocol: str, domain: str, timeout: int = 10):
try:
conn = requests.head(protocol "://" domain, timeout=timeout)
return conn.status_code
except Exception:
return None
#--- main ---#
start_time = time.time()
worker = 400
protocol = "http"
timeout = 10
print("Number of worker:", worker)
with concurrent.futures.ThreadPoolExecutor(max_workers=worker) as executor:
# The file object that the subdomain lives on will be written to
file_live_subdomain = open("live_subdomains.txt", "a")
# load domain/subdomain list from file
URLS = open("subdomains.txt", "r").read().split("\n")
URLS_length = len(URLS)
# Count the number of live subdomains
live_count = 0
# Start the load operations and mark each future with its URL
future_to_url = {
executor.submit(load_url_http, protocol, url, timeout): url for url in URLS
}
for i, future in zip(range(URLS_length), concurrent.futures.as_completed(future_to_url)):
url = future_to_url[future]
print(f"\r--> Checking live subdomain.........{i 1}/{URLS_length}", end="")
try:
data = future.result()
# If `load_url_http` returns any status code
if data != None:
# print(f'{protocol}://{url}:{data}')
live_count = live_count 1
file_live_subdomain.write(f"\n{protocol}://" url)
except Exception as exc:
print(exc)
print(f"\n[ ] Live domain: {live_count}/{URLS_length}", end="")
file_live_subdomain.close()
print("\n--- %s seconds ---" % (time.time() - start_time))
跑：
┌──(quangtb?QuangTB)-[/mnt/e/DATA/Downloads]
└─$ python3 python-request-multil.py
Number of worker: 100
--> Checking live subdomain.........1117/1117
[ ] Live domain: 344/1117
--- 67.41670227050781 seconds ---
┌──(quangtb?QuangTB)-[/mnt/e/DATA/Downloads]
└─$ python3 python-request-multil.py
Number of worker: 200
--> Checking live subdomain.........1117/1117
[ ] Live domain: 344/1117
--- 54.6825795173645 seconds ---
┌──(quangtb?QuangTB)-[/mnt/e/DATA/Downloads]
└─$ python3 python-request-multil.py
Number of worker: 300
--> Checking live subdomain.........1117/1117
[ ] Live domain: 339/1117
--- 54.186068058013916 seconds ---
┌──(quangtb?QuangTB)-[/mnt/e/DATA/Downloads]
└─$ python3 python-request-multil.py
Number of worker: 400
--> Checking live subdomain.........1117/1117
[ ] Live domain: 344/1117
--- 54.19181728363037 seconds ---
uj5u.com熱心網友回復：
在 pythonmultithreading中實際上并不并行運行，所有執行緒都在 1 個行程下運行，并且該行程僅在 1 個核心（cpu）上運行。
您可以根據threads需要創建任意數量，但不會解決問題，實際上會使情況變得更糟，因為這 300 個threads在 1 上運行，cpu core但 1cpu core一次只能運行 1command個，所以最終發生的事情是cpu core需要commands在 1 上運行一些thread，然后switch在另一個上運行一些，然后在另一個上運行一些，然后在另一個上thread運行，等等......之間的操作需要資源，并且在那個時候你的程式代碼不會運行。所以最后，如果你向許多人開放，你會花更多的時間在兩者之間而不是執行你的程式。commandsthreadswitchswitchingthreadsthreadscpu coreswitchingthreads
你可以做的是實際同時運行你的代碼是打開幾個processes而不是threads使用multiprocessing庫，然后你的代碼將運行在多個上cores，同樣的事情，不要打開數百個processes，只打開幾個，我建議process為每個core您cpu擁有的開口 1，該multiprocessing庫具有一個內置函式，可回傳cores您cpu擁有的數量：
import multiprocessing
print(multiprocessing.cpu_count())
請注意，由于processes實際同時運行您的代碼，您prints可能會相互干擾，因此您需要使用multiprocessing.Lock()并執行以下操作：
import multiprocessing
lock = multiprocessing.Lock()
lock.acquire()
print("something")
lock.release()
lock.acquire()在每個之前print和之后執行lock.release()（如果您不釋放鎖定，您的程式將被卡住），這將確保您prints不會互相干擾。
編輯：
在您的情況下，因為timeout實際上最好打開一些processes，并且每個都process像 20 threads。
因為如果process為每個失敗的地址等待 10 秒，它最終會比打開很多threads
所以對于你的情況，fastest我能想到的方式是打開幾個processes，每個process打開 20 到 30 個threads
你可以嘗試這樣的事情：
import multiprocessing
import random
import threading
import time
import requests
MAX_NUMBER_OF_PROCESSES = multiprocessing.cpu_count()
MAX_NUMBER_OF_THREADS_IN_EACH_PROCESS = 30
PROTOCOL = "http"
TIMEOUT = 4
START_TIME = time.time()
def load_url_http(protocol: str, domains: list[str], timeout: int = 10):
with open("live_subdomains.txt", "a") as live_domains_file:
for domain in domains:
try:
conn = requests.head(protocol "://" domain, timeout=timeout)
if conn.status_code is not None:
live_domains_file.write(f"{protocol}://{domain}\n")
except Exception:
pass
return
def create_threads_for_process(protocol: str, domains: list[str], timeout: int = 10):
# create threads
threads_list = []
threads_urls = {}
start = 0
number_of_threads_to_open = len(domains) if len(domains) < MAX_NUMBER_OF_THREADS_IN_EACH_PROCESS \
else MAX_NUMBER_OF_THREADS_IN_EACH_PROCESS
for i in range(1, number_of_threads_to_open 1):
# distribute the work of this process evenly between all the threads
if i != number_of_threads_to_open:
threads_urls[i] = domains[start:(len(domains) // number_of_threads_to_open) * i]
start = (len(domains) // number_of_threads_to_open) * i
else:
threads_urls[i] = domains[start:]
# create and start the thread
thread = threading.Thread(target=load_url_http,
args=(protocol, threads_urls[i], timeout,),
daemon=True)
thread.start()
threads_list.append(thread)
# wait for all threads to finish
while threads_list:
for thread in threads_list:
if not thread.is_alive():
threads_list.remove(thread)
time.sleep(0.8)
def main():
with open("live_subdomains.txt", "w") as file:
file.write("")
with open("subdomains.txt", "r") as file:
urls = file.read().split("\n")
random.shuffle(urls) # shuffle the urls list
# create the processes
processes_list = []
processes_urls = {}
start = 0
number_of_processes_to_open = len(urls) if len(urls) < MAX_NUMBER_OF_PROCESSES else MAX_NUMBER_OF_PROCESSES
for i in range(1, number_of_processes_to_open 1):
if i != number_of_processes_to_open:
# give each process an even amount of work
processes_urls[i] = urls[start:(len(urls) // number_of_processes_to_open) * i]
start = (len(urls) // number_of_processes_to_open) * i
else:
# the last process will get a bit more / a bit less in
# case len(urls) isn't dividable by number_of_processes_to_open
processes_urls[i] = urls[start:]
# create the process, start it and add to processes list
process = multiprocessing.Process(target=create_threads_for_process,
args=(PROTOCOL, processes_urls[i], TIMEOUT,),
daemon=True)
process.start()
processes_list.append(process)
# wait for all processes to finish
while multiprocessing.active_children():
time.sleep(0.8)
# print result
with open("live_subdomains.txt", "r") as live_urls_file:
live_count = len(live_urls_file.read().split("\n")) - 1 # -1 empty line at the end
print(f"\n[ ] Live domain: {live_count}/{len(urls)}", end="")
print("\n--- %s seconds ---" % (time.time() - START_TIME))
if __name__ == '__main__':
main()
在這段代碼中，我process為每個has 打開 1 ，并且在每個core我打開 30中，你也不需要等待 10 秒，比如 5 秒就足夠了，尤其是當你使用而不是.cpuprocessthreadsreplayHEADGET
我運行你的代碼，與我的代碼相比，你的代碼需要35 秒才能完成，我的需要25 秒，這是在 1117 個 url 上，url 串列越大，它就越重要。
轉載請註明出處，本文鏈接：https://www.uj5u.com/qiye/520367.html