下面是一個程式，它發出多個獲取請求并將回應影像寫入我的目錄。這些獲取請求意味著在單獨的執行緒中，因此比無執行緒更快，但我沒有看到性能差異。
列印 active_count() 顯示創建了 9 個執行緒。但是，無論我是否使用執行緒，執行時間仍然需要大約 40 秒。
下面是我使用執行緒。
from threading import active_count
import requests
import time
import concurrent.futures
img_urls = [
'https://images.unsplash.com/photo-1516117172878-fd2c41f4a759',
'https://images.unsplash.com/photo-1532009324734-20a7a5813719',
'https://images.unsplash.com/photo-1524429656589-6633a470097c',
'https://images.unsplash.com/photo-1530224264768-7ff8c1789d79',
'https://images.unsplash.com/photo-1564135624576-c5c88640f235',
'https://images.unsplash.com/photo-1541698444083-023c97d3f4b6',
'https://images.unsplash.com/photo-1522364723953-452d3431c267',
'https://images.unsplash.com/photo-1513938709626-033611b8cc03',
'https://images.unsplash.com/photo-1507143550189-fed454f93097',
'https://images.unsplash.com/photo-1493976040374-85c8e12f0c0e',
'https://images.unsplash.com/photo-1504198453319-5ce911bafcde',
'https://images.unsplash.com/photo-1530122037265-a5f1f91d3b99',
'https://images.unsplash.com/photo-1516972810927-80185027ca84',
'https://images.unsplash.com/photo-1550439062-609e1531270e',
'https://images.unsplash.com/photo-1549692520-acc6669e2f0c'
]
t1 = time.perf_counter()
def download_image(img_url):
img_bytes = requests.get(img_url).content
img_name = img_url.split('/')[3]
img_name = f'{img_name}.jpg'
with open(img_name, 'wb') as img_file:
img_file.write(img_bytes)
print(f'{img_name} was downloaded...')
with concurrent.futures.ThreadPoolExecutor() as executor:
executor.map(download_image, img_urls)
print(active_count())
t2 = time.perf_counter()
print(f'Finished in {t2-t1} seconds')
下面是沒有執行緒的
def download_image(img_url):
img_bytes = requests.get(img_url).content
img_name = img_url.split('/')[3]
img_name = f'{img_name}.jpg'
with open(img_name, 'wb') as img_file:
img_file.write(img_bytes)
print(f'{img_name} was downloaded...')
for img_url in img_urls:
download_image(img_url)
有人可以解釋為什么會這樣嗎？謝謝
uj5u.com熱心網友回復：
這是我用你的代碼得到的結果，下載旁邊有開始和結束時間。總時間大致相同（在我的“正常網路”上，而不是我在評論中所說的慢速網路）
原因是多執行緒不會增加I/O或帶寬，限制也可能是網站本身。這看起來問題不在于您的代碼。
編輯（誤導性宣告）：正如Miyagi在下面的評論中提到的（閱讀他的評論，他解釋了原因），它應該增加 I/O，這就是我在慢速網路上增加 10 秒的原因（我的作業實驗室連接有限） . 在特定情況下，這不會增加 I/O 或帶寬（在我的“正常”連接上具有完整帶寬），這可能來自很多來源，但在我看來，不是代碼本身。
我也嘗試使用 max_workers=5，出現相同的總時間。
photo-1516117172878-fd2c41f4a759.jpg was downloaded... 1.0464828 - 1.7136098
photo-1532009324734-20a7a5813719.jpg was downloaded... 1.7140197 - 5.6327612
photo-1524429656589-6633a470097c.jpg was downloaded... 5.6339666 - 8.3146478
photo-1530224264768-7ff8c1789d79.jpg was downloaded... 8.3160157 - 10.474087
photo-1564135624576-c5c88640f235.jpg was downloaded... 10.4749598 - 11.2431941
photo-1541698444083-023c97d3f4b6.jpg was downloaded... 11.2436369 - 15.6939695
photo-1522364723953-452d3431c267.jpg was downloaded... 15.6954112 - 18.3257819
photo-1513938709626-033611b8cc03.jpg was downloaded... 18.3269668 - 21.0607191
photo-1507143550189-fed454f93097.jpg was downloaded... 21.0621265 - 22.2371699
photo-1493976040374-85c8e12f0c0e.jpg was downloaded... 22.2375931 - 26.4375676
photo-1504198453319-5ce911bafcde.jpg was downloaded... 26.4393404 - 28.3477933
photo-1530122037265-a5f1f91d3b99.jpg was downloaded... 28.348679 - 30.4626719
photo-1516972810927-80185027ca84.jpg was downloaded... 30.4636931 - 32.2621345
photo-1550439062-609e1531270e.jpg was downloaded... 32.2628976 - 34.7331719
photo-1549692520-acc6669e2f0c.jpg was downloaded... 34.7341393 - 35.5910094
Finished in 34.545366900000005 seconds
21
photo-1516117172878-fd2c41f4a759.jpg was downloaded... 35.5960486 - 46.1692758
photo-1564135624576-c5c88640f235.jpg was downloaded... 35.6110777 - 47.3780254
photo-1507143550189-fed454f93097.jpg was downloaded... 35.6265503 - 47.4433963
photo-1549692520-acc6669e2f0c.jpg was downloaded... 35.6692061 - 49.7097683
photo-1516972810927-80185027ca84.jpg was downloaded... 35.6420564 - 57.2326763
photo-1504198453319-5ce911bafcde.jpg was downloaded... 35.6340008 - 61.4597509
photo-1550439062-609e1531270e.jpg was downloaded... 35.6637577 - 62.0488296
photo-1530224264768-7ff8c1789d79.jpg was downloaded... 35.6072146 - 63.4139648
photo-1513938709626-033611b8cc03.jpg was downloaded... 35.6223106 - 63.8149815
photo-1524429656589-6633a470097c.jpg was downloaded... 35.6032493 - 63.8284464
photo-1530122037265-a5f1f91d3b99.jpg was downloaded... 35.6352735 - 65.0513042
photo-1522364723953-452d3431c267.jpg was downloaded... 35.6182243 - 65.5005548
photo-1532009324734-20a7a5813719.jpg was downloaded... 35.5994888 - 66.2930857
photo-1541698444083-023c97d3f4b6.jpg was downloaded... 35.6144996 - 67.8115219
photo-1493976040374-85c8e12f0c0e.jpg was downloaded... 35.6301133 - 68.5357319
Finished in 32.946069800000004 seconds
編輯 2（更多測驗）：我嘗試使用我的一個網路服務器（相同的代碼，只是不同的影像串列），我的下載時間總體減少了 60-70%。在這種情況下，與有限的工人一起作業最好。問題來自網站，而不是您的代碼。
uj5u.com熱心網友回復：
使用多處理包時，我可以看到一些性能改進。
import multiprocessing
from multiprocessing import Pool
def download_image(img_url: str) -> None:
img_bytes = requests.get(img_url).content
img_name = img_url.split('/')[3]
img_name = f'{img_name}.jpg'
with open(img_name, 'wb') as img_file:
img_file.write(img_bytes)
print(f'{img_name} was downloaded...')
if __name__ == '__main__':
t1 = time.perf_counter()
with Pool(processes=multiprocessing.cpu_count() - 1 or 1) as pool:
pool.map(download_image, img_urls)
t2 = time.perf_counter()
print(f'Finished in {t2 - t1} seconds')
轉載請註明出處，本文鏈接：https://www.uj5u.com/net/461344.html
標籤：Python 多线程 python-多线程