無法在Aspx網站中抓取動態加載的HTML表

我正在嘗試從亞利桑那醫學委員會獲取一些資料。我在專業下拉串列中搜索 Anesthesiology，發現表格（帶有指向我要抓取的組態檔的鏈接）被動態加載到網站中。我注意到點擊“專業搜索”按鈕時，會向服務器發出 POST 請求，并且實際上從服務器回傳了 html 表。我已經嘗試模擬這個 post 請求，看看我是否收到了這個 html 表，然后嘗試用 bs4 決議它。這可能嗎？如果是這樣，我是否走在正確的軌道上？

我試圖將我在開發人員工具的網路選項卡中找到的表單資料包含在內，但我不確定這是否是正確的資料，或者我是否在這里或標題中忘記了一些資料。

如果我需要澄清，請告訴我，我理解這可能不是最好的措辭。謝謝！

import requests
# import re
import formdata

session = requests.Session()

url = "https://azbomprod.azmd.gov/GLSuiteWeb/Clients/AZBOM/public/WebVerificationSearch.aspx?q=azmd&t=20220622123512"

headers = {'User-Agent': 'My-Agent-Placeholder'}

res = session.get(url, headers=headers)

print("Response: {}".format(res))

payload = {
  "__VIEWSTATE": formdata.state,
  "__VIEWSTATEGENERATOR": formdata.generator,
  "__EVENTVALIDATION" : formdata.validation,
  "ctl00$ContentPlaceHolder1$Name": 'rbName1',
  "ctl00$ContentPlaceHolder1$Name": "rbName1",
  "ctl00$ContentPlaceHolder1$txtLastName" : '', 
  "ctl00$ContentPlaceHolder1$txtFirstName" : '',
  "ctl00$ContentPlaceHolder1$License": "rbLicense1",
  "ctl00$ContentPlaceHolder1$txtLicNum": '',
  "ctl00$ContentPlaceHolder1$Specialty": "rbSpecialty1",
  "ctl00$ContentPlaceHolder1$ddlSpecialty": '12155',
  "ctl00$ContentPlaceHolder1$ddlCounty": '15910',
  "ctl00$ContentPlaceHolder1$txtCity": '',
  "__EVENTTARGET": "ctl00$ContentPlaceHolder1$btnSpecial",
  "__EVENTARGUMENT": ''  
}

# params = {"q": "azmd",
# "t": "20220622123512"}

# #url = "https://azbomprod.azmd.gov/GLSuiteWeb/Clients/AZBOM/Public/Results.aspx"

res = session.post(url, data=payload, headers=headers)
print("Post response: {}".format(res))
print(res.text)

# res = requests.get('https://azbomprod.azmd.gov/GLSuiteWeb/Clients/AZBOM/Public/Results.aspx', headers=headers)

uj5u.com熱心網友回復：

嘗試：

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0"
}

url = "https://azbomprod.azmd.gov/GLSuiteWeb/Clients/AZBOM/public/WebVerificationSearch.aspx?q=azmd&t=20220622082816"

with requests.session() as s:
    soup = BeautifulSoup(s.get(url, headers=headers).content, "html.parser")

    data = {}
    for inp in soup.select("input"):
        data[inp.get("name")] = inp.get("value", "")

    data["ctl00$ContentPlaceHolder1$Name"] = "rbName1"
    data["ctl00$ContentPlaceHolder1$License"] = "rbLicense1"
    data["ctl00$ContentPlaceHolder1$Specialty"] = "rbSpecialty1"
    data["ctl00$ContentPlaceHolder1$ddlSpecialty"] = "12155"
    data["ctl00$ContentPlaceHolder1$ddlCounty"] = "15910"

    data["__EVENTTARGET"] = "ctl00$ContentPlaceHolder1$btnSpecial"
    data["__EVENTARGUMENT"] = ""

    soup = BeautifulSoup(
        s.post(url, data=data, headers=headers).content, "html.parser"
    )
    for row in soup.select("tr:has(a)"):
        name = row.select("td")[-1].text
        link = row.a["href"]

        print("{:<35} {}".format(name, link))

印刷：

Abad-Pelsang, Elma A.               https://azbomprod.azmd.gov/glsuiteweb/clients/azbom/Public/Profile.aspx?entID=1620623&licID=121089&licType=1
Abadi, Bilal Ibrahim                https://azbomprod.azmd.gov/glsuiteweb/clients/azbom/Public/Profile.aspx?entID=1755530&licID=525771&licType=1
Abbasian, Mohammad                  https://azbomprod.azmd.gov/glsuiteweb/clients/azbom/Public/Profile.aspx?entID=1635449&licID=492537&licType=1
Abdel-Al, Naglaa Z.                 https://azbomprod.azmd.gov/glsuiteweb/clients/azbom/Public/Profile.aspx?entID=1637612&licID=175204&licType=1
Abedi, Babak                        https://azbomprod.azmd.gov/glsuiteweb/clients/azbom/Public/Profile.aspx?entID=1641219&licID=169009&licType=1
Abel, Martin D.                     https://azbomprod.azmd.gov/glsuiteweb/clients/azbom/Public/Profile.aspx?entID=1624271&licID=510929&licType=1
Abenstein, John P.                  https://azbomprod.azmd.gov/glsuiteweb/clients/azbom/Public/Profile.aspx?entID=1622930&licID=502482&licType=1

...and so on.

標籤：Python 网 硒 网页抓取 美丽的汤

上一篇：BlazorWebAssembly向所有API請求添加標頭

下一篇：從用戶編輯的ASP.NET表單創建PDF？