難以從網頁中抓取表格

我一直在嘗試使用 requests 模塊從該

您可以從 API 呼叫中以 JSON 物件的形式獲取所需的資料。復制 curl 命令并將其匯入 Postman 并使用腳本中的代碼。

在下面的螢屏截圖中，您可以看到您想要的實際資料。

編輯這將給出預期的結果。

import requests
import json

url = "https://cityofdavid.secure.force.com/apexremote"

payload = json.dumps([
  {
    "action": "OrderTourController",
    "method": "getAllTimeForTour",
    "data": [
      "01t0600000AE2dR",
      "2022-1-24 12:18:4",
      0,
      "en_us"
    ],
    "type": "rpc",
    "tid": 3,
    "ctx": {
      "csrf": "VmpFPSxNakF5TWkwd01TMHlOMVF3TmpvME9Eb3dNeTQ1TlRaYSxJZExXMldWSU1neGFlWDdFb01YNi1ULE1XTmpaV1F4",
      "vid": "06624000003gGO2",
      "ns": "",
      "ver": 35
    }
  }
])
headers = {
  'Content-Type': 'application/json',
  'Accept': '*/*',
  'Referer': 'https://cityofdavid.secure.force.com/?productId=01t0600000AE2dR&lang=en_us&_ga=2.113284295.1937905009.1643006618-1494711041.1643006618',
}

response = requests.request("POST", url, headers=headers, data=payload)




json_result=json.loads(json.loads(response.text)[0]['result'])
for res in json_result:
    print(res['TourStartTime__c'], res['TourStartHour__c'] ,res['EndTime__c'],res['MaxNumerOfParticipants__c'])

輸出

2022-01-25T08:00:00.000 0000 10:00 11:00 15
2022-01-25T10:00:00.000 0000 12:00 13:00 15
2022-01-25T12:00:00.000 0000 14:00 15:00 15
2022-01-25T14:00:00.000 0000 16:00 17:00 15
2022-01-26T08:00:00.000 0000 10:00 11:00 15
2022-01-26T10:00:00.000 0000 12:00 13:00 15
2022-01-26T12:00:00.000 0000 14:00 15:00 15
2022-01-26T14:00:00.000 0000 16:00 17:00 15
2022-01-27T08:00:00.000 0000 10:00 11:00 15
2022-01-27T10:00:00.000 0000 12:00 13:00 15
2022-01-27T12:00:00.000 0000 14:00 15:00 15
2022-01-27T14:00:00.000 0000 16:00 17:00 15
2022-01-30T08:00:00.000 0000 10:00 11:00 15
2022-01-30T10:00:00.000 0000 12:00 13:00 15
2022-01-30T12:00:00.000 0000 14:00 15:00 15
2022-01-30T14:00:00.000 0000 16:00 17:00 15
2022-01-31T08:00:00.000 0000 10:00 11:00 15
2022-01-31T10:00:00.000 0000 12:00 13:00 15
2022-01-31T12:00:00.000 0000 14:00 15:00 15
2022-01-31T14:00:00.000 0000 16:00 17:00 15

uj5u.com熱心網友回復：

您正在獲取亂碼資料，因為內容是由 Javascript 生成的，要處理這個問題，您需要 selenium 或任何類似的東西來獲取 Javascript 資料，而不是請求，您可以使用 selenium 來獲取頁面源，然后像以前一樣繼續使用湯

import time

from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.microsoft import EdgeChromiumDriverManager

driver = webdriver.Edge(EdgeChromiumDriverManager().install())

driver.get('https://cityofdavid.secure.force.com/?productId=01t0600000AE2dR&lang=en_us')
time.sleep(5)
htmlSource = driver.page_source
soup = BeautifulSoup(htmlSource, "lxml")
for item in soup.select("table.table > tbody > tr"):
    data = [i.text for i in item.select("td")]
    print(data)

將回傳您可以根據需要使用的以下資料

['24/01/2022', '10:00', '11:00', '15', '\nSelect\n\n']
['24/01/2022', '12:00', '13:00', '15', '\nSelect\n\n']
['24/01/2022', '14:00', '15:00', '15', '\nSelect\n\n']
['24/01/2022', '16:00', '17:00', '15', '\nSelect\n\n']
['25/01/2022', '10:00', '11:00', '11', '\nSelect\n\n']
['25/01/2022', '12:00', '13:00', '15', '\nSelect\n\n']
['25/01/2022', '14:00', '15:00', '15', '\nSelect\n\n']
['25/01/2022', '16:00', '17:00', '15', '\nSelect\n\n']
['26/01/2022', '10:00', '11:00', '14', '\nSelect\n\n']
['26/01/2022', '12:00', '13:00', '15', '\nSelect\n\n']
['26/01/2022', '14:00', '15:00', '15', '\nSelect\n\n']
['26/01/2022', '16:00', '17:00', '15', '\nSelect\n\n']
['27/01/2022', '10:00', '11:00', '12', '\nSelect\n\n']
['27/01/2022', '12:00', '13:00', '11', '\nSelect\n\n']
['27/01/2022', '14:00', '15:00', '15', '\nSelect\n\n']
['27/01/2022', '16:00', '17:00', '15', '\nSelect\n\n']
['30/01/2022', '10:00', '11:00', '15', '\nSelect\n\n']
['30/01/2022', '12:00', '13:00', '15', '\nSelect\n\n']
['30/01/2022', '14:00', '15:00', '15', '\nSelect\n\n']
['30/01/2022', '16:00', '17:00', '15', '\nSelect\n\n']

標籤：

上一篇：根據熊貓中另一個df中的匹配元素替換資料框中的celle

下一篇：如何使用python從公司名稱中獲取公司官網網址？