并非所有容器都使用美湯裝載

我正在嘗試轉儲一個網站（網站鏈接在下面的代碼中給出）并且所有容器都沒有加載。就我而言，價格容器不是傾銷。有關更多詳細資訊，請參閱螢屏截圖。如何解決這個問題？

在這種情況下，類“I6yQz”內的容器未加載。

我的代碼：

url = "https://gomechanic.in/gurgaon/car-battery-replacement/maruti-suzuki-versa/petrol"
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
print(soup.prettify())

我需要螢屏截圖中顯示的以下內容

像這樣的事情：

data = {'CityName'：'Gurgaon'，'CarName'：'Versa-Petrol'，'serviceName'：'Excide（55 個月保修）'，'Price'：'4299'，'ServicesOffered'：'['免費接送”、“免費安裝”、“包括舊電池價格”、“可上門購買”]}

I have also got the API which is have all the information:

你需要做的是弄清楚如何在你的 python 代碼中復制這個請求：

import requests

headers = {
    # this website sues authroization for all requests
    'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJiNGJjM2NhZjVkMWVhOTlkYzk2YjQzM2NjYzQzMDI0ZTAyM2I0MGM2YjQ5ZjExN2JjMDk5OGY2MWU3ZDI1ZjM2MTU1YWU5ZDIxNjE2ZTc5NSIsInNjb3BlcyI6W10sInN1YiI6IjE2MzM5MzQwNjY5NCIsImV4cCI6MTYzNjUyNjA2Ny4wLCJhdWQiOiIzIiwibmJmIjoxNjMzOTM0MDY3LjAsImlhdCI6MTYzMzkzNDA2Ny4wfQ.QQI_iFpNgONAIp4bfoUbGDtnnYiiViEVsPQEK3ouYLjeyhMkEKyRclazuJ9i-ExQyqukFuqiAn4dw7drGUhRykJY6U67iSnbni0aXzzF9ZTEZrvMmqItHXjrdrxzYCqoKJAf2CYY-4hkO-NXIrTHZEnk-N_jhv30LHuK9A5I1qK8pajt4XIkC7grAn3gaMe3c6rX6Ko-AMZ801TVdACD4qIHb4o73a3vodEMvh4wjIcxRGUBGq4HBgAKxKLCcWaNz-z7XjvYrWhNJNB_iRjZ1YBN97Xk4CWxC0B4sSgA2dVsBWaKGW4ck8wvrHQyFRfFpPHux-6sCMqCC-e4okOhku3AasqPKwvUuJK4oov9tav4YsjfFevKkdsCZ1KmTehtvadoUXAHQcij0UqgMtzNPO-wKYoXwLc8yZGi_mfamAIX0izFOlFiuL26X8XUMP5HkuypUqDa3MLg91f-8oTMWfUjVYYsnjw7lwxKSl7KRKWWhuHwL6iDUjfB23qjEuq2h9JBVkoG71XpA9SrJbunWARYpQ48mc0LlYCXCbGkYIh9pOZba7JGMh7E15YyRla8qhU9pEkgWVYjzgYJaNkhrSNBaIdY56i_qlnTBpC00sqOnHRNVpYMb4gF3PPKalUMMJjbSqzEE2BNTFO5dGxGcz2cKP0smoVi_SK3XcKgPXc',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.15.2 Chrome/87.0.4280.144 Safari/537.36',
}

url = 'https://gomechanic.in/api/v1/priceList?city=gurgaon&brand=maruti-suzuki&service=car-battery-replacement'
response = requests.get(url, headers=headers)
print(response.json())

這將導致：

{
  "success": true,
  "data": [
    {
      "id": 1,
      "name": "800 Petrol",
      "price": 3400,
      "savings": "25%"
    },
    {
      "id": 2,
      "name": "800 CNG",
      "price": 3400,
      "savings": "25%"
    },
    {
      "id": 3,
      "name": "Alto Petrol",
      "price": 3400,
      "savings": "25%"
    },
    {
      "id": 4,
      "name": "Alto CNG",
      "price": 3400,
      "savings": "25%"
    },
    {
      "id": 5,
      "name": "Alto 800 Petrol",
      "price": 3400,
      "savings": "25%"
    },
    {
      "id": 6,
      "name": "Alto 800 CNG",
      "price": 3400,
      "savings": "25%"
    }
  ]
}

整個程序稱為逆向工程，更深入的介紹可以在這里查看我的教程博客：https : //scrapecrow.com/reverse-engineering-intro.html

As for parameters that are used in these backend API requests - they are most likely in initial html document initial state json object. If you view page source of the html page and ctrl f parameter name like city_id you can see it's hidden deep in some json. You can either extract this whole JSON and parse it or use regular expressions like re.findall('"city_id":(\d )', html)[0] to just get this one value.

標籤：python web-scraping

上一篇：如何在JupyterNotebook中為這個for回圈實際釋放CPU資源？

下一篇：在Python中將for回圈與bs4.element.tag一起使用