提供None抓取地址-有解無憂

我正在抓取地址，但他們不會向我提供這些是頁面鏈接https://www.baroulconstanta.ro/tabloul-avocatilor/avocati-definitivi/

import scrapy
from scrapy.http import Request


class TestSpider(scrapy.Spider):
    name = 'test'
    start_urls=["https://www.baroulconstanta.ro/tabloul-avocatilor/avocati-definitivi/"]
    
    
    
    
    def parse(self, response):
        address=response.xpath("//div[@class='col-md-12']//p[1]//text()[2]").get()
        print(address)

uj5u.com熱心網友回復：

您得到空輸出，因為資料是通過 ajax 請求作為 post 方法從外部生成的。所以你必須使用 API url

from scrapy.crawler import CrawlerProcess
from scrapy.http import Request

class TestSpider(scrapy.Spider):
    name = 'test'
    
    def start_requests(self):
        yield Request(
            url= "https://www.ifep.ro/Justice/Lawyers/LawyersPanel.aspx?CompanyId=1115&CurrentPanel=Definitivi&HideHeader=1&HideFooter=1&HideFilters=1",
            callback=self.parse,
            method= "POST",
            headers= {
                'content-type': 'application/x-www-form-urlencoded',
                'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36',
                'x-microsoftajax':'Delta=true',
                'x-requested-with': 'XMLHttpRequest'
            }
        )

    def parse(self, response):
        for address in response.xpath('//*[@]'):
            print(''.join([x.get() for x in address.xpath('.//p/text()[2]')]).replace('\r\n\t','').strip())

if __name__ == "__main__":
    process =CrawlerProcess()
    process.crawl(TestSpider)
    process.start()

輸出：

Sediu principal ?n Baroul Constan?a, adres?: NAVODARI STR .CONSTANTEI NR. 15 BL, E 1, SC. A, APRT.14
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA STR. BOGDAN VODA NR. 63 A
Sediu principal ?n Baroul Constan?a, adres?: NAVODARI CARTIER MAMAIA SAT STR. M 10 , NR. 26  CSediu principal ?n Baroul Constan?a, adres?: CONSTAN?A BD. FERDINAND NR. 26
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. VASILE PARVAN NR. 5
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. MIRCEA NR. 181, BL. M S 3 B, SC. B, AP. 38
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA STR. ILEANA COSANZEANA NR. 18
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA STR.PESCARILOR NR. 35 BL. fz 14, APRT. 
51
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA BD. TOMIS NR. 46 , ET.2 ,CAMERA 29     
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. NICOLAE MILESCU NR. 57 BL. P N 2 , SC.A ,APRT. 42
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A BD.FERDINAND NR. 87, BL. A 4, SC. A. APRT. 1
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. C?L?RA?I NR. 46 ET. 1
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA STR. CONSTANTIN BOBESCU NR. 23, BL C, AP. 6
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. DIMITRIE BOLINTINEANU NR. 32 A    
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. NEHOIULUI NR. 37

uj5u.com熱心網友回復：

您無法提取資料，因為您要查找的頁面部分位于 iframe（頁面內的頁面）中。您有兩種選擇：您可以手動查找 iframe 的 URL 并在代碼中決議它，或者您可以通過編程方式找到此 URL 并再次在代碼中決議它。

以下是第二種方法的方法（請多多包涵，這是我第一次使用scrapy，所以我很可能會對其進行屠宰，并且有更合適的方法）：

import scrapy

class TestSpider(scrapy.Spider):
    name = 'test'
    start_urls=["https://www.baroulconstanta.ro/tabloul-avocatilor/avocati-definitivi/"]
    
    def parse(self, response):
        frame = response.css('iframe::attr(src)').get()
        yield scrapy.Request(url=frame, callback=self.parse_frame, meta=response.meta)

    def parse_frame(self, response):
        addresses = response.xpath("//div[@class='col-md-12']//p[1]//text()[2]")

        for addr in addresses:
            print(addr.get().strip())

產生：

Sediu principal ?n Baroul Constan?a, adres?: NAVODARI STR .CONSTANTEI NR. 15 BL, E 1, SC. A, APRT.14
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA STR. BOGDAN VODA NR. 63 A
Sediu principal ?n Baroul Constan?a, adres?: NAVODARI CARTIER MAMAIA SAT STR. M 10 , NR. 26  C
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A BD. FERDINAND NR. 26
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. VASILE PARVAN NR. 5
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. MIRCEA NR. 181, BL. M S 3 B, SC. B, AP. 38
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA STR. ILEANA COSANZEANA NR. 18
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA STR.PESCARILOR NR. 35 BL. fz 14, APRT. 51
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA BD. TOMIS NR. 46 , ET.2 ,CAMERA 29
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. NICOLAE MILESCU NR. 57 BL. P N 2 , SC.A ,APRT. 42
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A BD.FERDINAND NR. 87, BL. A 4, SC. A. APRT. 1
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. C?L?RA?I NR. 46 ET. 1
Sediu principal ?n Baroul Constan?a, adres?: CONSTANTA STR. CONSTANTIN BOBESCU NR. 23, BL C, AP. 6
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. DIMITRIE BOLINTINEANU NR. 32 A
Sediu principal ?n Baroul Constan?a, adres?: CONSTAN?A STR. NEHOIULUI NR. 37

轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/496241.html

標籤：Python 网页抓取刮擦

上一篇：如果它們都具有相同的div類名稱，我如何單擊第一個DIV類“鏈接”？

下一篇：格式化抓取的內容