SeleniumC#網頁抓取-無法決議/決議htmlPageSource

我在 C# 中為自己制作了一個簡單的 .NET 控制臺應用程式，以使用 Selenium C# 抓取動態頁面供個人使用。

selenium 導航作業得很好，但是當我要決議結果頁面源并檢索房地產地址串列時，它回傳 null。最重要的是，它還提供與 chrome 瀏覽器相關的警告和錯誤。

完整代碼：

public static void SeleniumExtract()
{
    // initial setup
    IWebDriver driver = new ChromeDriver();

    driver.Navigate().GoToUrl("https://www.knightfrank.co.uk/");

    WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(1));

    // dropdown
    var dropdown1 = driver.FindElement(By.Id("cpMain_ucc1_ctl00_liResidentialFront"));
    dropdown1.Click();
    
    // enter search query
    var search = driver.FindElement(By.Id("cpMain_ucc1_ctl00_txtResidentialSearchBox"));
    search.Click();
    search.SendKeys("London");

    // select search suggection from dropdown menu
    wait.Until(SeleniumExtras.WaitHelpers.ExpectedConditions.ElementExists(By.Id("ui-id-4")));
    var suggestion = driver.FindElement(By.Id("ui-id-4"));
    suggestion.Click();

    // submit search
    var submit = driver.FindElement(By.XPath("//div[@id='cpMain_ucc1_ctl00_pnlContentResidential']//a[@class='search-button']"));
    submit.Click();

    // get the data
    var elements = driver.FindElements(By.XPath("//div[@class='grid-address']"));
    
    foreach(var item in elements)
    {
        Console.WriteLine(item.Text);
    }

}

問題 #1 - 硒：

Selenium 不會向控制臺回傳任何結果。xpath//div[@class='grid-address']是絕對準確的，所以沒有拼寫錯誤。我不知道為什么它不將 foreach 專案結果輸出到控制臺，即這部分代碼不起作用：

// get the data
var elements = driver.FindElements(By.XPath("//div[@class='grid-address']"));

foreach(var item in elements)
{
    Console.WriteLine(item.Text);
}

問題 #2 - Html 敏捷包：

或者，我嘗試使用 Html Agility Pack 來決議 PageSource，它只回傳空的 null 例外。

首先，我從 SeleniumExtract() 回傳頁面源：

// export current pagesource
var currentPage = driver.PageSource;
return currentPage;

And then, I load the page source into Html Agility Pack to work with. It returns nothing!

public static void HapParse(string currentPage)
{
    HtmlDocument htmlDoc = new HtmlDocument();
    htmlDoc.Load(currentPage);

    var address = htmlDoc.DocumentNode
        .SelectNodes("//div[@class='grid-address']")
        .ToList();

    foreach(var item in address)
    {
        Console.WriteLine(item.InnerText);
    }
    
}

Problem #3 - AngleSharp:

Tried to do the same with Angle Sharp, and still not working.

public static async void AngleSharpParse(string currentPage)
{
    var config = Configuration.Default;
    var context = BrowsingContext.New(config);

    var document = await context.OpenAsync(req => req.Content(currentPage));

    var elements = document.QuerySelectorAll("div.grid-address");

    foreach (var item in elements)
    {
        Console.WriteLine(item.TextContent);
    }
}

Problem #4 - Warning and error relating to Chrome browser:

The console also returns the following errors everytime I execute the code:

[39536:28224:1103/183909.271:ERROR:chrome_browser_main_extra_parts_metrics.cc(230)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that this ends.
[39536:28224:1103/183909.271:ERROR:chrome_browser_main_extra_parts_metrics.cc(233)] crbug.com/1216328: Checking Bluetooth availability ended.
[39536:28224:1103/183909.271:ERROR:chrome_browser_main_extra_parts_metrics.cc(236)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that this ends.
[39536:39768:1103/183909.275:ERROR:device_event_log_impl.cc(214)] [18:39:09.274] USB: usb_service_win.cc:389 Could not read device interface GUIDs: The system cannot find the file specified. (0x2)
[39536:39768:1103/183909.280:ERROR:device_event_log_impl.cc(214)] [18:39:09.280] USB: usb_device_handle_win.cc:1048 
Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[39536:28224:1103/183909.291:ERROR:chrome_browser_main_extra_parts_metrics.cc(240)] crbug.com/1216328: Checking default browser status ended.
[39536:39768:1103/183909.310:ERROR:device_event_log_impl.cc(214)] [18:39:09.310] USB: usb_device_handle_win.cc:1048 
Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)

Please can someone tell me why this very simple code just doesn't return anyresult?

p.s.

I actually tried to code the exact same thing in Python using Selenium and Beautiful Soup and everything works perfectly.

What am I missing here?

uj5u.com熱心網友回復：

我只需要在 Selenium 提交搜索查詢后添加一個等待計時器，例如Thread.Sleep(3000)，讓頁面在決議 HTML 之前完全加載。

標籤：c# selenium web-scraping html-agility-pack anglesharp

上一篇：Scrapy-抓取概覽頁面和詳細資訊頁面？

下一篇：檢查值的范圍是升序、降序還是非單調