使用RSelenium在地圖上抓取多個專案位置-有解無憂

我正在嘗試對地圖進行網路抓取，以便下載街道照明圖層中的所有位置。我RSelenium用來獲取資料：

library(tidyverse)
library(rvest)
library(RSelenium)

# Open a browser
rD <- rsDriver(browser="firefox", port=4545L, verbose=F)
remDr <- rD[["client"]]
# Navigate to site
remDr$navigate("https://gis2.westberks.gov.uk/webapps/OnlineMap/")

此時，通過瀏覽器，我打開 Street Lighting 圖層（在 Highways 下），然后在地圖上選擇一個 Street Light。如果我然后運行：

h <- read_html(remDr$getPageSource()[[1]]) %>% html_nodes(".attrTable") %>% html_table()

我得到了那個單一路燈的資料。但是，我想獲取地圖上顯示的所有街道照明的資料。我不知道該怎么做。是否可以在運行前以編程方式選擇地圖上的所有燈光remDr$getPageSource()？

我看過這篇文章，但并沒有完全解決問題：問題抓取網站與反應塊

uj5u.com熱心網友回復：

它將圖層顯示為單個影像，您無法獲得位置。
它需要一些Computer Vision來檢測影像上的圓圈。

當您單擊地圖時，它將坐標發送到服務器
并將其發送回JSON data它顯示為彈出視窗。

它發送這樣的東西（帶有一些值xmin, xmax, ymin, ymax）

https://gis2.westberks.gov.uk/arcgis/rest/services/maps/Wbc_Highways/MapServer/11/query?f=json&returnGeometry=true&spatialRel=esriSpatialRelIntersects&geometry={"xmin":442520.89976831735,"ymin ":178788.66371417744,"xmax":443314.65135582053,"ymax":179582.41530168062,"spatialReference":{"wkid":2Wkid%2"77B"wkid"%2Wkid%2%2st }}&geometryType=esriGeometryEnvelope&inSR=27700&外場= OBJECTID％2CItem_Type％2CItem_Identity_Code％2CLocation_Description％2CAssigned_Street％2CLocality％2CTown％2CType％2CBracket_Type％2CLantern_Type％2CLamp_Type％2CBallast_Type％2CControl_Type％2CSign_Lantern_Type％2CSign_Bracket_Type％2CSign_Post_Type％2CBollard_Base_Type％2CBollard_Shell_Type％2CColumn_Manufacturer％2CMaterial_Type％2CLamp_Wattage％2CLantern_Manufacturer％2CNumber_of_Lamps％2CSwitching_Regime_Code％ 2CSwitching_Regime,Lamp_Type2,Easting,Northing&outSR=27700

（您可以點擊鏈接查看 JSON 資料）

也許如果您將它xmin, xmax, ymin, ymax用于更大的區域，那么您將獲得所有值。

編輯：

我沒有經驗，R但我可以在Python.

它不需要 Selenium（inPython和 in R）。

import requests

# full url with parameters
#url = 'https://gis2.westberks.gov.uk/arcgis/rest/services/maps/Wbc_Highways/MapServer/11/query?f=json&returnGeometry=true&spatialRel=esriSpatialRelIntersects&geometry={"xmin":442520.89976831735,"ymin":178788.66371417744,"xmax":443314.65135582053,"ymax":179582.41530168062,"spatialReference":{"wkid":27700,"latestWkid":27700}}&geometryType=esriGeometryEnvelope&inSR=27700&outFields=OBJECTID,Item_Type,Item_Identity_Code,Location_Description,Assigned_Street,Locality,Town,Type,Bracket_Type,Lantern_Type,Lamp_Type,Ballast_Type,Control_Type,Sign_Lantern_Type,Sign_Bracket_Type,Sign_Post_Type,Bollard_Base_Type,Bollard_Shell_Type,Column_Manufacturer,Material_Type,Lamp_Wattage,Lantern_Manufacturer,Number_of_Lamps,Switching_Regime_Code,Switching_Regime,Lamp_Type2,Easting,Northing&outSR=27700'

# only parameters
params = {
    'f': ['json'],
    'geometry': [
         '{"xmin":442520.89976831735,"ymin":178788.66371417744,"xmax":443314.65135582053,"ymax":179582.41530168062,"spatialReference":{"wkid":27700,"latestWkid":27700}}'
    ],
    'geometryType': ['esriGeometryEnvelope'],
    'inSR': ['27700'],
    'outFields': ['OBJECTID,Item_Type,Item_Identity_Code,Location_Description,Assigned_Street,Locality,Town,Type,Bracket_Type,Lantern_Type,Lamp_Type,Ballast_Type,Control_Type,Sign_Lantern_Type,Sign_Bracket_Type,Sign_Post_Type,Bollard_Base_Type,Bollard_Shell_Type,Column_Manufacturer,Material_Type,Lamp_Wattage,Lantern_Manufacturer,Number_of_Lamps,Switching_Regime_Code,Switching_Regime,Lamp_Type2,Easting,Northing'],
    'outSR': ['27700'],
    'returnGeometry': ['true'],
    'spatialRel': ['esriSpatialRelIntersects']
}

# url without parameters
url = 'https://gis2.westberks.gov.uk/arcgis/rest/services/maps/Wbc_Highways/MapServer/11/query'

response = requests.get(url, params=params)
#print(response.url)
#print(response.status_code)

data = response.json()

for item in data['features']:
    print('Locality:', item['attributes']['Locality'].strip())
    print('Town    :', item['attributes']['Town'].strip())
    print('Street  :', item['attributes']['Assigned_Street'].strip())
    print('Geometry:', item['geometry'])
    print('---')

結果：

Locality: BRIGHTWALTON
Town    : NEWBURY
Street  : SAXONS ACRE
Geometry: {'x': 442763, 'y': 179193}
---
Locality: BRIGHTWALTON
Town    : NEWBURY
Street  : ASH CLOSE
Geometry: {'x': 442782, 'y': 179248}
---
Locality: BRIGHTWALTON
Town    : NEWBURY
Street  : SAXONS ACRE
Geometry: {'x': 442770, 'y': 179214}
---

編輯：

版本在R

> install.packages("jsonlite")

> library(jsonlite)

> URL = 'https://gis2.westberks.gov.uk/arcgis/rest/services/maps/Wbc_Highways/MapServer/11/query?f=json&returnGeometry=true&spatialRel=esriSpatialRelIntersects&geometry={"xmin":442520.89976831735,"ymin":178788.66371417744,"xmax":443314.65135582053,"ymax":179582.41530168062,"spatialReference":{"wkid":27700,"latestWkid":27700}}&geometryType=esriGeometryEnvelope&inSR=27700&outFields=OBJECTID,Item_Type,Item_Identity_Code,Location_Description,Assigned_Street,Locality,Town,Type,Bracket_Type,Lantern_Type,Lamp_Type,Ballast_Type,Control_Type,Sign_Lantern_Type,Sign_Bracket_Type,Sign_Post_Type,Bollard_Base_Type,Bollard_Shell_Type,Column_Manufacturer,Material_Type,Lamp_Wattage,Lantern_Manufacturer,Number_of_Lamps,Switching_Regime_Code,Switching_Regime,Lamp_Type2,Easting,Northing&outSR=27700'

> data <- fromJSON(URL)

> library(magrittr)  # to use `%>%
> data <- URL %>% fromJSON

> data$features$attributes$Locality

[1] "BRIGHTWALTON                                                                                        "
[2] "BRIGHTWALTON                                                                                        "
[3] "BRIGHTWALTON                                                                                        "

> data$features$attributes$Town

[1] "NEWBURY                                                                                             "
[2] "NEWBURY                                                                                             "
[3] "NEWBURY                                                                                             "

> data$features$attributes$Assigned_Street

[1] "SAXONS ACRE                                                                                         "
[2] "ASH CLOSE                                                                                           "
[3] "SAXONS ACRE                                                                                         "

> data$features$geometry

       x      y
1 442763 179193
2 442782 179248
3 442770 179214

> library(stringr)

> data$features$attributes$Locality %>% str_trim

[1] "BRIGHTWALTON" "BRIGHTWALTON" "BRIGHTWALTON"

編輯：

類似于Python版本的東西

> library(magrittr)  # to use `%>%
> library(httr)
> library(jsonlite)

> URL = 'https://gis2.westberks.gov.uk/arcgis/rest/services/maps/Wbc_Highways/MapServer/11/query'

> query = list(
    f = list('json'),
    geometry = list(
         '{"xmin":442520.89976831735,"ymin":178788.66371417744,"xmax":443314.65135582053,"ymax":179582.41530168062,"spatialReference":{"wkid":27700,"latestWkid":27700}}'
    ),
    geometryType = list('esriGeometryEnvelope'),
    inSR = list('27700'),
    outFields = list('OBJECTID,Item_Type,Item_Identity_Code,Location_Description,Assigned_Street,Locality,Town,Type,Bracket_Type,Lantern_Type,Lamp_Type,Ballast_Type,Control_Type,Sign_Lantern_Type,Sign_Bracket_Type,Sign_Post_Type,Bollard_Base_Type,Bollard_Shell_Type,Column_Manufacturer,Material_Type,Lamp_Wattage,Lantern_Manufacturer,Number_of_Lamps,Switching_Regime_Code,Switching_Regime,Lamp_Type2,Easting,Northing'),
    outSR = list('27700'),
    returnGeometry = list('true'),
    spatialRel = list('esriSpatialRelIntersects')
  )

> response <- GET(URL, query=query)
> data <- response %>% content %>% fromJSON

> data <- GET(URL, query=query) %>% content %>% fromJSON

> items <- data$features

轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/441999.html

標籤：r 网页抓取硒

上一篇：如何從字串中提取url并保存到串列中

下一篇：使用R進行網頁抓取-表格內容