我一直在關注本教程：https : //kb.objectrocket.com/postgresql/scrape-a-website-to-postgres-with-python-938
我的 app.py 檔案如下所示（取自上述教程）：
from flask import Flask # needed for flask-dependent libraries below
from flask import render_template # to render the error page
from selenium import webdriver # to grab source from URL
from bs4 import BeautifulSoup # for searching through HTML
import psycopg2 # for database access
# set up Postgres database connection and cursor.
t_host = "localhost" # either "localhost", a domain name, or an IP address.
t_port = "5432" # default postgres port
t_dbname = "scrape"
t_user = "postgres"
t_pw = "********"
db_conn = psycopg2.connect(host=t_host, port=t_port, dbname=t_dbname, user=t_user, password=t_pw)
db_cursor = db_conn.cursor()
app = Flask(__name__)
@app.route("/")
@app.route('/import_temp')
def import_temp():
# set up your webdriver to use Chrome web browser
my_web_driver = webdriver.Chrome("/usr/local/bin/chromedriver")
# designate the URL we want to scrape
# NOTE: the long string of characters at the end of this URL below is a clue that
# maybe this page is so dynamic, like maybe refers to a specific web session and/or day/time,
# that we can't necessarily count on it to be the same more than one time.
# Which means... we may want to find another source for our data; one that is more
# dependable. That said, whatever URL you use, the methodology in this lesson stands.
t_url = "https://weather.com/weather/today/l/7ebb344012f0c5ff88820d763da89ed94306a86c770fda50c983bf01a0f55c0d"
# initiate scrape of website page data
my_web_driver.get(t_url)
# return entire page into "t_content"
t_content = my_web_driver.page_source
# use soup to make page content easily searchable
soup_in_bowl = BeautifulSoup(t_content, 'html.parser')
# search for the UNIQUE span and class for the data we are looking for:
o_temp = soup_in_bowl.find('span', attrs={'class': 'deg-feels'})
# from the resulting object, "o_temp", get the text parameter and assign it to "n_temp"
n_temp = o_temp.text
# Build SQL for purpose of:
# saving the temperature data to a new row
s = ""
s = "INSERT INTO tbl_temperatures"
s = "("
s = "n_temp"
s = ") VALUES ("
s = "(%n_temp)"
s = ")"
# Trap errors for opening the file
try:
db_cursor.execute(s, [n_temp, n_temp])
db_conn.commit()
except psycopg2.Error as e:
t_msg = "Database error: " e "/n open() SQL: " s
return render_template("error_page.html", t_msg = t_msg)
# Success!
# Show a message to user.
t_msg = "Successful scrape!"
return render_template("progress.html", t_msg = t_msg)
# Clean up the cursor and connection objects
db_cursor.close()
db_conn.close()
我已成功匯入所需的庫，并創建了一個資料庫來存盤抓取的資料。但是，當我運行應用程式時，我現在收到一個錯誤：
FLASK_APP = app.py
FLASK_ENV = development
FLASK_DEBUG = 0
In folder /home/lloyd/PycharmProjects/flaskProject
/home/lloyd/PycharmProjects/flaskProject/venv/bin/python -m flask run
* Serving Flask app 'app.py' (lazy loading)
* Environment: development
* Debug mode: off
* Running on http://127.0.0.1:5000/ (Press CTRL C to quit)
[2021-12-29 20:51:32,817] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/lloyd/PycharmProjects/flaskProject/app.py", line 41, in import_temp
n_temp = o_temp.text
AttributeError: 'NoneType' object has no attribute 'text'
127.0.0.1 - - [29/Dec/2021 20:51:32] "GET / HTTP/1.1" 500 -
我認為該錯誤表明傳遞給 BeautifulSoup 方法的資料不正確： Line 41, in import_temp n_temp = o_temp.text
如果有人能夠告訴我此例外的原因以及如何解決它，將不勝感激。
謝謝
uj5u.com熱心網友回復：
您需要在瀏覽器開發工具中查看網站結構。這是它在這個特定網站上的樣子：
<div
data-testid="FeelsLikeSection"
class="TodayDetailsCard--feelsLikeTemp--3fwAJ"
>
<span
data-testid="TemperatureValue"
class="TodayDetailsCard--feelsLikeTempValue--Cf9Sl"
>54°</span
><span
data-testid="FeelsLikeLabel"
class="TodayDetailsCard--feelsLikeTempLabel--3v2vt"
>Feels Like</span
>
</div>
在<span>與實感溫度具有獨特的類名，但是這個類名是自動生成的（可能是類似樣式化組件），并且每個站點重建時間可能會發生變化，所以你不應該依賴于這個。
<div>另一方面，可以通過data-testid屬性來區分封閉，這是站點開發人員為他們的自動化測驗放置的。這不太可能改變，因為它會破壞他們的測驗。
在<span>與溫度也有data-testid屬性，但在這個網站上有很多<span>s的相同屬性值，所以由查詢{'data-testid': 'TemperatureValue'}不會給你一個獨特的元素。
所以你可以像這樣進行兩級查詢：
o_temp_feel = soup_in_bowl.find('div', attrs={'data-testid': 'FeelsLikeSection'})
o_temp = o_temp_feel.find('span', attrs={'data-testid': 'TemperatureValue'})
# from the resulting object, "o_temp", get the text parameter and assign it to "n_temp"
n_temp = o_temp.text
請注意，n_temp將類似于“54°”，因此如果您需要數值，則需要在轉換為整數之前將其截斷。
轉載請註明出處，本文鏈接：https://www.uj5u.com/gongcheng/399533.html
標籤：Python PostgreSQL的 烧瓶 美汤
上一篇：從表列評估比較運算子