我已經通過 pip 成功地在本地安裝了 apache 氣流 .. 有一些需要的引腳
pip3 install zipp==3.1.0
pip3 install sqlalchemy==1.3.24
python3 -m pip install virtualenv
pip3 install apache-airflow[cncf.kubernetes]
pip3 install apache-airflow
因為我是n00b所有這些東西，所以我從基礎開始......我第一次嘗試airflow standalone但沒有在檔案中的任何地方說明它的默認用戶名和密碼是什么......所以我進入并開始了基本用戶的服務很少......
airflow db init
airflow users create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin
現在這只是需要啟動.. 我意識到我需要啟動調度程式和 web 應用程式......出于某種原因，我的自動腳本不這樣做.. 我必須手動執行但..
airflow scheduler &
airflow webapp
現在一切都好..我可以看到我啟動并運行了一個啟動 GUI.. 事情看起來很好..我想啟動我發現的第一個 DAG
example_bash_operator
問題是...當我點擊名稱...或點擊go...一半時間它有效...但通常是前幾次我點擊任何東西...我遇到一個錯誤
Python version: 3.8.10
Airflow version: 2.2.3
Node: juju-2dd159-310.lxd
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 51, in decorated
return func(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/decorators.py", line 72, in wrapper
return f(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/views.py", line 1732, in trigger
if unpause and dag.is_paused:
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 1081, in is_paused
warnings.warn(
File "/usr/lib/python3.8/warnings.py", line 109, in _showwarnmsg
sw(msg.message, msg.category, msg.filename, msg.lineno,
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/settings.py", line 117, in custom_show_warning
write_console.print(msg, soft_wrap=True)
File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 1642, in print
self._buffer.extend(new_segments)
File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 842, in __exit__
self._exit_buffer()
File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 800, in _exit_buffer
self._check_buffer()
File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 1935, in _check_buffer
self.file.flush()
BrokenPipeError: [Errno 32] Broken pipe
如果我忽略這一點，也許等一下，或者再試一次..突然它起作用了......任何線索如何平滑這種體驗？
編輯：柜面這有助于回答問題
ubuntu@juju-2dd159-311:~$ pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
ubuntu@juju-2dd159-311:~$ python3 --version
Python 3.8.10
編輯#2
我按照這些說明安裝了他們想要的約束https://airflow.apache.org/docs/apache-airflow/stable/start/local.html
這極大地修復了 GUI 和 UI 中的穩定性......但是......然后我開始連接 postgresql 資料庫......現在它甚至不會在沒有brokenpipe錯誤的情況下登錄
Python version: 3.8.10
Airflow version: 2.2.3
Node: juju-2dd159-318.lxd
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 51, in decorated
return func(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/views.py", line 718, in index
paging=wwwutils.generate_pages(
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/utils.py", line 113, in generate_pages
previous_node = Markup(
File "/home/ubuntu/.local/lib/python3.8/site-packages/jinja2/utils.py", line 838, in __new__
warnings.warn(
File "/usr/lib/python3.8/warnings.py", line 109, in _showwarnmsg
sw(msg.message, msg.category, msg.filename, msg.lineno,
File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/settings.py", line 117, in custom_show_warning
write_console.print(msg, soft_wrap=True)
File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 1642, in print
self._buffer.extend(new_segments)
File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 842, in __exit__
self._exit_buffer()
File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 800, in _exit_buffer
self._check_buffer()
File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 1935, in _check_buffer
self.file.flush()
uj5u.com熱心網友回復：
您是否嘗試過遵循“快速入門”說明？
https://airflow.apache.org/docs/apache-airflow/stable/start/index.html
Airflow 有關于如何開始的詳盡而全面的說明，如果您按部就班地遵循它，您將啟動并運行 Airflow。這可以通過 docker compose 或本地 virtualenv 來完成。
您的問題可能是資源不足 - 記憶體（最有可能）。Airflow 需要相當多的記憶體 (4GB) 才能啟動，因為它是一個復雜的系統。它是作為先決條件撰寫的，尤其是在 Docker Compose 快速入門中。如果你沒有足夠的資源，Docker Compose 甚至會警告你，所以如果你想要真正可靠和強大的快速入門，我推薦這個。
您需要查看日志以了解為什么會出現管道損壞錯誤。但缺乏資源是最有可能的原因。
關于“獨立”模式和用戶密碼 - 您可能錯過了氣流寫給您的內容。它在啟動時動態生成密碼，并實際告訴您應該使用什么密碼：
standalone |
standalone | Airflow is ready
standalone | Login with username: admin password: 4hfH8mATcvMFmne9
standalone | Airflow Standalone is for development purposes only. Do not use this in production!
standalone |
uj5u.com熱心網友回復：
所以結果是.. 撰寫一個簡單的腳本來在后臺運行 apache 氣流
#!/bin/bash
airflow webserver -D
在 Ubuntu LTS 20.04 上不能正常作業或玩得不好.. 我現在發現我應該讓 SystemD 處理啟動停止.. 現在作業得很好.. 這是我在 Ubuntu 中注入所需腳本的方法20.04 LTS
這將注冊 webserver、scheduler 和 triggerer 服務
if [ ! -e /etc/systemd/system/airflow-scheduler.service ]; then
cat <<EOT >> /etc/systemd/system/airflow-scheduler.service
[Unit]
Description=Airflow scheduler daemon
[Service]
Environment="PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
User=ubuntu
Type=simple
ExecStart=/home/ubuntu/.local/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
EOT
fi
if [ ! -e /etc/systemd/system/airflow-webserver.service ]; then
cat <<EOT >> /etc/systemd/system/airflow-webserver.service
[Unit]
Description=Airflow webserver daemon
[Service]
Environment="PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
User=ubuntu
Type=simple
ExecStart=/home/ubuntu/.local/bin/airflow webserver
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target
EOT
fi
if [ ! -e /etc/systemd/system/airflow-triggerer.service ]; then
cat <<EOT >> /etc/systemd/system/airflow-triggerer.service
[Unit]
Description=Airflow triggerer daemon
[Service]
Environment="PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
User=ubuntu
Type=simple
ExecStart=/home/ubuntu/.local/bin/airflow triggerer
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target
EOT
fi
systemctl daemon-reload
systemctl enable airflow-scheduler
systemctl enable airflow-webserver
systemctl enable airflow-triggerer
然后開始我只是做
systemctl start airflow-webserver
systemctl start airflow-scheduler
systemctl start airflow-triggerer
轉載請註明出處，本文鏈接：https://www.uj5u.com/gongcheng/403411.html
標籤：