目標
客戶用產品可能是功能設計好,也可能是因為回應快穩定可靠,例如滴滴用不了用高德,券商app故障受罰,微信凌晨服務崩潰,所以穩定性建設作業價值是保障客戶體驗,避免資損,社會負面輿論,
故障生命周期處理
圍繞故障生命周期,在整個故障定位體系,可分為故障開始前,預案階段,做量化分析找到潛在隱患;故障開始后,盡快發現定位故障直接原因,直接原因定位是為了止損,根因可以后續排查;故障恢復后就是復盤,行程TODO list,針對性改進,
預案階段
1.可觀測性體系
基礎設施和軟體架構都比較完善情況下并不能萬事大吉,線上問題防不勝防,建設可觀測體系是必需的,預防階段兩件事,埋點資料采集,資料組織,便于后續排障,
可觀測性資料通常分四類,指標,鏈路,日志,事件,
指標,存盤成本小應用最廣泛,其中很多熟知的產品包括,成為監控系統業界標準的Prometheus,時間序列資料庫VictoriaMetrics,OpenTSDB,采集器Telegraf、Categraf、Grafana-agent、Datadog-agent,
鏈路,服務數量眾多,關系復雜,導致服務故障很難排查的情況下需要引入鏈路追蹤系統,業界推出了觀測度量框架OpenTelemetry,可以基本解決鏈路監控需求,推動落地需要所有模塊都接入才有價值,
日志,是最重要的問題排查手段,存盤成本高,所以管理日志需要精細化,較久遠資料幾乎沒有查詢需求,近期資料存ES用于排查問題,選取日志中數值類指標存時序庫,保存更久一點,
排查問題,通常是指標先提示例外,然后查看相關時間段日志,日志里可能有traceid,再去查詢鏈路資料,從而更快找到故障直接原因,
事件,通常包含告警事件,變更事件,故障定位是事件也需要統一收集,從時間維度做關聯分析
2.風險量化體系
主要是分析評價可觀測性體系成果,是否完備,還能量化變更體系,確認各團隊變更操作是否值得依賴,例如不做灰度直接全量,經常高峰期上線,經常回滾,量化團隊健康分,督促差的業務線去改進,
以上內容摘選于大佬(秦曉輝@快貓星云)的文章<穩定性體系建設白皮書>
夜鶯方案
夜鶯(Nightingale)是一款可視化監控工具產品,夜鶯開源版源于滴滴運維團隊,是國內最活躍的企業級云原生監控方案,被很多團隊選用部署落地,經過生產實踐,通過Categraf、VictoriaMetrics、Nightingale可以方便我們快速搭建可觀測性體系,夜鶯是一個服務端組件,類似Grafana,可以接入不同的資料源,夜鶯就可以對資料源的資料進行分析、告警、可視化,以及后續的事件處理、告警自愈(和Grafana一樣提供可視化,對告警規則管理不同于Prometheus通過組態檔來實作,夜鶯通過WebUI來統一協同) 夜鶯商業版產品提供了更多企業級功能,用于統一監控和故障定位場景,
夜鶯V6入門
作業系統 云耀云服務器(Hyper Elastic Cloud Server)
cat /etc/centos-release
AlmaLinux release 8.4 (Electric Cheetah)
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: AlmaLinux
Description: AlmaLinux release 8.4 (Electric Cheetah)
Release: 8.4
Codename: ElectricCheetah
uname -a
Linux hecs-34116 4.18.0-372.26.1.el8_6.x86_64 #1 SMP Tue Sep 13 06:07:14 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
cpu資訊
lscpu|grep CPU
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 4
On-line CPU(s) list: 0-3
CPU family: 6
Model name: Intel(R) Xeon(R) Gold 6278C CPU @ 2.60GHz
CPU MHz: 2600.000
NUMA node0 CPU(s): 0-3
x86_64,x64,AMD64基本上是同一個東西,我們現在用的intel/amd的桌面級CPU基本上都是x86_64
mariadb安裝
默認源
yum info mariadb
Last metadata expiration check: 3:17:56 ago on Thu 13 Apr 2023 07:04:57 AM CST.
Available Packages
Name : mariadb
Epoch : 3
Version : 10.3.35
Release : 1.module_el8.6.0+3265+230ed96b
Architecture : x86_64
Size : 6.0 M
Source : mariadb-10.3.35-1.module_el8.6.0+3265+230ed96b.src.rpm
Repository : appstream
Summary : A very fast and robust SQL database server
URL : http://mariadb.org
License : GPLv2 with exceptions and LGPLv2 and BSD
Description : MariaDB is a community developed branch of MySQL - a multi-user, multi-threaded
: SQL database server. It is a client/server implementation consisting of
: a server daemon (mysqld) and many different client programs and libraries.
: The base package contains the standard MariaDB/MySQL client programs and
: generic MySQL files.
添加MariaDB yum 源, 官網按需要選擇源
vim /etc/yum.repos.d/MariaDB.repo
# MariaDB 11.0 [RC] CentOS repository list - created 2023-04-13 02:37 UTC
# https://mariadb.org/download/
[mariadb]
name = MariaDB
# rpm.mariadb.org is a dynamic mirror if your preferred mirror goes offline. See https://mariadb.org/mirrorbits/ for details.
# baseurl = https://rpm.mariadb.org/11.0/centos/$releasever/$basearch
baseurl = https://mirrors.neusoft.edu.cn/mariadb/yum/11.0/centos/$releasever/$basearch
module_hotfixes = 1
# gpgkey = https://rpm.mariadb.org/RPM-GPG-KEY-MariaDB
gpgkey = https://mirrors.neusoft.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck = 1
重新構建快取,
yum clean all
yum makecache
卸載舊版
yum remove mariadb-server mariadb mariadb-libs
yum clean all
找出并洗掉殘留目錄
find / -name mariadb
find / -name mysql
安裝新版及啟動資料庫
yum install MariaDB-server
一路y下去
查看狀態:
systemctl status mariadb
● mariadb.service - MariaDB 11.0.1 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: inactive (dead)
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
啟動
systemctl start mariadb
此時意外發現報錯了
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
按照提示去查看錯誤,埠被占用
...
Apr 13 10:59:59 hecs-34116 mariadbd[2579389]: 2023-04-13 10:59:59 0 [Note] Server socket created on IP: '0.0.0.0'.
Apr 13 10:59:59 hecs-34116 mariadbd[2579389]: 2023-04-13 10:59:59 0 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 98: Address already in use
Apr 13 10:59:59 hecs-34116 mariadbd[2579389]: 2023-04-13 10:59:59 0 [ERROR] Do you already have another server running on port: 3306 ?
Apr 13 10:59:59 hecs-34116 mariadbd[2579389]: 2023-04-13 10:59:59 0 [ERROR] Aborting
...
netstat -ntuap
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 232910/nginx: maste
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 232910/nginx: maste
換埠
vi /etc/my.cnf.d/server.cnf
搜索行統計以[mysqld]開始,并在[mysqld]陳述句下放置以下埠指令,如以下檔案摘錄所示, 相應地更換埠變數,
[mysqld]
port = 12345
再次啟動,并查看狀態
systemctl start mariadb
systemctl status mariadb
● mariadb.service - MariaDB 11.0.1 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Thu 2023-04-13 11:21:07 CST; 2min 44s ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 2579593 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 2579562 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment>
Process: 2579560 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Main PID: 2579576 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 9 (limit: 49448)
Memory: 169.3M
CGroup: /system.slice/mariadb.service
└─2579576 /usr/sbin/mariadbd
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: 2023-04-13 11:21:07 0 [Note] InnoDB: log sequence number 47295; transaction id 14
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: 2023-04-13 11:21:07 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: 2023-04-13 11:21:07 0 [Note] Plugin 'FEEDBACK' is disabled.
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: 2023-04-13 11:21:07 0 [Note] Plugin 'wsrep-provider' is disabled.
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: 2023-04-13 11:21:07 0 [Note] Server socket created on IP: '0.0.0.0'.
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: 2023-04-13 11:21:07 0 [Note] Server socket created on IP: '::'.
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: 2023-04-13 11:21:07 0 [Note] InnoDB: Buffer pool(s) load completed at 230413 11:21:07
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: 2023-04-13 11:21:07 0 [Note] /usr/sbin/mariadbd: ready for connections.
Apr 13 11:21:07 hecs-34116 mariadbd[2579576]: Version: '11.0.1-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 12345 MariaDB Server
Apr 13 11:21:07 hecs-34116 systemd[1]: Started MariaDB 11.0.1 database server.
成功了接下來,給MariaDB設定用戶及密碼
連接資料庫
mysql
select user, host, plugin from mysql.user;
+-------------+------------+-----------------------+
| User | Host | plugin |
+-------------+------------+-----------------------+
| mariadb.sys | localhost | mysql_native_password |
| root | localhost | mysql_native_password |
| mysql | localhost | mysql_native_password |
| PUBLIC | | |
| | localhost | |
| | hecs-34116 | |
+-------------+------------+-----------------------+
設定權限和密碼
GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY '123456';
退出后再登錄mysql
mysql -uroot -p123456 -e "show databases"
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
連接test庫
mysql -uroot -p -D test
安裝redis
查看可用源
dnf list|grep redis
hiredis.x86_64 0.13.3-13.el8 epel
hiredis-devel.x86_64 0.13.3-13.el8 epel
pcp-pmda-redis.x86_64 5.3.7-7.el8 appstream
perl-RDF-Trine-redis.noarch 1.019-8.el8 epel
python3-redis.noarch 3.5.3-1.el8 epel
redis.x86_64 5.0.3-5.module_el8.4.0+2583+b9845322 appstream
redis-devel.x86_64 5.0.3-5.module_el8.4.0+2583+b9845322 appstream
redis-doc.noarch 5.0.3-5.module_el8.4.0+2583+b9845322 appstream
syslog-ng-redis.x86_64 3.23.1-3.el8 epel
uwsgi-logger-redis.x86_64 2.0.21-2.el8 epel
uwsgi-router-redis.x86_64 2.0.21-2.el8 epel
官網下載新版
cd /home/tarball/
wget https://codeload.github.com/redis/redis/tar.gz/refs/tags/7.0.10 -O redis-7.0.10.tar.gz
tar zxvf redis-7.0.10.tar.gz
cd redis-7.0.10
make
成功后輸出
Hint: It's a good idea to run 'make test' ;)
make[1]: Leaving directory '/home/tarball/redis-7.0.10/src'
make PREFIX=/home/tarball/redis-7.0.10 install
成功后輸出
cd src && make install
make[1]: Entering directory '/home/tarball/redis-7.0.10/src'
CC Makefile.dep
Hint: It's a good idea to run 'make test' ;)
INSTALL redis-server
INSTALL redis-benchmark
INSTALL redis-cli
make[1]: Leaving directory '/home/tarball/redis-7.0.10/src'
啟動redis
./bin/redis-server& ./redis.conf
查看redis狀態
ps -aux | grep redis
root 2584263 0.0 0.1 62644 10196 pts/0 Sl 12:51 0:00 ./bin/redis-server *:6379
root 2584270 0.0 0.0 12140 1116 pts/1 S+ 12:51 0:00 grep --color=auto redis
netstat -ntuap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2487/sshd
tcp 0 0 0.0.0.0:12345 0.0.0.0:* LISTEN 2579576/mariadbd
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 2584263/./bin/redis
安裝 TSDB
按官方檔案描述‘小規模使用,比如 1000 臺機器以下,用 Prometheus 做存盤即可,超過 1000 臺機器,選擇 VictoriaMetrics 可能更合適
VictoriaMetrics, 提供單機版和集群版,如果您的每秒寫入資料點數小于100萬(這個數量是個什么概念呢,如果只是做機器設備的監控,每個機器差不多采集200個指標,采集頻率是10秒的話每臺機器每秒采集20個指標左右,100萬/20=5萬臺機器),VictoriaMetrics 官方默認推薦您使用單機版,單機版可以通過增加服務器的CPU核心數,增加記憶體,增加IOPS來獲得線性的性能提升,且單機版易于配置和運維,’
另外聽大佬說,大規模使用時候夜鶯的主要瓶頸在TSDB上,所以這次選用單機版VictoriaMetrics
下載VictoriaMetrics
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.90.0/victoria-metrics-linux-amd64-v1.90.0.tar.gz
下載還是有點慢的
mkdir victoria-metrics
tar xf victoria-metrics-linux-amd64-v1.90.0.tar.gz -C victoria-metrics
cd victoria-metrics
啟動
nohup ./victoria-metrics-prod &>victoria.log &
查看默認埠8428
ss -ntpl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=2487,fd=5))
LISTEN 0 80 0.0.0.0:12345 0.0.0.0:* users:(("mariadbd",pid=2579576,fd=21))
LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* users:(("redis-server",pid=2584263,fd=6))
LISTEN 0 1024 0.0.0.0:8428 0.0.0.0:* users:(("victoria-metric",pid=2584341,fd=10))
安裝夜鶯
官網下載
wget https://download.flashcat.cloud/n9e-v6.0.0-ga.3-linux-amd64.tar.gz
mkdir n9e
tar zxvf n9e-v6.0.0-ga.3-linux-amd64.tar.gz -C n9e
匯入資料庫
mysql -uroot -p <n9e.sql
修改 N9e 的組態檔 (需要注意上線前修改密鑰Auth相關欄位)
vim etc/config.toml
[DB]
# postgres: host=%s port=%s user=%s dbname=%s password=%s sslmode=%s
DSN="root:123456@tcp(127.0.0.1:12345)/n9e_v6?charset=utf8mb4&parseTime=True&loc=Local&allowNativePasswords=true" #更換db密碼,埠
[[Pushgw.Writers]]
# Url = "http://127.0.0.1:8480/insert/0/prometheus/api/v1/write"
Url = "http://127.0.0.1:8428/api/v1/write" #更換為vm的埠
啟動n9e服務
$ nohup ./n9e &>n9e.log &
ss -ntlp | grep 17000
LISTEN 0 1024 *:17000 *:* users:(("n9e",pid=2584479,fd=9))
配置nginx
server {
listen 80;
server_name xxxx.xxxx.com;
location / {
proxy_pass http://localhost:17000;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
瀏覽器訪問,然后輸入用戶名root,密碼root.2020即可登錄系統,
下載catgraf
wget https://download.flashcat.cloud/categraf-v0.2.38-linux-amd64.tar.gz
tar xf categraf-v0.2.38-linux-amd64.tar.gz
cd categraf-v0.2.38-linux-amd64/
vim conf/config.toml
修改這兩項
[[writers]]
url = "http://127.0.0.1:17000/prometheus/v1/write"
[heartbeat]
enable = true
啟動catgraf
nohup ./categraf &>categraf.log &
這樣完成了夜鶯精簡的中心匯聚式部署方案搭建,再通過WebUI作業系統配置-資料源來添加資料源,告警規則來告警判斷等等操作
夜鶯的V6版本架構圖和部署方式可以通過官方博客了解到,簡單來說就是n9e利用mysql存盤資料 告警資訊,配置資訊, redis存盤驗證資訊,元資料,心跳資訊,TSDB時序資料庫存盤告警指標,categraf進行資料采集,
n9e可以做集群,多個n9e分擔告警規則的處理和壓力(此時n9e有狀態服務),
最后感謝看完,由于作者水平有限,使用很多工具并不熟悉,如有錯誤和遺漏歡迎指出,感謝諒解,
參考資料:
https://flashcat.cloud/blog/sre-practice-white-paper/
https://mp.weixin.qq.com/s/5Ik-Kk1_B7jjgXLxHH1Oug
https://blog.csdn.net/m0_61323675/article/details/130114281
https://flashcat.cloud/blog/nightingale-v6-arch/
https://blog.csdn.net/wf19930209/article/details/79536506
https://www.cnblogs.com/xunzhiyou/p/16365158.html
https://blog.csdn.net/sqlquan/article/details/122093702
https://blog.csdn.net/w892824196/article/details/107062729
https://www.cnblogs.com/pxyblog/p/mysql.html
https://www.cnblogs.com/hunanzp/p/12304622.html
https://developer.aliyun.com/article/789869
https://flashcat.cloud/docs/content/flashcat-monitor/nightingale/install/victoriametrics/
https://blog.csdn.net/qihoo_tech/article/details/120558834
轉載請註明出處,本文鏈接:https://www.uj5u.com/qita/550019.html
標籤:其他
