學習 SUSE Storage 系列文章
(1)SUSE Storage6 實驗環境搭建詳細步驟 - Win10 + VMware WorkStation
(2)SUSE Linux Enterprise 15 SP1 系統安裝
(3)SUSE Ceph 快速部署 - Storage6
(4)SUSE Ceph 增加節點、減少節點、 洗掉OSD磁盤等操作 - Storage6
(5)深入理解 DeepSea 和 Salt 部署工具 - Storage6
一、安裝環境描述
整個環境采用VMware workstation搭建,一共5臺虛擬機,所有虛擬機安裝SLES15SP1系統,其中一臺安裝apache作為倉庫使用,剩余4臺用于搭建Storage6集群,
(1)硬體環境:
- 筆記本一臺,CPU、記憶體和磁盤空間足夠
- 筆記本上建立一塊環回介面用于分布式存盤管理網路
(2)軟體環境:
- 筆記本安裝 Win10 作業系統
- 虛擬環境:VMware Workstation 14 Pro
使用VMnet2和VMnet3 作為分布式存盤 public和cluster網路
- 虛擬機作業系統:SLES15SP1 企業版作業系統
1、網路
主機名 public網路 管理網路 集群網路 描述 smt 172.200.50.19 SUSE倉庫admin 192.168.2.39 172.200.50.39 192.168.3.39 管理主機node001 192.168.2.41 172.200.50.41 192.168.3.41 MONnode002 192.168.2.42 172.200.50.42 192.168.3.42 MONnode003 192.168.2.43 172.200.50.43 192.168.3.43 MON
2、磁盤
每個節點系統有2塊 OSD 盤和1塊 NVME 磁盤
1 # lsblk 2 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT 3 sda 8:0 0 20G 0 disk # 作業系統盤 4 ├─sda1 8:1 0 1G 0 part /boot 5 └─sda2 8:2 0 19G 0 part 6 ├─vgoo-lvroot 254:0 0 17G 0 lvm / 7 └─vgoo-lvswap 254:1 0 2G 0 lvm [SWAP] 8 sdb 8:16 0 10G 0 disk # osd 資料盤 9 sdc 8:32 0 10G 0 disk # osd 資料盤10 nvme0n1 259:0 0 20G 0 disk # wal db
二、作業系統初始化安裝
1、臨時IP地址設定
1 ip link set eth0 up2 ip addr add 172.200.50.50/24 dev eth0
連接上了設定永久地址
yast lan list
yast lan edit id=0 ip=192.168.2.40 netmask=255.255.255.0
2、設定bash環境變數和別名
# vim /root/.bash_profile alias cd..='cd ..'alias dir='ls -l'alias egrep='egrep --color=auto'alias fgrep='fgrep --color=auto'alias grep='grep --color=auto'alias l='ls -alF'alias la='ls -la'alias ll='ls -l'alias ls-l='ls -l'
3、配置after.local檔案
touch /etc/init.d/after.localchmod 744 /etc/init.d/after.local
復制該內容進去
#! /bin/sh## Copyright (c) 2010 SuSE LINUX Products GmbH, Germany. All rights reserved.## Author: Werner Fink, 2010## /etc/init.d/after.local## script with local commands to be executed from init after all scripts# of a runlevel have been executed.## Here you should add things, that should happen directly after# runlevel has been reached.#
4、倉庫配置(所有節點和admin)
## Poolzypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Product-SLES/15-SP1/x86_64/product/ SLE-Product-SLES15-SP1-Poolzypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Basesystem/15-SP1/x86_64/product/ SLE-Module-Basesystem-SLES15-SP1-Poolzypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Server-Applications/15-SP1/x86_64/product/ SLE-Module-Server-Applications-SLES15-SP1-Poolzypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Legacy/15-SP1/x86_64/product/ SLE-Module-Legacy-SLES15-SP1-Poolzypper ar http://172.200.50.19/repo/SUSE/Products/Storage/6/x86_64/product/ SUSE-Enterprise-Storage-6-Pool## Updatezypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Product-SLES/15-SP1/x86_64/update/ SLE-Product-SLES15-SP1-Updateszypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Basesystem/15-SP1/x86_64/update/ SLE-Module-Basesystem-SLES15-SP1-Upadateszypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Server-Applications/15-SP1/x86_64/update/ SLE-Module-Server-Applications-SLES15-SP1-Upadateszypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Legacy/15-SP1/x86_64/update/ SLE-Module-Legacy-SLES15-SP1-Updateszypper ar http://172.200.50.19/repo/SUSE/Updates/Storage/6/x86_64/update/ SUSE-Enterprise-Storage-6-Updates
# zypper lr# | Alias | Name ---+----------------------------------------------------+---------------------------------------------------- 1 | SLE-Module-Basesystem-SLES15-SP1-Pool | SLE-Module-Basesystem-SLES15-SP1-Pool 2 | SLE-Module-Basesystem-SLES15-SP1-Upadates | SLE-Module-Basesystem-SLES15-SP1-Upadates 3 | SLE-Module-Legacy-SLES15-SP1-Pool | SLE-Module-Legacy-SLES15-SP1-Pool 4 | SLE-Module-Legacy-SLES15-SP1-Updates | SLE-Module-Legacy-SLES15-SP1-Updates 5 | SLE-Module-Server-Applications-SLES15-SP1-Pool | SLE-Module-Server-Applications-SLES15-SP1-Pool 6 | SLE-Module-Server-Applications-SLES15-SP1-Upadates | SLE-Module-Server-Applications-SLES15-SP1-Upadates 7 | SLE-Product-SLES15-SP1-Pool | SLE-Product-SLES15-SP1-Pool 8 | SLE-Product-SLES15-SP1-Updates | SLE-Product-SLES15-SP1-Updates 9 | SUSE-Enterprise-Storage-6-Pool | SUSE-Enterprise-Storage-6-Pool 10 | SUSE-Enterprise-Storage-6-Updates | SUSE-Enterprise-Storage-6-Updates
5、安裝基本軟體 (所有節點和admin)
zypper in -y -t pattern yast2_basis basezypper in -y net-tools vim man sudo tuned irqbalancezypper in -y ethtool rsyslog iputils less supportutils-plugin-seszypper in -y net-tools-deprecated tree wget
6、關閉IPV6 (所有節點和admin)
# vim /etc/sysctl.confnet.ipv6.conf.all.disable_ipv6 = 1 # 關閉 IPV6net.ipv6.conf.default.disable_ipv6 = 1net.ipv6.conf.lo.disable_ipv6 = 1vm.min_free_kbytes = 2097152 # 128GB的RAM,系統預留2GBkernel.pid_max = 4194303 # 執行緒數設定最大
執行生效
# sysctl -p
7、調整網路優化引數 (所有節點和admin)
# tuned-adm profile throughput-performance# tuned-adm active# systemctl start tuned.service# systemctl enable tuned.service
8、編輯hosts檔案 (所有節點和admin)
# vim /etc/hosts192.168.2.39 admin.example.com admin192.168.2.40 node001.example.com node001192.168.2.41 node002.example.com node002192.168.2.42 node003.example.com node003
9、執行更新作業系統,并重啟 (所有節點和admin)
# zypper ref# zypper -n update# reboot
三、安裝Storage6集群
1、安裝 salt(admin節點)
zypper -n in deepseasystemctl restart salt-master.servicesystemctl enable salt-master.servicesystemctl status salt-master.service
osd節點 和 admin節點
zypper -n in salt-minionsed -i '17i\master: 192.168.2.39' /etc/salt/minionsystemctl restart salt-minion.servicesystemctl enable salt-minion.servicesystemctl status salt-minion.service
接受所有請求(admin節點)
salt-keysalt-key --accept-allsalt-keysalt '*' test.ping
2、Admin節點,配置NTP服務
如果沒有ntp服務器,默認以admin節點為ntp server
Admin節點:
# vim /etc/chrony.conf# Sync to local clock # 添加本地時鐘源server 127.0.0.1 allow 127.0.0.0/8allow 192.168.2.0/24allow 172.200.50.0/24local stratum 10
systemctl restart chronyd.servicesystemctl enable chronyd.servicesystemctl status chronyd.service
# chronyc sources 210 Number of sources = 1MS Name/IP address Stratum Poll Reach LastRx Last sample ===============================================================================^* 127.127.1.0 12 6 37 23 +1461ns[+3422ns] +/- 166us
# chronyc -n sources -v
3、修改組(admin節點)
cp -p /srv/pillar/ceph/deepsea_minions.sls /tmp/sed -i "4c # deepsea_minions: 'G@deepsea:*'" /srv/pillar/ceph/deepsea_minions.slssed -i "6c deepsea_minions: '*'" /srv/pillar/ceph/deepsea_minions.sls
4、遠程連接終端監控(admin節點)
該程式監控提供一個詳細的,實時的可視化操作行為,當在執行運行salt-run state.orch時,監控執行期間運行了什么
# deepsea monitor

5、更新補丁,并收集硬體資訊(admin節點)
salt-run state.orch ceph.stage.0salt-run state.orch ceph.stage.1
報錯資訊可忽略
No minions matched the target. No command was sent, no jid was assigned.No minions matched the target. No command was sent, no jid was assigned.[ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure[ERROR ] Exception during resolving address: [Errno 2] Host name lookup failure[WARNING ] /usr/lib/python3.6/site-packages/salt/grains/core.py:2827: DeprecationWarning: This server_id iscomputed nor by Adler32 neither by CRC32. Please use "server_id_use_crc" option and define algorithm youprefer (default "Adler32"). The server_id will be computed withAdler32 by default.
GitHub??
https://github.com/SUSE/DeepSea/issues/1593
6、查看網路組態檔(admin節點)
# vim /srv/pillar/ceph/proposals/config/stack/default/ceph/cluster.yml cluster_network: 192.168.3.0/24 fsid: 10aca2da-ead5-438d-b104-da37870b50b8 public_network: 192.168.2.0/24
7、配置集群policy.cfg檔案(admin節點)
(1)模板檔案,復制policy.cfg-rolebased模板
# ll /usr/share/doc/packages/deepsea/examples/total 12-rw-r--r-- 1 root root 329 Jun 13 16:00 policy.cfg-generic-rw-r--r-- 1 root root 489 Jun 13 16:00 policy.cfg-regex-rw-r--r-- 1 root root 577 Jun 13 16:00 policy.cfg-rolebased
# cp /usr/share/doc/packages/deepsea/examples/policy.cfg-rolebased /srv/pillar/ceph/proposals/policy.cfg (2)編輯模板檔案(admin節點)
# vim /srv/pillar/ceph/proposals/policy.cfg## Cluster Assignmentcluster-ceph/cluster/*.sls## Roles# ADMIN role-master/cluster/admin*.slsrole-admin/cluster/admin*.sls# Monitoringrole-prometheus/cluster/admin*.slsrole-grafana/cluster/admin*.sls# MONrole-mon/cluster/node00[1-3]*.sls# MGR (mgrs are usually colocated with mons)role-mgr/cluster/node00[1-3]*.sls# COMMONconfig/stack/default/global.ymlconfig/stack/default/ceph/cluster.yml# Storage # 定義為 storage 角色role-storage/cluster/node00*.sls
(3)執行stage2命令 (admin節點)
# salt-run state.orch ceph.stage.2# salt '*' pillar.items # 查看設定是否正確
尤其是NTP,role 角色定義,public network 網路 是否定義正確
public_network: 192.168.2.0/24 roles: - mon - mgr - storage time_server: admin.example.com
(4)如果3個節點需要修改 (admin節點)
由于測驗環境,只用到3臺OSD節點,官方建議生產環境必須是4臺節點或以上
# sed -i 's/if (not self.in_dev_env and len(storage) < 4/if (not self.in_dev_env and len(storage) < 2/g' /srv/modules/runners/validate.py
8、定義和創建 OSD 磁盤
(1)備份組態檔
# cp /srv/salt/ceph/configuration/files/drive_groups.yml /srv/salt/ceph/configuration/files/drive_groups.yml.bak(2)查看OSD節點磁盤情況(node001,node002,node003)
# ceph-volume inventory stderr: blkid: error: /dev/sr0: No medium foundDevice Path Size rotates available Model name/dev/nvme0n1 20.00 GB False True VMware Virtual NVMe Disk/dev/sdb 10.00 GB True True VMware Virtual S/dev/sdc 10.00 GB True True VMware Virtual S/dev/sda 20.00 GB True False VMware Virtual S/dev/sr0 1024.00 MB True False VMware SATA CD01
(3)編輯組態檔
# vim /srv/salt/ceph/configuration/files/drive_groups.ymldrive_group_hdd_nvme: # 目標為 storage角色節點 target: 'I@roles:storage' data_devices: size: '9GB:12GB' # 資料設備按照磁盤大小來區分,9G到12G之間 db_devices: rotational: 0 # 非機械設備 SSD or NVME block_db_size: '2G' # 指定 db大小為2GB (大小按實際情況)
(4)顯示OSD配置報告
可以清楚的看到一塊nvme0n1作為BlueStore的DB設備,2G大小,分割成2個LV對應2塊OSD資料磁盤,
# salt-run disks.report node003.example.com: |_ - 0 - Total OSDs: 2 Solid State VG: Targets: block.db Total size: 19.00 GB Total LVs: 2 Size per LV: 1.86 GB Devices: /dev/nvme0n1 Type Path LV Size % of device ---------------------------------------------------------------- [data] /dev/sdb 9.00 GB 100.0% [block.db] vg: vg/lv 1.86 GB 10% ---------------------------------------------------------------- [data] /dev/sdc 9.00 GB 100.0% [block.db] vg: vg/lv 1.86 GB 10%
注意:如果磁盤無法識別請使用如下命令格式化,不能有GPT磁區
# ceph-volume lvm zap /dev/xx
# ceph-volume lvm zap /dev/xx --destroy
(5)運行stage3 stage4
# salt-run state.orch ceph.stage.3# salt-run state.orch ceph.stage.4
四、配置HAproxy和Dashboard
1、添加SUSE HA倉庫、安裝 Haproxy (admin節點)
# zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Product-HA/15-SP1/x86_64/product/ SLE-Products-HA-SLES15-SP1-Pool# zypper -n in haproxy
2、配置
# vim /etc/haproxy/haproxy.cfg ……frontend http_web option tcplog bind 0.0.0.0:8443 # 定義bind系結,監聽那個套接字,如果是node001作為admin節點,改成9443 mode tcp default_backend dashboardbackend dashboard mode tcp option log-health-checks option httpchk GET / http-check expect status 200 server mgr1 172.200.50.40:8443 check ssl verify none server mgr2 172.200.50.41:8443 check ssl verify none server mgr3 172.200.50.42:8443 check ssl verify none
3) 啟動haproxy服務
# systemctl start haproxy.service# systemctl enable haproxy.service# systemctl status haproxy.service4) 查看dashboard管理員密碼:
# salt-call grains.get dashboard_credslocal: ---------- admin: 9KyIXZSrdW
5)windows主機添加域名決議
C:\Windows\System32\drivers\etc\host127.0.0.1 localhost172.200.50.39 admin.example.com
6)訪問SES6 Dashboard頁面
http://172.200.50.39:8443/#/dashboard



轉載請註明出處,本文鏈接:https://www.uj5u.com/qita/63707.html
標籤:其他
上一篇:maya python命令
