文章目錄
- 修改特定本地路徑權限
- 啟動HDFS
- 修改HDFS特定路徑訪問權限
- 啟動Yarn
- 啟動HistoryServer
修改特定本地路徑權限
| local | $HADOOP_LOG_DIR | hdfs:hadoop | drwxrwxr-x |
|---|---|---|---|
| local | dfs.namenode.name.dir | hdfs:hadoop | drwx------ |
| local | dfs.datanode.data.dir | hdfs:hadoop | drwx------ |
| local | dfs.namenode.checkpoint.dir | hdfs:hadoop | drwx------ |
| local | yarn.nodemanager.local-dirs | yarn:hadoop | drwxrwxr-x |
| local | yarn.nodemanager.log-dirs | yarn:hadoop | drwxrwxr-x |
1）$HADOOP_LOG_DIR（所有節點）
該變數位于hadoop-env.sh檔案，默認值為 ${HADOOP_HOME}/logs
[root@hadoop102 ~]# chown hdfs:hadoop /opt/module/hadoop-3.1.3/logs/
[root@hadoop102 ~]# chmod 775 /opt/module/hadoop-3.1.3/logs/
[root@hadoop103 ~]# chown hdfs:hadoop /opt/module/hadoop-3.1.3/logs/
[root@hadoop103 ~]# chmod 775 /opt/module/hadoop-3.1.3/logs/
[root@hadoop104 ~]# chown hdfs:hadoop /opt/module/hadoop-3.1.3/logs/
[root@hadoop104 ~]# chmod 775 /opt/module/hadoop-3.1.3/logs/
2）dfs.namenode.name.dir（NameNode節點）
該引數位于hdfs-site.xml檔案，默認值為file://${hadoop.tmp.dir}/dfs/name
[root@hadoop102 ~]# chown -R hdfs:hadoop /opt/module/hadoop-3.1.3/data/dfs/name/
[root@hadoop102 ~]# chmod 700 /opt/module/hadoop-3.1.3/data/dfs/name/
3）dfs.datanode.data.dir（DataNode節點）
該引數為于hdfs-site.xml檔案，默認值為file://${hadoop.tmp.dir}/dfs/data
[root@hadoop102 ~]# chown -R hdfs:hadoop /opt/module/hadoop-3.1.3/data/dfs/data/
[root@hadoop102 ~]# chmod 700 /opt/module/hadoop-3.1.3/data/dfs/data/
[root@hadoop103 ~]# chown -R hdfs:hadoop /opt/module/hadoop-3.1.3/data/dfs/data/
[root@hadoop103 ~]# chmod 700 /opt/module/hadoop-3.1.3/data/dfs/data/
[root@hadoop104 ~]# chown -R hdfs:hadoop /opt/module/hadoop-3.1.3/data/dfs/data/
[root@hadoop104 ~]# chmod 700 /opt/module/hadoop-3.1.3/data/dfs/data/
4）dfs.namenode.checkpoint.dir（SecondaryNameNode節點）
該引數位于hdfs-site.xml檔案，默認值為file://${hadoop.tmp.dir}/dfs/namesecondary
[root@hadoop104 ~]# chown -R hdfs:hadoop /opt/module/hadoop-3.1.3/data/dfs/namesecondary/
[root@hadoop104 ~]# chmod 700 /opt/module/hadoop-3.1.3/data/dfs/namesecondary/
5）yarn.nodemanager.local-dirs（NodeManager節點）
該引數位于yarn-site.xml檔案，默認值為file://${hadoop.tmp.dir}/nm-local-dir
[root@hadoop102 ~]# chown -R yarn:hadoop /opt/module/hadoop-3.1.3/data/nm-local-dir/
[root@hadoop102 ~]# chmod -R 775 /opt/module/hadoop-3.1.3/data/nm-local-dir/
[root@hadoop103 ~]# chown -R yarn:hadoop /opt/module/hadoop-3.1.3/data/nm-local-dir/
[root@hadoop103 ~]# chmod -R 775 /opt/module/hadoop-3.1.3/data/nm-local-dir/
[root@hadoop104 ~]# chown -R yarn:hadoop /opt/module/hadoop-3.1.3/data/nm-local-dir/
[root@hadoop104 ~]# chmod -R 775 /opt/module/hadoop-3.1.3/data/nm-local-dir/
6）yarn.nodemanager.log-dirs（NodeManager節點）
該引數位于yarn-site.xml檔案，默認值為$HADOOP_LOG_DIR/userlogs
[root@hadoop102 ~]# chown yarn:hadoop /opt/module/hadoop-3.1.3/logs/userlogs/
[root@hadoop102 ~]# chmod 775 /opt/module/hadoop-3.1.3/logs/userlogs/
[root@hadoop103 ~]# chown yarn:hadoop /opt/module/hadoop-3.1.3/logs/userlogs/
[root@hadoop103 ~]# chmod 775 /opt/module/hadoop-3.1.3/logs/userlogs/
[root@hadoop104 ~]# chown yarn:hadoop /opt/module/hadoop-3.1.3/logs/userlogs/
[root@hadoop104 ~]# chmod 775 /opt/module/hadoop-3.1.3/logs/userlogs/
啟動HDFS
需要注意的是，啟動不同服務時需要使用對應的用戶
1.單點啟動
（1）啟動NameNode
[root@hadoop102 ~]# sudo -i -u hdfs hdfs --daemon start namenode
（2）啟動DataNode
[root@hadoop102 ~]# sudo -i -u hdfs hdfs --daemon start datanode
[root@hadoop103 ~]# sudo -i -u hdfs hdfs --daemon start datanode
[root@hadoop104 ~]# sudo -i -u hdfs hdfs --daemon start datanode
（3）啟動SecondaryNameNode
[root@hadoop104 ~]# sudo -i -u hdfs hdfs --daemon start secondarynamenode
說明：
? -i：重新加載環境變數
? -u：以特定用戶的身份執行后續命令
2.群起
1）在主節點（hadoop102）配置hdfs用戶到所有節點的免密登錄，
[root@hadoop102 ~]# su hdfs
[hdfs@hadoop102 root]$ ssh-keygen
[hdfs@hadoop102 root]$ ssh-copy-id hadoop102
[hdfs@hadoop102 root]$ ssh-copy-id hadoop103
[hdfs@hadoop102 root]$ ssh-copy-id hadoop104
ssh的密碼是hdfs
2）修改主節點（hadoop102）節點的$HADOOP_HOME/sbin/start-dfs.sh腳本，在頂部增加以下環境變數，
[root@hadoop102 ~]# vim $HADOOP_HOME/sbin/start-dfs.sh
在頂部增加如下內容
HDFS_DATANODE_USER=hdfs
HDFS_NAMENODE_USER=hdfs
HDFS_SECONDARYNAMENODE_USER=hdfs
注：$HADOOP_HOME/sbin/stop-dfs.sh也需在頂部增加上述環境變數才可使用，
3）以root用戶執行群起腳本，即可啟動HDFS集群，
[root@hadoop102 ~]# start-dfs.sh
到最后會出現權限問題：/etc/security/keytab/keystore
排除需要用 ：
在hadoop103和hadoop104中輸入即可
chown -R root:hadoop /etc/security/keytab/keystore
3.查看HFDS web頁面
訪問地址為https://hadoop102:9871
修改HDFS特定路徑訪問權限
| hdfs | / | hdfs:hadoop | drwxr-xr-x |
|---|---|---|---|
| hdfs | /tmp | hdfs:hadoop | drwxrwxrwxt |
| hdfs | /user | hdfs:hadoop | drwxrwxr-x |
| hdfs | yarn.nodemanager.remote-app-log-dir | yarn:hadoop | drwxrwxrwxt |
| hdfs | mapreduce.jobhistory.intermediate-done-dir | mapred:hadoop | drwxrwxrwxt |
| hdfs | mapreduce.jobhistory.done-dir | mapred:hadoop | drwxrwx— |
說明：
若上述路徑不存在，需手動創建
1）創建hdfs/hadoop主體，執行以下命令并按照提示輸入密碼（123456）
[root@hadoop102 ~]# kadmin.local -q "addprinc hdfs/hadoop"
2）認證hdfs/hadoop主體，執行以下命令并按照提示輸入密碼（123456）
[root@hadoop102 ~]# kinit hdfs/hadoop
3）按照上述要求修改指定路徑的所有者和權限
（1）修改/、/tmp、/user路徑
[root@hadoop102 ~]# hadoop fs -chown hdfs:hadoop / /tmp /user
[root@hadoop102 ~]# hadoop fs -chmod 755 /
[root@hadoop102 ~]# hadoop fs -chmod 1777 /tmp
[root@hadoop102 ~]# hadoop fs -chmod 775 /user
（2）引數yarn.nodemanager.remote-app-log-dir位于yarn-site.xml檔案，默認值/tmp/logs
[root@hadoop102 ~]# hadoop fs -chown yarn:hadoop /tmp/logs
[root@hadoop102 ~]# hadoop fs -chmod 1777 /tmp/logs
（3）引數mapreduce.jobhistory.intermediate-done-dir位于mapred-site.xml檔案，默認值為/tmp/hadoop-yarn/staging/history/done_intermediate，需保證該路徑的所有上級目錄（除/tmp）的所有者均為mapred，所屬組為hadoop，權限為770
[root@hadoop102 ~]# hadoop fs -chown -R mapred:hadoop /tmp/hadoop-yarn/staging/history/done_intermediate
[root@hadoop102 ~]# hadoop fs -chmod -R 1777 /tmp/hadoop-yarn/staging/history/done_intermediate
[root@hadoop102 ~]# hadoop fs -chown mapred:hadoop /tmp/hadoop-yarn/staging/history/
[root@hadoop102 ~]# hadoop fs -chown mapred:hadoop /tmp/hadoop-yarn/staging/
[root@hadoop102 ~]# hadoop fs -chown mapred:hadoop /tmp/hadoop-yarn/
[root@hadoop102 ~]# hadoop fs -chmod 770 /tmp/hadoop-yarn/staging/history/
[root@hadoop102 ~]# hadoop fs -chmod 770 /tmp/hadoop-yarn/staging/
[root@hadoop102 ~]# hadoop fs -chmod 770 /tmp/hadoop-yarn/
4）引數mapreduce.jobhistory.done-dir位于mapred-site.xml檔案，默認值為/tmp/hadoop-yarn/staging/history/done，需保證該路徑的所有上級目錄（除/tmp）的所有者均為mapred，所屬組為hadoop，權限為770
[root@hadoop102 ~]# hadoop fs -chown -R mapred:hadoop /tmp/hadoop-yarn/staging/history/done
[root@hadoop102 ~]# hadoop fs -chmod -R 750 /tmp/hadoop-yarn/staging/history/done
[root@hadoop102 ~]# hadoop fs -chown mapred:hadoop /tmp/hadoop-yarn/staging/history/
[root@hadoop102 ~]# hadoop fs -chown mapred:hadoop /tmp/hadoop-yarn/staging/
[root@hadoop102 ~]# hadoop fs -chown mapred:hadoop /tmp/hadoop-yarn/
[root@hadoop102 ~]# hadoop fs -chmod 770 /tmp/hadoop-yarn/staging/history/
[root@hadoop102 ~]# hadoop fs -chmod 770 /tmp/hadoop-yarn/staging/
[root@hadoop102 ~]# hadoop fs -chmod 770 /tmp/hadoop-yarn/
啟動Yarn
1，單點啟動
啟動ResourceManager
[root@hadoop103 ~]# sudo -i -u yarn yarn --daemon start resourcemanager
啟動NodeManager
[root@hadoop102 ~]# sudo -i -u yarn yarn --daemon start nodemanager
[root@hadoop103 ~]# sudo -i -u yarn yarn --daemon start nodemanager
[root@hadoop104 ~]# sudo -i -u yarn yarn --daemon start nodemanager
2，群起
1）在Yarn主節點（hadoop103）配置yarn用戶到所有節點的免密登錄，
[root@hadoop103 ~]# su yarn
[yarn@hadoop103 root]$ ssh-keygen
[yarn@hadoop103 root]$ ssh-copy-id hadoop102
[yarn@hadoop103 root]$ ssh-copy-id hadoop103
[yarn@hadoop103 root]$ ssh-copy-id hadoop104
密碼是yarn
2）修改主節點（hadoop103）的$HADOOP_HOME/sbin/start-yarn.sh，在頂部增加以下環境變數，
[root@hadoop103 ~]# vim $HADOOP_HOME/sbin/start-yarn.sh
在頂部增加如下內容
YARN_RESOURCEMANAGER_USER=yarn
YARN_NODEMANAGER_USER=yarn
注：stop-yarn.sh也需在頂部增加上述環境變數才可使用，
3）以root用戶執行$HADOOP_HOME/sbin/start-yarn.sh腳本即可啟動yarn集群，
[root@hadoop103 ~]# start-yarn.sh
3，訪問Yarn web頁面
訪問地址為http://hadoop103:8088
啟動HistoryServer
1，啟動歷史服務器
[root@hadoop102 ~]# sudo -i -u mapred mapred --daemon start historyserver
關閉歷史服務器
[root@hadoop102 ~]# sudo -i -u mapred mapred --daemon stop historyserver
2，查看歷史服務器web頁面
訪問地址為http://hadoop102:19888
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/423271.html
標籤：其他