一、概念
Hadoop是由java語言撰寫的，在分布式服務器集群上存盤海量資料并運行分布式分析應用的開源框架，其核心部件是HDFS與MapReduce，HDFS是一個分布式檔案系統，類似mogilefs，但又不同于mogilefs，hdfs由存放檔案元資料資訊的namenode和存放資料的服務器datanode組成；hdfs它不同于mogilefs，hdfs把元資料資訊放在記憶體中，而mogilefs把元資料放在資料庫中；而對于hdfs的元資料資訊持久化是依靠secondary name node（第二名稱節點），第二名稱節點并不是真正扮演名稱節點角色，它的主要任務是周期性地將編輯日志合并至名稱空間鏡像檔案中以免編輯日志變得過大；它可以獨立運行在一個物理主機上，并需要同名稱節點同樣大小的記憶體資源來完成檔案合并；另外它還保持一份名稱空間鏡像的副本，以防名稱節點掛了，丟失資料；然而根據其作業機制，第二名稱節點要滯后主節點，所以當主名稱節點掛掉以后，丟失資料是在所難免的；所以snn（secondary name node）保存鏡像副本的主要作用是盡可能的減少資料的丟失；MapReduce是一個計算框架，這種計算框架主要有兩個階段，第一階段是map計算；第二階段是Reduce計算；map計算的作用是把相同key的資料始終發送給同一個mapper進行計算；reduce就是把mapper計算的結果進行折疊計算（我們可以理解為合并），最終得到一個結果；在hadoop v1版本是這樣的架構，v2就不是了，v2版本中把mapreduce框架拆分yarn框架和mapreduce，其計算任務可以跑在yarn框架上；所以hadoop v1核心就是hdfs+mapreduce兩個集群；v2的架構就是hdfs+yarn+mapreduce；
HDFS架構
提示：從上圖架構可以看到，客戶端訪問hdfs上的某一檔案，首先要向namenode請求檔案的元資料資訊，然后nn就會告訴客戶端，訪問的檔案在datanode上的位置，然后客戶端再依次向datanode請求對應的資料，最后拼接成一個完整的檔案；這里需要注意一個概念，datanode存放檔案資料是按照檔案大小和塊大小來切分存放的，什么意思呢？比如一個檔案100M大小，假設dn（datanode）上的塊大小為10M一塊，那么它存放在dn上是把100M切分為10M一塊，共10塊，然后把這10塊資料分別存放在不同的dn上；同時這些塊分別存放在不同的dn上，還會分別在不同的dn上存在副本，這樣一來使得一個檔案的資料塊被多個dn分散冗余的存放；對于nn節點，它主要維護了那個檔案的資料存放在那些節點，和那些dn存放了那些檔案的資料塊（這個資料是通過dn周期性的向nn發送）；我們可以理解為nn內部有兩張表分別記錄了那些檔案的資料塊分別存放在那些dn上（以檔案為中心），和那些dn存放了那些檔案的資料塊（以節點為中心）；從上面的描述不難想象，當nn掛掉以后，整個存放在hdfs上的檔案都將找不到，所以在生產中我們會使用zk（zookeeper）來對nn節點做高可用；對于hdfs來講，它本質上不是內核檔案系統，所以它依賴本地Linux檔案系統；
mapreduce計算程序
提示：如上圖所示，首先mapreduce會把給定的資料切分為多個（切分之前通程序式員寫程式實作把給定的資料切分為多分，并抽取成kv鍵值對），然后啟動多個mapper對其進行map計算，多個mapper計算后的結果在通過combiner進行合并（combiner是有程式員撰寫程式實作，主要實作合并規則），把相同key的值根據某種計算規則合并在一起，然后把結果在通過partitoner（磁區器，這個磁區器是通程序式員寫程式實作，主要實作對map后的結果和對應reducer進行關聯）分別發送給不同的reducer進行計算，最終每個reducer會產生一個最終的唯一結果；簡單講mapper的作用是讀入kv鍵值對，輸出新的kv鍵值對，會有新的kv產生；combiner的作用是把當前mapper生成的新kv鍵值對進行相同key的鍵值對進行合并，至于怎么合并，合并規則是什么是由程式員定義，所以combiner就是程式員寫的程式實作，本質上combiner是讀入kv鍵值對，輸出kv鍵值對，不會產生新的kv；partitioner的作用就是把combiner合并后的鍵值對進行調度至reducer，至于怎么調度，該發往那個reducer，以及由幾個reducer進行處理，由程式員定義；最終reducer折疊計算以后生成新的kv鍵值對；
hadoop v1與v2架構
提示：在hadoop v1的架構中，所有計算任務都跑在mapreduce之上，mapreduce就主要擔任了兩個角色，第一個是集群資源管理器和資料處理；到了hadoop v2 其架構就為hdfs+yarn+一堆任務，其實我們可以把一堆任務理解為v1中的mapreduce，不同于v1中的mapreduce，v2中mapreduce只負責資料計算，不在負責集群資源管理，集群資源管理由yarn實作；對于v2來講其計算任務都跑在了執yarn之上；對于hdfs來講，v1和v2中的作用都是一樣的，都是起存盤檔案作用；
hadoop v2 計算任務資源調度程序
提示：rm（resource manager）收到客戶端的任務請求，此時rm會根據各dn上運行的nm(node manager)周期性報告的狀態資訊來決定把客戶端的任務調度給那個nm來執行；當rm選定好nm后，就把任務發送給對應nm，對應nm內部會起一個appmaster（am）的容器，負責本次任務的主控端，而appmaster需要啟動container來運行任務，它會向rm請求，然后rm會根據am的請求在對應的nm上啟動一個或多個container；最后各container運行后的結果會發送給am，然后再由am回傳給rm，rm再回傳給客戶端；在這其中rm主要用來接收個nm發送的各節點狀態資訊和資源調度以及接收各am計算任務后的結果并反饋給各客戶端；nm主要用來管理各node上的資源和上報狀態資訊給rm；am主要用來管理各任務的資源申請和各任務執行后端結果回傳給rm；
hadoop生態圈
提示：上圖是hadoop v2生態圈架構圖，其中hdfs和yarn是hadoop的核心組件，對于運行在其上的各種任務都必須依賴hadoop，也必須支持呼叫mapreduce介面；
二、hadoop集群部署
環境說明
| 名稱 | 角色 | ip |
| node01 | nn,snn,rm | 192.168.0.41 |
| node02 | dn,nm | 192.168.0.42 |
| node03 | dn,nm | 192.168.0.43 |
| node04 | dn,nm | 192.168.0.44 |
各節點同步時間
配置/etc/hosts決議個節點主機名
各節點安裝jdk
yum install -y java-1.8.0-openjdk-devel
提示：安裝devel包才會有jps命令
驗證jdk是否安裝完成，版本是否正確，確定java命令所在位置
添加JAVA_HOME環境變數
驗證JAVA_HOME變數配置是否正確
創建目錄，用于存放hadoop安裝包
mkdir /bigdata
到此基礎環境就準備OK，接下來下載hadoop二進制包
[root@node01 ~]# wget https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz --2020-09-27 22:50:16-- https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz Resolving mirror.bit.edu.cn (mirror.bit.edu.cn)... 202.204.80.77, 219.143.204.117, 2001:da8:204:1205::22 Connecting to mirror.bit.edu.cn (mirror.bit.edu.cn)|202.204.80.77|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 366447449 (349M) [application/octet-stream] Saving to: ‘hadoop-2.9.2.tar.gz’ 100%[============================================================================>] 366,447,449 1.44MB/s in 2m 19s 2020-09-27 22:52:35 (2.51 MB/s) - ‘hadoop-2.9.2.tar.gz’ saved [366447449/366447449] [root@node01 ~]# ls hadoop-2.9.2.tar.gz [root@node01 ~]#
解壓hadoop-2.9.3.tar.gz到/bigdata/目錄，并將解壓到目錄鏈接至hadoop
匯出hadoop環境變數配置
[root@node01 ~]# cat /etc/profile.d/hadoop.sh
export HADOOP_HOME=/bigdata/hadoop
export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin
export HADOOP_YARN_HOME=${HADOOP_HOME}
export HADOOP_MAPPERD_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
[root@node01 ~]#
創建hadoop用戶，并設定其密碼為admin
[root@node01 ~]# useradd hadoop [root@node01 ~]# echo "admin" |passwd --stdin hadoop Changing password for user hadoop. passwd: all authentication tokens updated successfully. [root@node01 ~]#
各節點間hadoop用戶做免密登錄
[hadoop@node01 ~]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: SHA256:6CNhqdagySJXc4iRBVSoLENddO7JLZMCsdjQzqSFnmw [email protected] The key's randomart image is: +---[RSA 2048]----+ | o*==o . | | o=Bo o | |=oX+ . | |+E =.oo.+ | |o.o B.oBS. | |.o * =. o | |=.+ o o | |oo . . | | | +----[SHA256]-----+ [hadoop@node01 ~]$ ssh-copy-id node01 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub" The authenticity of host 'node01 (192.168.0.41)' can't be established. ECDSA key fingerprint is SHA256:lE8/Vyni4z8hsXaa8OMMlDpu3yOIRh6dLcIr+oE57oE. ECDSA key fingerprint is MD5:14:59:02:30:c0:16:b8:6c:1a:84:c3:0f:a7:ac:67:b3. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hadoop@node01's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'node01'" and check to make sure that only the key(s) you wanted were added. [hadoop@node01 ~]$ scp -r ./.ssh node02:/home/hadoop/ The authenticity of host 'node02 (192.168.0.42)' can't be established. ECDSA key fingerprint is SHA256:lE8/Vyni4z8hsXaa8OMMlDpu3yOIRh6dLcIr+oE57oE. ECDSA key fingerprint is MD5:14:59:02:30:c0:16:b8:6c:1a:84:c3:0f:a7:ac:67:b3. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node02,192.168.0.42' (ECDSA) to the list of known hosts. hadoop@node02's password: id_rsa 100% 1679 636.9KB/s 00:00 id_rsa.pub 100% 404 186.3KB/s 00:00 known_hosts 100% 362 153.4KB/s 00:00 authorized_keys 100% 404 203.9KB/s 00:00 [hadoop@node01 ~]$ scp -r ./.ssh node03:/home/hadoop/ The authenticity of host 'node03 (192.168.0.43)' can't be established. ECDSA key fingerprint is SHA256:lE8/Vyni4z8hsXaa8OMMlDpu3yOIRh6dLcIr+oE57oE. ECDSA key fingerprint is MD5:14:59:02:30:c0:16:b8:6c:1a:84:c3:0f:a7:ac:67:b3. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node03,192.168.0.43' (ECDSA) to the list of known hosts. hadoop@node03's password: id_rsa 100% 1679 755.1KB/s 00:00 id_rsa.pub 100% 404 165.7KB/s 00:00 known_hosts 100% 543 350.9KB/s 00:00 authorized_keys 100% 404 330.0KB/s 00:00 [hadoop@node01 ~]$ scp -r ./.ssh node04:/home/hadoop/ The authenticity of host 'node04 (192.168.0.44)' can't be established. ECDSA key fingerprint is SHA256:lE8/Vyni4z8hsXaa8OMMlDpu3yOIRh6dLcIr+oE57oE. ECDSA key fingerprint is MD5:14:59:02:30:c0:16:b8:6c:1a:84:c3:0f:a7:ac:67:b3. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node04,192.168.0.44' (ECDSA) to the list of known hosts. hadoop@node04's password: id_rsa 100% 1679 707.0KB/s 00:00 id_rsa.pub 100% 404 172.8KB/s 00:00 known_hosts 100% 724 437.7KB/s 00:00 authorized_keys 100% 404 165.2KB/s 00:00 [hadoop@node01 ~]$
驗證：用node01去連接node02,node03,node04看看是否是免密登錄了
創建資料目錄/data/hadoop/hdfs/{nn,snn,dn},并將其屬主屬組更改為hadoop
進入到hadoop安裝目錄，創建其logs目錄，并將其安裝目錄的屬主和屬組更改為hadoop
提示：以上所有步驟都需要在各節點挨著做一遍；
配置hadoop的core-site.xml
提示：hadoop的組態檔語法都是xml格式的組態檔，其中<property>和</property>是一對標簽，里面用name標簽來參考配置的選項的key的名稱，其value標簽用來配置對應key的值；上面配置表示配置默認的檔案系統地址；hdfs://node01:8020是hdfs檔案系統訪問的地址；
完整的配置
[root@node01 hadoop]# cat core-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href=https://www.cnblogs.com/qiuhom-1874/archive/2020/09/28/"configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://node01:8020</value> <final>true</final> </property> </configuration> [root@node01 hadoop]#View Code
配置hdfs-site.xml
提示：以上配置主要指定hdfs相關目錄以及訪問web埠資訊，副本數量；
完整的配置
[root@node01 hadoop]# cat hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href=https://www.cnblogs.com/qiuhom-1874/archive/2020/09/28/"configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:///data/hadoop/hdfs/nn</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>node01:50090</value> </property> <property> <name>dfs.namenode.http-address</name> <value>node01:50070</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///data/hadoop/hdfs/dn</value> </property> <property> <name>fs.checkpoint.dir</name> <value>file:///data/hadoop/hdfs/snn</value> </property> <property> <name>fs.checkpoint.edits.dir</name> <value>file:///data/hadoop/hdfs/snn</value> </property> </configuration> [root@node01 hadoop]#View Code
配置mapred-site.xml
提示：以上配置主要指定了mapreduce的框架為yarn;默認沒有mapred-site.xml，我們需要將mapred-site.xml.template修改成mapred.site.xml；這里需要注意我上面是通過復制修改檔案名，當然屬主資訊都會變成root，不要忘記把屬組資訊修改成hadoop;
完整的配置
[root@node01 hadoop]# cat mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href=https://www.cnblogs.com/qiuhom-1874/archive/2020/09/28/"configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> [root@node01 hadoop]#View Code
配置yarn-site.xml
提示：以上配置主要配置了yarn框架rm和nm相關地址和指定相關類；
完整的配置
[root@node01 hadoop]# cat yarn-site.xml <?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <property> <name>yarn.resourcemanager.address</name> <value>node01:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>node01:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>node01:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>node01:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>node01:8088</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.auxservices.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> </property> </configuration> [root@node01 hadoop]#View Code
配置slave.xml
[root@node01 hadoop]# cat slaves node02 node03 node04 [root@node01 hadoop]#
復制各組態檔到其他節點
到此hadoop配置就完成了；
接下來切換到hadoop用戶下，初始化hdfs
hdfs namenode -format
提示：如果執行hdfs namenode -format 出現紅框中的提示，說明hdfs格式化就成功了；
啟動hdfs集群
提示：hdfs主要由namenode、secondarynamenode和datanode組成，只要看到對應節點上的行程啟動起來，就沒有多大問題；
到此hdfs集群就正常啟動了
驗證：把/etc/passwd上傳到hdfs的/test目錄下，看看是否可以正常上傳？
提示：可以看到/etc/passwd檔案已經上傳至hdfs的/test目錄下了；
驗證：查看hdfs /test目錄下passwd檔案，看看是否同/etc/passwd檔案內容相同？
提示：可以看到hdfs上的/test/passwd檔案內容同/etc/passwd檔案內容相同；
驗證：在dn節點查看對應目錄下的檔案內容，看看是否同/etc/passwd檔案內容相同？
[root@node02 ~]# tree /data
/data
└── hadoop
└── hdfs
├── dn
│ ├── current
│ │ ├── BP-157891879-192.168.0.41-1601224158145
│ │ │ ├── current
│ │ │ │ ├── finalized
│ │ │ │ │ └── subdir0
│ │ │ │ │ └── subdir0
│ │ │ │ │ ├── blk_1073741825
│ │ │ │ │ └── blk_1073741825_1001.meta
│ │ │ │ ├── rbw
│ │ │ │ └── VERSION
│ │ │ ├── scanner.cursor
│ │ │ └── tmp
│ │ └── VERSION
│ └── in_use.lock
├── nn
└── snn
13 directories, 6 files
[root@node02 ~]# cat /data/hadoop/hdfs/dn/current/BP-157891879-192.168.0.41-1601224158145/
current/ scanner.cursor tmp/
[root@node02 ~]# cat /data/hadoop/hdfs/dn/current/BP-157891879-192.168.0.41-1601224158145/current/finalized/subdir0/subdir0/blk_1073741825
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:997:User for polkitd:/:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
chrony:x:998:996::/var/lib/chrony:/sbin/nologin
hadoop:x:1000:1000::/home/hadoop:/bin/bash
[root@node02 ~]#
提示：可以看到在dn節點上的dn目錄下能夠找到我們上傳的passwd檔案；
驗證：查看其它節點是否有相同的檔案？是否有我們指定數量的副本？
提示：在node03和node04上也有相同的目錄和檔案；說明我們設定的副本數量為3生效了；
啟動yarn集群
提示：可以看到對應節點上的nm啟動了；主節點上的rm也正常啟動了；
訪問nn的50070和8088，看看對應的web地址是否能夠訪問到頁面？
提示：這個地址是hdfs的web地址，在這個界面可以看到hdfs的存盤狀況，以及對hdfs上的檔案做操作；
提示：8088是yarn集群的管理地址；在這個界面上能夠看到運行的計算任務的狀態資訊，集群配置資訊，日志等等；
驗證：在yarn上跑一個計算任務，統計/test/passwd檔案的單詞數量，看看對應的計算任務是否能夠跑起來？
[hadoop@node01 hadoop]$ yarn jar /bigdata/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
wordmean: A map/reduce program that counts the average length of the words in the input files.
wordmedian: A map/reduce program that counts the median length of the words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
[hadoop@node01 hadoop]$ yarn jar /bigdata/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar wordcount
Usage: wordcount <in> [<in>...] <out>
[hadoop@node01 hadoop]$ yarn jar /bigdata/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar wordcount /test/passwd /test/passwd-word-count20/09/28 00:58:01 INFO client.RMProxy: Connecting to ResourceManager at node01/192.168.0.41:8032
20/09/28 00:58:01 INFO input.FileInputFormat: Total input files to process : 1
20/09/28 00:58:01 INFO mapreduce.JobSubmitter: number of splits:1
20/09/28 00:58:01 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
20/09/28 00:58:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1601224871685_0001
20/09/28 00:58:02 INFO impl.YarnClientImpl: Submitted application application_1601224871685_0001
20/09/28 00:58:02 INFO mapreduce.Job: The url to track the job: http://node01:8088/proxy/application_1601224871685_0001/
20/09/28 00:58:02 INFO mapreduce.Job: Running job: job_1601224871685_0001
20/09/28 00:58:08 INFO mapreduce.Job: Job job_1601224871685_0001 running in uber mode : false
20/09/28 00:58:08 INFO mapreduce.Job: map 0% reduce 0%
20/09/28 00:58:14 INFO mapreduce.Job: map 100% reduce 0%
20/09/28 00:58:20 INFO mapreduce.Job: map 100% reduce 100%
20/09/28 00:58:20 INFO mapreduce.Job: Job job_1601224871685_0001 completed successfully
20/09/28 00:58:20 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1144
FILE: Number of bytes written=399079
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1053
HDFS: Number of bytes written=1018
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2753
Total time spent by all reduces in occupied slots (ms)=2779
Total time spent by all map tasks (ms)=2753
Total time spent by all reduce tasks (ms)=2779
Total vcore-milliseconds taken by all map tasks=2753
Total vcore-milliseconds taken by all reduce tasks=2779
Total megabyte-milliseconds taken by all map tasks=2819072
Total megabyte-milliseconds taken by all reduce tasks=2845696
Map-Reduce Framework
Map input records=22
Map output records=30
Map output bytes=1078
Map output materialized bytes=1144
Input split bytes=95
Combine input records=30
Combine output records=30
Reduce input groups=30
Reduce shuffle bytes=1144
Reduce input records=30
Reduce output records=30
Spilled Records=60
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=87
CPU time spent (ms)=620
Physical memory (bytes) snapshot=444997632
Virtual memory (bytes) snapshot=4242403328
Total committed heap usage (bytes)=285212672
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=958
File Output Format Counters
Bytes Written=1018
[hadoop@node01 hadoop]$
查看計算后生成的報告
[hadoop@node01 hadoop]$ hdfs dfs -ls -R /test -rw-r--r-- 3 hadoop supergroup 958 2020-09-28 00:32 /test/passwd drwxr-xr-x - hadoop supergroup 0 2020-09-28 00:58 /test/passwd-word-count -rw-r--r-- 3 hadoop supergroup 0 2020-09-28 00:58 /test/passwd-word-count/_SUCCESS -rw-r--r-- 3 hadoop supergroup 1018 2020-09-28 00:58 /test/passwd-word-count/part-r-00000 [hadoop@node01 hadoop]$ hdfs dfs -cat /test/passwd-word-count/part-r-00000 Management:/:/sbin/nologin 1 Network 1 SSH:/var/empty/sshd:/sbin/nologin 1 User:/var/ftp:/sbin/nologin 1 adm:x:3:4:adm:/var/adm:/sbin/nologin 1 bin:x:1:1:bin:/bin:/sbin/nologin 1 bus:/:/sbin/nologin 1 chrony:x:998:996::/var/lib/chrony:/sbin/nologin 1 daemon:x:2:2:daemon:/sbin:/sbin/nologin 1 dbus:x:81:81:System 1 for 1 ftp:x:14:50:FTP 1 games:x:12:100:games:/usr/games:/sbin/nologin 1 hadoop:x:1000:1000::/home/hadoop:/bin/bash 1 halt:x:7:0:halt:/sbin:/sbin/halt 1 lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin 1 mail:x:8:12:mail:/var/spool/mail:/sbin/nologin 1 message 1 nobody:x:99:99:Nobody:/:/sbin/nologin 1 ntp:x:38:38::/etc/ntp:/sbin/nologin 1 operator:x:11:0:operator:/root:/sbin/nologin 1 polkitd:/:/sbin/nologin 1 polkitd:x:999:997:User 1 postfix:x:89:89::/var/spool/postfix:/sbin/nologin 1 root:x:0:0:root:/root:/bin/bash 1 shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown 1 sshd:x:74:74:Privilege-separated 1 sync:x:5:0:sync:/sbin:/bin/sync 1 systemd-network:x:192:192:systemd 1 tcpdump:x:72:72::/:/sbin/nologin 1 [hadoop@node01 hadoop]$
在8088頁面上查看任務的狀態資訊
到此hadoop v2集群就搭建完畢了；
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/135021.html
標籤：其他