???突然想學大資料了，加上上課老師把大資料說的天花亂墜，我還沒聽懂😟，于是只能課下開小灶，大致了解一下大資料的一些基本概念和名詞，
1.大資料的概念
大資料：是指無法在一定時間范圍內用常規軟體工具進行捕捉、管理和處理的資料集合，是需要新的處理模式才能具有更強的決策力、洞察發現力和流程優化能力的海量、高增長率和多樣化的資訊資產，一般來說就是指存盤資料在TB、PB、EB量級的資料，
主要解決海量資料的采集、存盤和分析計算的問題， 資料量大 產生速度快 資料型別多樣：結構化（資料庫/文本）和非結構化資料（網路日志、音頻、視頻、圖片、地理位置等） 密度大，價值低，即價值密度的高低和資料量大小成反比， 2.Hadoop介紹
Hadoop是一個由Apache基金會所開發的分布式系統基礎架構， 主要解決海量資料的存盤 和海量資料的分析計算 的問題， 廣義上來說，Hadoop 通常是指一個更廣泛的概念——Hadoop 生態圈， 3.Hadoop的版本
hadoop的三大發行版本：Apache、Cloudera、Hortonworks，
Apache版本最原始（最基礎），對入門學習最好， Cloudera內部集成了很多大資料框架，對應產品CDH， Hortonworks檔案較好，對應產品HDP ， Hortonwork和Cloudera合并 3.Hadoop的特點
高可靠性：Hadoop底層維護多個資料副本，即使某個計算單元存盤出現故障，也不會導致資料丟失，
<style>#mermaid-svg-Dea3eFcIXASpWujp .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-Dea3eFcIXASpWujp .label text{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .node rect,#mermaid-svg-Dea3eFcIXASpWujp .node circle,#mermaid-svg-Dea3eFcIXASpWujp .node ellipse,#mermaid-svg-Dea3eFcIXASpWujp .node polygon,#mermaid-svg-Dea3eFcIXASpWujp .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-Dea3eFcIXASpWujp .node .label{text-align:center;fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .node.clickable{cursor:pointer}#mermaid-svg-Dea3eFcIXASpWujp .arrowheadPath{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-Dea3eFcIXASpWujp .flowchart-link{stroke:#333;fill:none}#mermaid-svg-Dea3eFcIXASpWujp .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-Dea3eFcIXASpWujp .edgeLabel rect{opacity:0.9}#mermaid-svg-Dea3eFcIXASpWujp .edgeLabel span{color:#333}#mermaid-svg-Dea3eFcIXASpWujp .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-Dea3eFcIXASpWujp .cluster text{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-Dea3eFcIXASpWujp .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-Dea3eFcIXASpWujp text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-Dea3eFcIXASpWujp .actor-line{stroke:grey}#mermaid-svg-Dea3eFcIXASpWujp .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-Dea3eFcIXASpWujp .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-Dea3eFcIXASpWujp #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-Dea3eFcIXASpWujp .sequenceNumber{fill:#fff}#mermaid-svg-Dea3eFcIXASpWujp #sequencenumber{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp #crosshead path{fill:#333;stroke:#333}#mermaid-svg-Dea3eFcIXASpWujp .messageText{fill:#333;stroke:#333}#mermaid-svg-Dea3eFcIXASpWujp .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-Dea3eFcIXASpWujp .labelText,#mermaid-svg-Dea3eFcIXASpWujp .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-Dea3eFcIXASpWujp .loopText,#mermaid-svg-Dea3eFcIXASpWujp .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-Dea3eFcIXASpWujp .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-Dea3eFcIXASpWujp .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-Dea3eFcIXASpWujp .noteText,#mermaid-svg-Dea3eFcIXASpWujp .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-Dea3eFcIXASpWujp .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-Dea3eFcIXASpWujp .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-Dea3eFcIXASpWujp .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-Dea3eFcIXASpWujp .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp .section{stroke:none;opacity:0.2}#mermaid-svg-Dea3eFcIXASpWujp .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-Dea3eFcIXASpWujp .section2{fill:#fff400}#mermaid-svg-Dea3eFcIXASpWujp .section1,#mermaid-svg-Dea3eFcIXASpWujp .section3{fill:#fff;opacity:0.2}#mermaid-svg-Dea3eFcIXASpWujp .sectionTitle0{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .sectionTitle1{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .sectionTitle2{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .sectionTitle3{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-Dea3eFcIXASpWujp .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp .grid path{stroke-width:0}#mermaid-svg-Dea3eFcIXASpWujp .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-Dea3eFcIXASpWujp .task{stroke-width:2}#mermaid-svg-Dea3eFcIXASpWujp .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp .taskText:not([font-size]){font-size:11px}#mermaid-svg-Dea3eFcIXASpWujp .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-Dea3eFcIXASpWujp .task.clickable{cursor:pointer}#mermaid-svg-Dea3eFcIXASpWujp .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-Dea3eFcIXASpWujp .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-Dea3eFcIXASpWujp .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-Dea3eFcIXASpWujp .taskText0,#mermaid-svg-Dea3eFcIXASpWujp .taskText1,#mermaid-svg-Dea3eFcIXASpWujp .taskText2,#mermaid-svg-Dea3eFcIXASpWujp .taskText3{fill:#fff}#mermaid-svg-Dea3eFcIXASpWujp .task0,#mermaid-svg-Dea3eFcIXASpWujp .task1,#mermaid-svg-Dea3eFcIXASpWujp .task2,#mermaid-svg-Dea3eFcIXASpWujp .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-Dea3eFcIXASpWujp .taskTextOutside0,#mermaid-svg-Dea3eFcIXASpWujp .taskTextOutside2{fill:#000}#mermaid-svg-Dea3eFcIXASpWujp .taskTextOutside1,#mermaid-svg-Dea3eFcIXASpWujp .taskTextOutside3{fill:#000}#mermaid-svg-Dea3eFcIXASpWujp .active0,#mermaid-svg-Dea3eFcIXASpWujp .active1,#mermaid-svg-Dea3eFcIXASpWujp .active2,#mermaid-svg-Dea3eFcIXASpWujp .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-Dea3eFcIXASpWujp .activeText0,#mermaid-svg-Dea3eFcIXASpWujp .activeText1,#mermaid-svg-Dea3eFcIXASpWujp .activeText2,#mermaid-svg-Dea3eFcIXASpWujp .activeText3{fill:#000 !important}#mermaid-svg-Dea3eFcIXASpWujp .done0,#mermaid-svg-Dea3eFcIXASpWujp .done1,#mermaid-svg-Dea3eFcIXASpWujp .done2,#mermaid-svg-Dea3eFcIXASpWujp .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-Dea3eFcIXASpWujp .doneText0,#mermaid-svg-Dea3eFcIXASpWujp .doneText1,#mermaid-svg-Dea3eFcIXASpWujp .doneText2,#mermaid-svg-Dea3eFcIXASpWujp .doneText3{fill:#000 !important}#mermaid-svg-Dea3eFcIXASpWujp .crit0,#mermaid-svg-Dea3eFcIXASpWujp .crit1,#mermaid-svg-Dea3eFcIXASpWujp .crit2,#mermaid-svg-Dea3eFcIXASpWujp .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-Dea3eFcIXASpWujp .activeCrit0,#mermaid-svg-Dea3eFcIXASpWujp .activeCrit1,#mermaid-svg-Dea3eFcIXASpWujp .activeCrit2,#mermaid-svg-Dea3eFcIXASpWujp .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-Dea3eFcIXASpWujp .doneCrit0,#mermaid-svg-Dea3eFcIXASpWujp .doneCrit1,#mermaid-svg-Dea3eFcIXASpWujp .doneCrit2,#mermaid-svg-Dea3eFcIXASpWujp .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-Dea3eFcIXASpWujp .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-Dea3eFcIXASpWujp .milestoneText{font-style:italic}#mermaid-svg-Dea3eFcIXASpWujp .doneCritText0,#mermaid-svg-Dea3eFcIXASpWujp .doneCritText1,#mermaid-svg-Dea3eFcIXASpWujp .doneCritText2,#mermaid-svg-Dea3eFcIXASpWujp .doneCritText3{fill:#000 !important}#mermaid-svg-Dea3eFcIXASpWujp .activeCritText0,#mermaid-svg-Dea3eFcIXASpWujp .activeCritText1,#mermaid-svg-Dea3eFcIXASpWujp .activeCritText2,#mermaid-svg-Dea3eFcIXASpWujp .activeCritText3{fill:#000 !important}#mermaid-svg-Dea3eFcIXASpWujp .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-Dea3eFcIXASpWujp g.classGroup text .title{font-weight:bolder}#mermaid-svg-Dea3eFcIXASpWujp g.clickable{cursor:pointer}#mermaid-svg-Dea3eFcIXASpWujp g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-Dea3eFcIXASpWujp g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-Dea3eFcIXASpWujp .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-Dea3eFcIXASpWujp .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-Dea3eFcIXASpWujp .dashed-line{stroke-dasharray:3}#mermaid-svg-Dea3eFcIXASpWujp #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp .commit-id,#mermaid-svg-Dea3eFcIXASpWujp .commit-msg,#mermaid-svg-Dea3eFcIXASpWujp .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-Dea3eFcIXASpWujp g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-Dea3eFcIXASpWujp g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-Dea3eFcIXASpWujp g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-Dea3eFcIXASpWujp .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-Dea3eFcIXASpWujp .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-Dea3eFcIXASpWujp .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-Dea3eFcIXASpWujp .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-Dea3eFcIXASpWujp .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-Dea3eFcIXASpWujp .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-Dea3eFcIXASpWujp .edgeLabel text{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-Dea3eFcIXASpWujp .node circle.state-start{fill:black;stroke:black}#mermaid-svg-Dea3eFcIXASpWujp .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-Dea3eFcIXASpWujp #statediagram-barbEnd{fill:#9370db}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-state .divider{stroke:#9370db}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-Dea3eFcIXASpWujp .note-edge{stroke-dasharray:5}#mermaid-svg-Dea3eFcIXASpWujp .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-Dea3eFcIXASpWujp .error-icon{fill:#522}#mermaid-svg-Dea3eFcIXASpWujp .error-text{fill:#522;stroke:#522}#mermaid-svg-Dea3eFcIXASpWujp .edge-thickness-normal{stroke-width:2px}#mermaid-svg-Dea3eFcIXASpWujp .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-Dea3eFcIXASpWujp .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-Dea3eFcIXASpWujp .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-Dea3eFcIXASpWujp .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-Dea3eFcIXASpWujp .marker{fill:#333}#mermaid-svg-Dea3eFcIXASpWujp .marker.cross{stroke:#333}
:root { --mermaid-font-family: "trebuchet ms", verdana, arial;}</style>
<style>#mermaid-svg-Dea3eFcIXASpWujp {
color: rgba(0, 0, 0, 0.75);
font: ;
}</style>
Hapood101
Hapood102
Hapood103
高拓展性：在集群之間分配任務資料，可方便的擴展數以萬計節點， 高效性：在MapReduce的思想下，Hadoop是并行作業的，以加快任務處理速度， 高容錯性：能夠自動將失敗的任務重新分配， 4.Hadoop的組成
Hadoop1.X版本的組成MapReduce負責計算和計算所需的cpu、記憶體等資源的調度
<style>#mermaid-svg-qiLoTQLzopbdAv3r .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-qiLoTQLzopbdAv3r .label text{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .node rect,#mermaid-svg-qiLoTQLzopbdAv3r .node circle,#mermaid-svg-qiLoTQLzopbdAv3r .node ellipse,#mermaid-svg-qiLoTQLzopbdAv3r .node polygon,#mermaid-svg-qiLoTQLzopbdAv3r .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-qiLoTQLzopbdAv3r .node .label{text-align:center;fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .node.clickable{cursor:pointer}#mermaid-svg-qiLoTQLzopbdAv3r .arrowheadPath{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-qiLoTQLzopbdAv3r .flowchart-link{stroke:#333;fill:none}#mermaid-svg-qiLoTQLzopbdAv3r .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-qiLoTQLzopbdAv3r .edgeLabel rect{opacity:0.9}#mermaid-svg-qiLoTQLzopbdAv3r .edgeLabel span{color:#333}#mermaid-svg-qiLoTQLzopbdAv3r .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-qiLoTQLzopbdAv3r .cluster text{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-qiLoTQLzopbdAv3r .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-qiLoTQLzopbdAv3r text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-qiLoTQLzopbdAv3r .actor-line{stroke:grey}#mermaid-svg-qiLoTQLzopbdAv3r .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-qiLoTQLzopbdAv3r .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-qiLoTQLzopbdAv3r #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-qiLoTQLzopbdAv3r .sequenceNumber{fill:#fff}#mermaid-svg-qiLoTQLzopbdAv3r #sequencenumber{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r #crosshead path{fill:#333;stroke:#333}#mermaid-svg-qiLoTQLzopbdAv3r .messageText{fill:#333;stroke:#333}#mermaid-svg-qiLoTQLzopbdAv3r .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-qiLoTQLzopbdAv3r .labelText,#mermaid-svg-qiLoTQLzopbdAv3r .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-qiLoTQLzopbdAv3r .loopText,#mermaid-svg-qiLoTQLzopbdAv3r .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-qiLoTQLzopbdAv3r .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-qiLoTQLzopbdAv3r .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-qiLoTQLzopbdAv3r .noteText,#mermaid-svg-qiLoTQLzopbdAv3r .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-qiLoTQLzopbdAv3r .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-qiLoTQLzopbdAv3r .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-qiLoTQLzopbdAv3r .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-qiLoTQLzopbdAv3r .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r .section{stroke:none;opacity:0.2}#mermaid-svg-qiLoTQLzopbdAv3r .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-qiLoTQLzopbdAv3r .section2{fill:#fff400}#mermaid-svg-qiLoTQLzopbdAv3r .section1,#mermaid-svg-qiLoTQLzopbdAv3r .section3{fill:#fff;opacity:0.2}#mermaid-svg-qiLoTQLzopbdAv3r .sectionTitle0{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .sectionTitle1{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .sectionTitle2{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .sectionTitle3{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-qiLoTQLzopbdAv3r .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r .grid path{stroke-width:0}#mermaid-svg-qiLoTQLzopbdAv3r .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-qiLoTQLzopbdAv3r .task{stroke-width:2}#mermaid-svg-qiLoTQLzopbdAv3r .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r .taskText:not([font-size]){font-size:11px}#mermaid-svg-qiLoTQLzopbdAv3r .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-qiLoTQLzopbdAv3r .task.clickable{cursor:pointer}#mermaid-svg-qiLoTQLzopbdAv3r .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-qiLoTQLzopbdAv3r .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-qiLoTQLzopbdAv3r .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-qiLoTQLzopbdAv3r .taskText0,#mermaid-svg-qiLoTQLzopbdAv3r .taskText1,#mermaid-svg-qiLoTQLzopbdAv3r .taskText2,#mermaid-svg-qiLoTQLzopbdAv3r .taskText3{fill:#fff}#mermaid-svg-qiLoTQLzopbdAv3r .task0,#mermaid-svg-qiLoTQLzopbdAv3r .task1,#mermaid-svg-qiLoTQLzopbdAv3r .task2,#mermaid-svg-qiLoTQLzopbdAv3r .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-qiLoTQLzopbdAv3r .taskTextOutside0,#mermaid-svg-qiLoTQLzopbdAv3r .taskTextOutside2{fill:#000}#mermaid-svg-qiLoTQLzopbdAv3r .taskTextOutside1,#mermaid-svg-qiLoTQLzopbdAv3r .taskTextOutside3{fill:#000}#mermaid-svg-qiLoTQLzopbdAv3r .active0,#mermaid-svg-qiLoTQLzopbdAv3r .active1,#mermaid-svg-qiLoTQLzopbdAv3r .active2,#mermaid-svg-qiLoTQLzopbdAv3r .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-qiLoTQLzopbdAv3r .activeText0,#mermaid-svg-qiLoTQLzopbdAv3r .activeText1,#mermaid-svg-qiLoTQLzopbdAv3r .activeText2,#mermaid-svg-qiLoTQLzopbdAv3r .activeText3{fill:#000 !important}#mermaid-svg-qiLoTQLzopbdAv3r .done0,#mermaid-svg-qiLoTQLzopbdAv3r .done1,#mermaid-svg-qiLoTQLzopbdAv3r .done2,#mermaid-svg-qiLoTQLzopbdAv3r .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-qiLoTQLzopbdAv3r .doneText0,#mermaid-svg-qiLoTQLzopbdAv3r .doneText1,#mermaid-svg-qiLoTQLzopbdAv3r .doneText2,#mermaid-svg-qiLoTQLzopbdAv3r .doneText3{fill:#000 !important}#mermaid-svg-qiLoTQLzopbdAv3r .crit0,#mermaid-svg-qiLoTQLzopbdAv3r .crit1,#mermaid-svg-qiLoTQLzopbdAv3r .crit2,#mermaid-svg-qiLoTQLzopbdAv3r .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-qiLoTQLzopbdAv3r .activeCrit0,#mermaid-svg-qiLoTQLzopbdAv3r .activeCrit1,#mermaid-svg-qiLoTQLzopbdAv3r .activeCrit2,#mermaid-svg-qiLoTQLzopbdAv3r .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-qiLoTQLzopbdAv3r .doneCrit0,#mermaid-svg-qiLoTQLzopbdAv3r .doneCrit1,#mermaid-svg-qiLoTQLzopbdAv3r .doneCrit2,#mermaid-svg-qiLoTQLzopbdAv3r .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-qiLoTQLzopbdAv3r .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-qiLoTQLzopbdAv3r .milestoneText{font-style:italic}#mermaid-svg-qiLoTQLzopbdAv3r .doneCritText0,#mermaid-svg-qiLoTQLzopbdAv3r .doneCritText1,#mermaid-svg-qiLoTQLzopbdAv3r .doneCritText2,#mermaid-svg-qiLoTQLzopbdAv3r .doneCritText3{fill:#000 !important}#mermaid-svg-qiLoTQLzopbdAv3r .activeCritText0,#mermaid-svg-qiLoTQLzopbdAv3r .activeCritText1,#mermaid-svg-qiLoTQLzopbdAv3r .activeCritText2,#mermaid-svg-qiLoTQLzopbdAv3r .activeCritText3{fill:#000 !important}#mermaid-svg-qiLoTQLzopbdAv3r .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-qiLoTQLzopbdAv3r g.classGroup text .title{font-weight:bolder}#mermaid-svg-qiLoTQLzopbdAv3r g.clickable{cursor:pointer}#mermaid-svg-qiLoTQLzopbdAv3r g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-qiLoTQLzopbdAv3r g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-qiLoTQLzopbdAv3r .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-qiLoTQLzopbdAv3r .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-qiLoTQLzopbdAv3r .dashed-line{stroke-dasharray:3}#mermaid-svg-qiLoTQLzopbdAv3r #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r .commit-id,#mermaid-svg-qiLoTQLzopbdAv3r .commit-msg,#mermaid-svg-qiLoTQLzopbdAv3r .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-qiLoTQLzopbdAv3r g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-qiLoTQLzopbdAv3r g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-qiLoTQLzopbdAv3r g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-qiLoTQLzopbdAv3r .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-qiLoTQLzopbdAv3r .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-qiLoTQLzopbdAv3r .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-qiLoTQLzopbdAv3r .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-qiLoTQLzopbdAv3r .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-qiLoTQLzopbdAv3r .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-qiLoTQLzopbdAv3r .edgeLabel text{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-qiLoTQLzopbdAv3r .node circle.state-start{fill:black;stroke:black}#mermaid-svg-qiLoTQLzopbdAv3r .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-qiLoTQLzopbdAv3r #statediagram-barbEnd{fill:#9370db}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-state .divider{stroke:#9370db}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-qiLoTQLzopbdAv3r .note-edge{stroke-dasharray:5}#mermaid-svg-qiLoTQLzopbdAv3r .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-qiLoTQLzopbdAv3r .error-icon{fill:#522}#mermaid-svg-qiLoTQLzopbdAv3r .error-text{fill:#522;stroke:#522}#mermaid-svg-qiLoTQLzopbdAv3r .edge-thickness-normal{stroke-width:2px}#mermaid-svg-qiLoTQLzopbdAv3r .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-qiLoTQLzopbdAv3r .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-qiLoTQLzopbdAv3r .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-qiLoTQLzopbdAv3r .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-qiLoTQLzopbdAv3r .marker{fill:#333}#mermaid-svg-qiLoTQLzopbdAv3r .marker.cross{stroke:#333}
:root { --mermaid-font-family: "trebuchet ms", verdana, arial;}</style>
<style>#mermaid-svg-qiLoTQLzopbdAv3r {
color: rgba(0, 0, 0, 0.75);
font: ;
}</style>
hadoop1.x
HDFS 資料存盤
Common 輔助工具
MapReduce 計算+資源調度
Hadoop2.X版本的組成增加了Yarn進行資源調度，原來的MapReduce只負責計算，
<style>#mermaid-svg-J1r9t624mwhXcD9y .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-J1r9t624mwhXcD9y .label text{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .node rect,#mermaid-svg-J1r9t624mwhXcD9y .node circle,#mermaid-svg-J1r9t624mwhXcD9y .node ellipse,#mermaid-svg-J1r9t624mwhXcD9y .node polygon,#mermaid-svg-J1r9t624mwhXcD9y .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-J1r9t624mwhXcD9y .node .label{text-align:center;fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .node.clickable{cursor:pointer}#mermaid-svg-J1r9t624mwhXcD9y .arrowheadPath{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-J1r9t624mwhXcD9y .flowchart-link{stroke:#333;fill:none}#mermaid-svg-J1r9t624mwhXcD9y .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-J1r9t624mwhXcD9y .edgeLabel rect{opacity:0.9}#mermaid-svg-J1r9t624mwhXcD9y .edgeLabel span{color:#333}#mermaid-svg-J1r9t624mwhXcD9y .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-J1r9t624mwhXcD9y .cluster text{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-J1r9t624mwhXcD9y .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-J1r9t624mwhXcD9y text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-J1r9t624mwhXcD9y .actor-line{stroke:grey}#mermaid-svg-J1r9t624mwhXcD9y .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-J1r9t624mwhXcD9y .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-J1r9t624mwhXcD9y #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-J1r9t624mwhXcD9y .sequenceNumber{fill:#fff}#mermaid-svg-J1r9t624mwhXcD9y #sequencenumber{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y #crosshead path{fill:#333;stroke:#333}#mermaid-svg-J1r9t624mwhXcD9y .messageText{fill:#333;stroke:#333}#mermaid-svg-J1r9t624mwhXcD9y .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-J1r9t624mwhXcD9y .labelText,#mermaid-svg-J1r9t624mwhXcD9y .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-J1r9t624mwhXcD9y .loopText,#mermaid-svg-J1r9t624mwhXcD9y .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-J1r9t624mwhXcD9y .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-J1r9t624mwhXcD9y .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-J1r9t624mwhXcD9y .noteText,#mermaid-svg-J1r9t624mwhXcD9y .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-J1r9t624mwhXcD9y .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-J1r9t624mwhXcD9y .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-J1r9t624mwhXcD9y .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-J1r9t624mwhXcD9y .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y .section{stroke:none;opacity:0.2}#mermaid-svg-J1r9t624mwhXcD9y .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-J1r9t624mwhXcD9y .section2{fill:#fff400}#mermaid-svg-J1r9t624mwhXcD9y .section1,#mermaid-svg-J1r9t624mwhXcD9y .section3{fill:#fff;opacity:0.2}#mermaid-svg-J1r9t624mwhXcD9y .sectionTitle0{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .sectionTitle1{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .sectionTitle2{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .sectionTitle3{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-J1r9t624mwhXcD9y .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y .grid path{stroke-width:0}#mermaid-svg-J1r9t624mwhXcD9y .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-J1r9t624mwhXcD9y .task{stroke-width:2}#mermaid-svg-J1r9t624mwhXcD9y .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y .taskText:not([font-size]){font-size:11px}#mermaid-svg-J1r9t624mwhXcD9y .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-J1r9t624mwhXcD9y .task.clickable{cursor:pointer}#mermaid-svg-J1r9t624mwhXcD9y .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-J1r9t624mwhXcD9y .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-J1r9t624mwhXcD9y .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-J1r9t624mwhXcD9y .taskText0,#mermaid-svg-J1r9t624mwhXcD9y .taskText1,#mermaid-svg-J1r9t624mwhXcD9y .taskText2,#mermaid-svg-J1r9t624mwhXcD9y .taskText3{fill:#fff}#mermaid-svg-J1r9t624mwhXcD9y .task0,#mermaid-svg-J1r9t624mwhXcD9y .task1,#mermaid-svg-J1r9t624mwhXcD9y .task2,#mermaid-svg-J1r9t624mwhXcD9y .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-J1r9t624mwhXcD9y .taskTextOutside0,#mermaid-svg-J1r9t624mwhXcD9y .taskTextOutside2{fill:#000}#mermaid-svg-J1r9t624mwhXcD9y .taskTextOutside1,#mermaid-svg-J1r9t624mwhXcD9y .taskTextOutside3{fill:#000}#mermaid-svg-J1r9t624mwhXcD9y .active0,#mermaid-svg-J1r9t624mwhXcD9y .active1,#mermaid-svg-J1r9t624mwhXcD9y .active2,#mermaid-svg-J1r9t624mwhXcD9y .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-J1r9t624mwhXcD9y .activeText0,#mermaid-svg-J1r9t624mwhXcD9y .activeText1,#mermaid-svg-J1r9t624mwhXcD9y .activeText2,#mermaid-svg-J1r9t624mwhXcD9y .activeText3{fill:#000 !important}#mermaid-svg-J1r9t624mwhXcD9y .done0,#mermaid-svg-J1r9t624mwhXcD9y .done1,#mermaid-svg-J1r9t624mwhXcD9y .done2,#mermaid-svg-J1r9t624mwhXcD9y .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-J1r9t624mwhXcD9y .doneText0,#mermaid-svg-J1r9t624mwhXcD9y .doneText1,#mermaid-svg-J1r9t624mwhXcD9y .doneText2,#mermaid-svg-J1r9t624mwhXcD9y .doneText3{fill:#000 !important}#mermaid-svg-J1r9t624mwhXcD9y .crit0,#mermaid-svg-J1r9t624mwhXcD9y .crit1,#mermaid-svg-J1r9t624mwhXcD9y .crit2,#mermaid-svg-J1r9t624mwhXcD9y .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-J1r9t624mwhXcD9y .activeCrit0,#mermaid-svg-J1r9t624mwhXcD9y .activeCrit1,#mermaid-svg-J1r9t624mwhXcD9y .activeCrit2,#mermaid-svg-J1r9t624mwhXcD9y .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-J1r9t624mwhXcD9y .doneCrit0,#mermaid-svg-J1r9t624mwhXcD9y .doneCrit1,#mermaid-svg-J1r9t624mwhXcD9y .doneCrit2,#mermaid-svg-J1r9t624mwhXcD9y .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-J1r9t624mwhXcD9y .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-J1r9t624mwhXcD9y .milestoneText{font-style:italic}#mermaid-svg-J1r9t624mwhXcD9y .doneCritText0,#mermaid-svg-J1r9t624mwhXcD9y .doneCritText1,#mermaid-svg-J1r9t624mwhXcD9y .doneCritText2,#mermaid-svg-J1r9t624mwhXcD9y .doneCritText3{fill:#000 !important}#mermaid-svg-J1r9t624mwhXcD9y .activeCritText0,#mermaid-svg-J1r9t624mwhXcD9y .activeCritText1,#mermaid-svg-J1r9t624mwhXcD9y .activeCritText2,#mermaid-svg-J1r9t624mwhXcD9y .activeCritText3{fill:#000 !important}#mermaid-svg-J1r9t624mwhXcD9y .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-J1r9t624mwhXcD9y g.classGroup text .title{font-weight:bolder}#mermaid-svg-J1r9t624mwhXcD9y g.clickable{cursor:pointer}#mermaid-svg-J1r9t624mwhXcD9y g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-J1r9t624mwhXcD9y g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-J1r9t624mwhXcD9y .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-J1r9t624mwhXcD9y .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-J1r9t624mwhXcD9y .dashed-line{stroke-dasharray:3}#mermaid-svg-J1r9t624mwhXcD9y #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y .commit-id,#mermaid-svg-J1r9t624mwhXcD9y .commit-msg,#mermaid-svg-J1r9t624mwhXcD9y .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-J1r9t624mwhXcD9y g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-J1r9t624mwhXcD9y g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-J1r9t624mwhXcD9y g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-J1r9t624mwhXcD9y .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-J1r9t624mwhXcD9y .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-J1r9t624mwhXcD9y .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-J1r9t624mwhXcD9y .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-J1r9t624mwhXcD9y .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-J1r9t624mwhXcD9y .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-J1r9t624mwhXcD9y .edgeLabel text{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-J1r9t624mwhXcD9y .node circle.state-start{fill:black;stroke:black}#mermaid-svg-J1r9t624mwhXcD9y .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-J1r9t624mwhXcD9y #statediagram-barbEnd{fill:#9370db}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-state .divider{stroke:#9370db}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-J1r9t624mwhXcD9y .note-edge{stroke-dasharray:5}#mermaid-svg-J1r9t624mwhXcD9y .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-J1r9t624mwhXcD9y .error-icon{fill:#522}#mermaid-svg-J1r9t624mwhXcD9y .error-text{fill:#522;stroke:#522}#mermaid-svg-J1r9t624mwhXcD9y .edge-thickness-normal{stroke-width:2px}#mermaid-svg-J1r9t624mwhXcD9y .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-J1r9t624mwhXcD9y .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-J1r9t624mwhXcD9y .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-J1r9t624mwhXcD9y .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-J1r9t624mwhXcD9y .marker{fill:#333}#mermaid-svg-J1r9t624mwhXcD9y .marker.cross{stroke:#333}
:root { --mermaid-font-family: "trebuchet ms", verdana, arial;}</style>
<style>#mermaid-svg-J1r9t624mwhXcD9y {
color: rgba(0, 0, 0, 0.75);
font: ;
}</style>
hadoop2.x
HDFS 資料存盤
Common 輔助工具
Yarn 資源調度
MapReduce 計算
Hadoop3.X版本的組成沒啥區別，在細節上還是有區別的， 5.HDFS
HDFS(Hadoop Distributed File System)是一個分布式檔案系統，害怕NameNode掛掉，然后丟失所有資訊， ）
NameNode（nn）：存盤檔案的元資料，如檔案名、檔案目錄結構、檔案屬性，以及每個檔案的塊串列和塊所在的DataNode等， DataNode(dn):在本地檔案系統存盤檔案塊資料，以及塊資料的校驗和， Secondary NameNode(2nn):每隔一段時間對NameNode進行備份，
<style>#mermaid-svg-wrX3SewFr2miNvc6 .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-wrX3SewFr2miNvc6 .label text{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .node rect,#mermaid-svg-wrX3SewFr2miNvc6 .node circle,#mermaid-svg-wrX3SewFr2miNvc6 .node ellipse,#mermaid-svg-wrX3SewFr2miNvc6 .node polygon,#mermaid-svg-wrX3SewFr2miNvc6 .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-wrX3SewFr2miNvc6 .node .label{text-align:center;fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .node.clickable{cursor:pointer}#mermaid-svg-wrX3SewFr2miNvc6 .arrowheadPath{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-wrX3SewFr2miNvc6 .flowchart-link{stroke:#333;fill:none}#mermaid-svg-wrX3SewFr2miNvc6 .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-wrX3SewFr2miNvc6 .edgeLabel rect{opacity:0.9}#mermaid-svg-wrX3SewFr2miNvc6 .edgeLabel span{color:#333}#mermaid-svg-wrX3SewFr2miNvc6 .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-wrX3SewFr2miNvc6 .cluster text{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-wrX3SewFr2miNvc6 .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-wrX3SewFr2miNvc6 text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-wrX3SewFr2miNvc6 .actor-line{stroke:grey}#mermaid-svg-wrX3SewFr2miNvc6 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-wrX3SewFr2miNvc6 .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-wrX3SewFr2miNvc6 #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-wrX3SewFr2miNvc6 .sequenceNumber{fill:#fff}#mermaid-svg-wrX3SewFr2miNvc6 #sequencenumber{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 #crosshead path{fill:#333;stroke:#333}#mermaid-svg-wrX3SewFr2miNvc6 .messageText{fill:#333;stroke:#333}#mermaid-svg-wrX3SewFr2miNvc6 .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-wrX3SewFr2miNvc6 .labelText,#mermaid-svg-wrX3SewFr2miNvc6 .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-wrX3SewFr2miNvc6 .loopText,#mermaid-svg-wrX3SewFr2miNvc6 .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-wrX3SewFr2miNvc6 .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-wrX3SewFr2miNvc6 .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-wrX3SewFr2miNvc6 .noteText,#mermaid-svg-wrX3SewFr2miNvc6 .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-wrX3SewFr2miNvc6 .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-wrX3SewFr2miNvc6 .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-wrX3SewFr2miNvc6 .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-wrX3SewFr2miNvc6 .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 .section{stroke:none;opacity:0.2}#mermaid-svg-wrX3SewFr2miNvc6 .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-wrX3SewFr2miNvc6 .section2{fill:#fff400}#mermaid-svg-wrX3SewFr2miNvc6 .section1,#mermaid-svg-wrX3SewFr2miNvc6 .section3{fill:#fff;opacity:0.2}#mermaid-svg-wrX3SewFr2miNvc6 .sectionTitle0{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .sectionTitle1{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .sectionTitle2{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .sectionTitle3{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-wrX3SewFr2miNvc6 .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 .grid path{stroke-width:0}#mermaid-svg-wrX3SewFr2miNvc6 .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-wrX3SewFr2miNvc6 .task{stroke-width:2}#mermaid-svg-wrX3SewFr2miNvc6 .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 .taskText:not([font-size]){font-size:11px}#mermaid-svg-wrX3SewFr2miNvc6 .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-wrX3SewFr2miNvc6 .task.clickable{cursor:pointer}#mermaid-svg-wrX3SewFr2miNvc6 .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-wrX3SewFr2miNvc6 .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-wrX3SewFr2miNvc6 .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-wrX3SewFr2miNvc6 .taskText0,#mermaid-svg-wrX3SewFr2miNvc6 .taskText1,#mermaid-svg-wrX3SewFr2miNvc6 .taskText2,#mermaid-svg-wrX3SewFr2miNvc6 .taskText3{fill:#fff}#mermaid-svg-wrX3SewFr2miNvc6 .task0,#mermaid-svg-wrX3SewFr2miNvc6 .task1,#mermaid-svg-wrX3SewFr2miNvc6 .task2,#mermaid-svg-wrX3SewFr2miNvc6 .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-wrX3SewFr2miNvc6 .taskTextOutside0,#mermaid-svg-wrX3SewFr2miNvc6 .taskTextOutside2{fill:#000}#mermaid-svg-wrX3SewFr2miNvc6 .taskTextOutside1,#mermaid-svg-wrX3SewFr2miNvc6 .taskTextOutside3{fill:#000}#mermaid-svg-wrX3SewFr2miNvc6 .active0,#mermaid-svg-wrX3SewFr2miNvc6 .active1,#mermaid-svg-wrX3SewFr2miNvc6 .active2,#mermaid-svg-wrX3SewFr2miNvc6 .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-wrX3SewFr2miNvc6 .activeText0,#mermaid-svg-wrX3SewFr2miNvc6 .activeText1,#mermaid-svg-wrX3SewFr2miNvc6 .activeText2,#mermaid-svg-wrX3SewFr2miNvc6 .activeText3{fill:#000 !important}#mermaid-svg-wrX3SewFr2miNvc6 .done0,#mermaid-svg-wrX3SewFr2miNvc6 .done1,#mermaid-svg-wrX3SewFr2miNvc6 .done2,#mermaid-svg-wrX3SewFr2miNvc6 .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-wrX3SewFr2miNvc6 .doneText0,#mermaid-svg-wrX3SewFr2miNvc6 .doneText1,#mermaid-svg-wrX3SewFr2miNvc6 .doneText2,#mermaid-svg-wrX3SewFr2miNvc6 .doneText3{fill:#000 !important}#mermaid-svg-wrX3SewFr2miNvc6 .crit0,#mermaid-svg-wrX3SewFr2miNvc6 .crit1,#mermaid-svg-wrX3SewFr2miNvc6 .crit2,#mermaid-svg-wrX3SewFr2miNvc6 .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-wrX3SewFr2miNvc6 .activeCrit0,#mermaid-svg-wrX3SewFr2miNvc6 .activeCrit1,#mermaid-svg-wrX3SewFr2miNvc6 .activeCrit2,#mermaid-svg-wrX3SewFr2miNvc6 .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-wrX3SewFr2miNvc6 .doneCrit0,#mermaid-svg-wrX3SewFr2miNvc6 .doneCrit1,#mermaid-svg-wrX3SewFr2miNvc6 .doneCrit2,#mermaid-svg-wrX3SewFr2miNvc6 .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-wrX3SewFr2miNvc6 .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-wrX3SewFr2miNvc6 .milestoneText{font-style:italic}#mermaid-svg-wrX3SewFr2miNvc6 .doneCritText0,#mermaid-svg-wrX3SewFr2miNvc6 .doneCritText1,#mermaid-svg-wrX3SewFr2miNvc6 .doneCritText2,#mermaid-svg-wrX3SewFr2miNvc6 .doneCritText3{fill:#000 !important}#mermaid-svg-wrX3SewFr2miNvc6 .activeCritText0,#mermaid-svg-wrX3SewFr2miNvc6 .activeCritText1,#mermaid-svg-wrX3SewFr2miNvc6 .activeCritText2,#mermaid-svg-wrX3SewFr2miNvc6 .activeCritText3{fill:#000 !important}#mermaid-svg-wrX3SewFr2miNvc6 .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-wrX3SewFr2miNvc6 g.classGroup text .title{font-weight:bolder}#mermaid-svg-wrX3SewFr2miNvc6 g.clickable{cursor:pointer}#mermaid-svg-wrX3SewFr2miNvc6 g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-wrX3SewFr2miNvc6 g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-wrX3SewFr2miNvc6 .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-wrX3SewFr2miNvc6 .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-wrX3SewFr2miNvc6 .dashed-line{stroke-dasharray:3}#mermaid-svg-wrX3SewFr2miNvc6 #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 .commit-id,#mermaid-svg-wrX3SewFr2miNvc6 .commit-msg,#mermaid-svg-wrX3SewFr2miNvc6 .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-wrX3SewFr2miNvc6 g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-wrX3SewFr2miNvc6 g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-wrX3SewFr2miNvc6 g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-wrX3SewFr2miNvc6 .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-wrX3SewFr2miNvc6 .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-wrX3SewFr2miNvc6 .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-wrX3SewFr2miNvc6 .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-wrX3SewFr2miNvc6 .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-wrX3SewFr2miNvc6 .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-wrX3SewFr2miNvc6 .edgeLabel text{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-wrX3SewFr2miNvc6 .node circle.state-start{fill:black;stroke:black}#mermaid-svg-wrX3SewFr2miNvc6 .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-wrX3SewFr2miNvc6 #statediagram-barbEnd{fill:#9370db}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-state .divider{stroke:#9370db}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-wrX3SewFr2miNvc6 .note-edge{stroke-dasharray:5}#mermaid-svg-wrX3SewFr2miNvc6 .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-wrX3SewFr2miNvc6 .error-icon{fill:#522}#mermaid-svg-wrX3SewFr2miNvc6 .error-text{fill:#522;stroke:#522}#mermaid-svg-wrX3SewFr2miNvc6 .edge-thickness-normal{stroke-width:2px}#mermaid-svg-wrX3SewFr2miNvc6 .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-wrX3SewFr2miNvc6 .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-wrX3SewFr2miNvc6 .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-wrX3SewFr2miNvc6 .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-wrX3SewFr2miNvc6 .marker{fill:#333}#mermaid-svg-wrX3SewFr2miNvc6 .marker.cross{stroke:#333}
:root { --mermaid-font-family: "trebuchet ms", verdana, arial;}</style>
<style>#mermaid-svg-wrX3SewFr2miNvc6 {
color: rgba(0, 0, 0, 0.75);
font: ;
}</style>
NameNode資料存盤的位置
存盤資料DataNode1
存盤資料DataNode2
存盤資料DataNode3
.....還有很多........
2NN備份
6.YARN
YARN(Yet Another Resource Negotiator),是一種資源協調者，是Hadpood的資源管理器，
ResourceManager(RM):整個集群資源（記憶體、CPU）的老大， NodeManager(NM):單個節點服務器資源老大， ApplicationMaster(AM):單個任務運行的老大， client:客戶端 Container:容器，相當于一臺獨立的服務器，里面封裝了運行所需的資源，如記憶體、CPU、磁盤、網路等， 客戶端可有多個、集群上可有運行多個ApplicationMaster、每個NodeManager上可以有多個Container.
<style>#mermaid-svg-hH77zcKpAI6KPpDk .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-hH77zcKpAI6KPpDk .label text{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .node rect,#mermaid-svg-hH77zcKpAI6KPpDk .node circle,#mermaid-svg-hH77zcKpAI6KPpDk .node ellipse,#mermaid-svg-hH77zcKpAI6KPpDk .node polygon,#mermaid-svg-hH77zcKpAI6KPpDk .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-hH77zcKpAI6KPpDk .node .label{text-align:center;fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .node.clickable{cursor:pointer}#mermaid-svg-hH77zcKpAI6KPpDk .arrowheadPath{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-hH77zcKpAI6KPpDk .flowchart-link{stroke:#333;fill:none}#mermaid-svg-hH77zcKpAI6KPpDk .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-hH77zcKpAI6KPpDk .edgeLabel rect{opacity:0.9}#mermaid-svg-hH77zcKpAI6KPpDk .edgeLabel span{color:#333}#mermaid-svg-hH77zcKpAI6KPpDk .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-hH77zcKpAI6KPpDk .cluster text{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-hH77zcKpAI6KPpDk .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-hH77zcKpAI6KPpDk text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-hH77zcKpAI6KPpDk .actor-line{stroke:grey}#mermaid-svg-hH77zcKpAI6KPpDk .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-hH77zcKpAI6KPpDk .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-hH77zcKpAI6KPpDk #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-hH77zcKpAI6KPpDk .sequenceNumber{fill:#fff}#mermaid-svg-hH77zcKpAI6KPpDk #sequencenumber{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk #crosshead path{fill:#333;stroke:#333}#mermaid-svg-hH77zcKpAI6KPpDk .messageText{fill:#333;stroke:#333}#mermaid-svg-hH77zcKpAI6KPpDk .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-hH77zcKpAI6KPpDk .labelText,#mermaid-svg-hH77zcKpAI6KPpDk .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-hH77zcKpAI6KPpDk .loopText,#mermaid-svg-hH77zcKpAI6KPpDk .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-hH77zcKpAI6KPpDk .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-hH77zcKpAI6KPpDk .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-hH77zcKpAI6KPpDk .noteText,#mermaid-svg-hH77zcKpAI6KPpDk .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-hH77zcKpAI6KPpDk .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-hH77zcKpAI6KPpDk .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-hH77zcKpAI6KPpDk .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-hH77zcKpAI6KPpDk .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk .section{stroke:none;opacity:0.2}#mermaid-svg-hH77zcKpAI6KPpDk .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-hH77zcKpAI6KPpDk .section2{fill:#fff400}#mermaid-svg-hH77zcKpAI6KPpDk .section1,#mermaid-svg-hH77zcKpAI6KPpDk .section3{fill:#fff;opacity:0.2}#mermaid-svg-hH77zcKpAI6KPpDk .sectionTitle0{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .sectionTitle1{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .sectionTitle2{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .sectionTitle3{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-hH77zcKpAI6KPpDk .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk .grid path{stroke-width:0}#mermaid-svg-hH77zcKpAI6KPpDk .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-hH77zcKpAI6KPpDk .task{stroke-width:2}#mermaid-svg-hH77zcKpAI6KPpDk .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk .taskText:not([font-size]){font-size:11px}#mermaid-svg-hH77zcKpAI6KPpDk .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-hH77zcKpAI6KPpDk .task.clickable{cursor:pointer}#mermaid-svg-hH77zcKpAI6KPpDk .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-hH77zcKpAI6KPpDk .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-hH77zcKpAI6KPpDk .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-hH77zcKpAI6KPpDk .taskText0,#mermaid-svg-hH77zcKpAI6KPpDk .taskText1,#mermaid-svg-hH77zcKpAI6KPpDk .taskText2,#mermaid-svg-hH77zcKpAI6KPpDk .taskText3{fill:#fff}#mermaid-svg-hH77zcKpAI6KPpDk .task0,#mermaid-svg-hH77zcKpAI6KPpDk .task1,#mermaid-svg-hH77zcKpAI6KPpDk .task2,#mermaid-svg-hH77zcKpAI6KPpDk .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-hH77zcKpAI6KPpDk .taskTextOutside0,#mermaid-svg-hH77zcKpAI6KPpDk .taskTextOutside2{fill:#000}#mermaid-svg-hH77zcKpAI6KPpDk .taskTextOutside1,#mermaid-svg-hH77zcKpAI6KPpDk .taskTextOutside3{fill:#000}#mermaid-svg-hH77zcKpAI6KPpDk .active0,#mermaid-svg-hH77zcKpAI6KPpDk .active1,#mermaid-svg-hH77zcKpAI6KPpDk .active2,#mermaid-svg-hH77zcKpAI6KPpDk .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-hH77zcKpAI6KPpDk .activeText0,#mermaid-svg-hH77zcKpAI6KPpDk .activeText1,#mermaid-svg-hH77zcKpAI6KPpDk .activeText2,#mermaid-svg-hH77zcKpAI6KPpDk .activeText3{fill:#000 !important}#mermaid-svg-hH77zcKpAI6KPpDk .done0,#mermaid-svg-hH77zcKpAI6KPpDk .done1,#mermaid-svg-hH77zcKpAI6KPpDk .done2,#mermaid-svg-hH77zcKpAI6KPpDk .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-hH77zcKpAI6KPpDk .doneText0,#mermaid-svg-hH77zcKpAI6KPpDk .doneText1,#mermaid-svg-hH77zcKpAI6KPpDk .doneText2,#mermaid-svg-hH77zcKpAI6KPpDk .doneText3{fill:#000 !important}#mermaid-svg-hH77zcKpAI6KPpDk .crit0,#mermaid-svg-hH77zcKpAI6KPpDk .crit1,#mermaid-svg-hH77zcKpAI6KPpDk .crit2,#mermaid-svg-hH77zcKpAI6KPpDk .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-hH77zcKpAI6KPpDk .activeCrit0,#mermaid-svg-hH77zcKpAI6KPpDk .activeCrit1,#mermaid-svg-hH77zcKpAI6KPpDk .activeCrit2,#mermaid-svg-hH77zcKpAI6KPpDk .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-hH77zcKpAI6KPpDk .doneCrit0,#mermaid-svg-hH77zcKpAI6KPpDk .doneCrit1,#mermaid-svg-hH77zcKpAI6KPpDk .doneCrit2,#mermaid-svg-hH77zcKpAI6KPpDk .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-hH77zcKpAI6KPpDk .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-hH77zcKpAI6KPpDk .milestoneText{font-style:italic}#mermaid-svg-hH77zcKpAI6KPpDk .doneCritText0,#mermaid-svg-hH77zcKpAI6KPpDk .doneCritText1,#mermaid-svg-hH77zcKpAI6KPpDk .doneCritText2,#mermaid-svg-hH77zcKpAI6KPpDk .doneCritText3{fill:#000 !important}#mermaid-svg-hH77zcKpAI6KPpDk .activeCritText0,#mermaid-svg-hH77zcKpAI6KPpDk .activeCritText1,#mermaid-svg-hH77zcKpAI6KPpDk .activeCritText2,#mermaid-svg-hH77zcKpAI6KPpDk .activeCritText3{fill:#000 !important}#mermaid-svg-hH77zcKpAI6KPpDk .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-hH77zcKpAI6KPpDk g.classGroup text .title{font-weight:bolder}#mermaid-svg-hH77zcKpAI6KPpDk g.clickable{cursor:pointer}#mermaid-svg-hH77zcKpAI6KPpDk g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-hH77zcKpAI6KPpDk g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-hH77zcKpAI6KPpDk .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-hH77zcKpAI6KPpDk .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-hH77zcKpAI6KPpDk .dashed-line{stroke-dasharray:3}#mermaid-svg-hH77zcKpAI6KPpDk #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk .commit-id,#mermaid-svg-hH77zcKpAI6KPpDk .commit-msg,#mermaid-svg-hH77zcKpAI6KPpDk .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-hH77zcKpAI6KPpDk g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-hH77zcKpAI6KPpDk g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-hH77zcKpAI6KPpDk g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-hH77zcKpAI6KPpDk .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-hH77zcKpAI6KPpDk .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-hH77zcKpAI6KPpDk .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-hH77zcKpAI6KPpDk .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-hH77zcKpAI6KPpDk .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-hH77zcKpAI6KPpDk .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-hH77zcKpAI6KPpDk .edgeLabel text{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-hH77zcKpAI6KPpDk .node circle.state-start{fill:black;stroke:black}#mermaid-svg-hH77zcKpAI6KPpDk .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-hH77zcKpAI6KPpDk #statediagram-barbEnd{fill:#9370db}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-state .divider{stroke:#9370db}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-hH77zcKpAI6KPpDk .note-edge{stroke-dasharray:5}#mermaid-svg-hH77zcKpAI6KPpDk .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-hH77zcKpAI6KPpDk .error-icon{fill:#522}#mermaid-svg-hH77zcKpAI6KPpDk .error-text{fill:#522;stroke:#522}#mermaid-svg-hH77zcKpAI6KPpDk .edge-thickness-normal{stroke-width:2px}#mermaid-svg-hH77zcKpAI6KPpDk .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-hH77zcKpAI6KPpDk .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-hH77zcKpAI6KPpDk .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-hH77zcKpAI6KPpDk .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-hH77zcKpAI6KPpDk .marker{fill:#333}#mermaid-svg-hH77zcKpAI6KPpDk .marker.cross{stroke:#333}
:root { --mermaid-font-family: "trebuchet ms", verdana, arial;}</style>
<style>#mermaid-svg-hH77zcKpAI6KPpDk {
color: rgba(0, 0, 0, 0.75);
font: ;
}</style>
i1
i2
i3
i4
NodeManager - 4G記憶體2CPU
Container1
NodeManager - 4G記憶體2CPU
Container里包含App Mstr
NodeManager - 4G記憶體2CPU
Container里包含App Mstr
Container4
NodeManager - 4G記憶體2CPU
Resource Manager 16G記憶體8CPU
client
client
7.MapReduce
MapReduce將計算程序劃分為兩個階段：MAP和Reduce
Map階段并行處理輸入資料， Reduce階段對Map結果進行匯總，
<style>#mermaid-svg-8RnLY626VSmyS0dd .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-8RnLY626VSmyS0dd .label text{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .node rect,#mermaid-svg-8RnLY626VSmyS0dd .node circle,#mermaid-svg-8RnLY626VSmyS0dd .node ellipse,#mermaid-svg-8RnLY626VSmyS0dd .node polygon,#mermaid-svg-8RnLY626VSmyS0dd .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-8RnLY626VSmyS0dd .node .label{text-align:center;fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .node.clickable{cursor:pointer}#mermaid-svg-8RnLY626VSmyS0dd .arrowheadPath{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-8RnLY626VSmyS0dd .flowchart-link{stroke:#333;fill:none}#mermaid-svg-8RnLY626VSmyS0dd .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-8RnLY626VSmyS0dd .edgeLabel rect{opacity:0.9}#mermaid-svg-8RnLY626VSmyS0dd .edgeLabel span{color:#333}#mermaid-svg-8RnLY626VSmyS0dd .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-8RnLY626VSmyS0dd .cluster text{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-8RnLY626VSmyS0dd .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-8RnLY626VSmyS0dd text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-8RnLY626VSmyS0dd .actor-line{stroke:grey}#mermaid-svg-8RnLY626VSmyS0dd .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-8RnLY626VSmyS0dd .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-8RnLY626VSmyS0dd #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-8RnLY626VSmyS0dd .sequenceNumber{fill:#fff}#mermaid-svg-8RnLY626VSmyS0dd #sequencenumber{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd #crosshead path{fill:#333;stroke:#333}#mermaid-svg-8RnLY626VSmyS0dd .messageText{fill:#333;stroke:#333}#mermaid-svg-8RnLY626VSmyS0dd .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-8RnLY626VSmyS0dd .labelText,#mermaid-svg-8RnLY626VSmyS0dd .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-8RnLY626VSmyS0dd .loopText,#mermaid-svg-8RnLY626VSmyS0dd .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-8RnLY626VSmyS0dd .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-8RnLY626VSmyS0dd .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-8RnLY626VSmyS0dd .noteText,#mermaid-svg-8RnLY626VSmyS0dd .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-8RnLY626VSmyS0dd .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-8RnLY626VSmyS0dd .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-8RnLY626VSmyS0dd .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-8RnLY626VSmyS0dd .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd .section{stroke:none;opacity:0.2}#mermaid-svg-8RnLY626VSmyS0dd .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-8RnLY626VSmyS0dd .section2{fill:#fff400}#mermaid-svg-8RnLY626VSmyS0dd .section1,#mermaid-svg-8RnLY626VSmyS0dd .section3{fill:#fff;opacity:0.2}#mermaid-svg-8RnLY626VSmyS0dd .sectionTitle0{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .sectionTitle1{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .sectionTitle2{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .sectionTitle3{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-8RnLY626VSmyS0dd .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd .grid path{stroke-width:0}#mermaid-svg-8RnLY626VSmyS0dd .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-8RnLY626VSmyS0dd .task{stroke-width:2}#mermaid-svg-8RnLY626VSmyS0dd .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd .taskText:not([font-size]){font-size:11px}#mermaid-svg-8RnLY626VSmyS0dd .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-8RnLY626VSmyS0dd .task.clickable{cursor:pointer}#mermaid-svg-8RnLY626VSmyS0dd .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-8RnLY626VSmyS0dd .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-8RnLY626VSmyS0dd .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-8RnLY626VSmyS0dd .taskText0,#mermaid-svg-8RnLY626VSmyS0dd .taskText1,#mermaid-svg-8RnLY626VSmyS0dd .taskText2,#mermaid-svg-8RnLY626VSmyS0dd .taskText3{fill:#fff}#mermaid-svg-8RnLY626VSmyS0dd .task0,#mermaid-svg-8RnLY626VSmyS0dd .task1,#mermaid-svg-8RnLY626VSmyS0dd .task2,#mermaid-svg-8RnLY626VSmyS0dd .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-8RnLY626VSmyS0dd .taskTextOutside0,#mermaid-svg-8RnLY626VSmyS0dd .taskTextOutside2{fill:#000}#mermaid-svg-8RnLY626VSmyS0dd .taskTextOutside1,#mermaid-svg-8RnLY626VSmyS0dd .taskTextOutside3{fill:#000}#mermaid-svg-8RnLY626VSmyS0dd .active0,#mermaid-svg-8RnLY626VSmyS0dd .active1,#mermaid-svg-8RnLY626VSmyS0dd .active2,#mermaid-svg-8RnLY626VSmyS0dd .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-8RnLY626VSmyS0dd .activeText0,#mermaid-svg-8RnLY626VSmyS0dd .activeText1,#mermaid-svg-8RnLY626VSmyS0dd .activeText2,#mermaid-svg-8RnLY626VSmyS0dd .activeText3{fill:#000 !important}#mermaid-svg-8RnLY626VSmyS0dd .done0,#mermaid-svg-8RnLY626VSmyS0dd .done1,#mermaid-svg-8RnLY626VSmyS0dd .done2,#mermaid-svg-8RnLY626VSmyS0dd .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-8RnLY626VSmyS0dd .doneText0,#mermaid-svg-8RnLY626VSmyS0dd .doneText1,#mermaid-svg-8RnLY626VSmyS0dd .doneText2,#mermaid-svg-8RnLY626VSmyS0dd .doneText3{fill:#000 !important}#mermaid-svg-8RnLY626VSmyS0dd .crit0,#mermaid-svg-8RnLY626VSmyS0dd .crit1,#mermaid-svg-8RnLY626VSmyS0dd .crit2,#mermaid-svg-8RnLY626VSmyS0dd .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-8RnLY626VSmyS0dd .activeCrit0,#mermaid-svg-8RnLY626VSmyS0dd .activeCrit1,#mermaid-svg-8RnLY626VSmyS0dd .activeCrit2,#mermaid-svg-8RnLY626VSmyS0dd .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-8RnLY626VSmyS0dd .doneCrit0,#mermaid-svg-8RnLY626VSmyS0dd .doneCrit1,#mermaid-svg-8RnLY626VSmyS0dd .doneCrit2,#mermaid-svg-8RnLY626VSmyS0dd .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-8RnLY626VSmyS0dd .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-8RnLY626VSmyS0dd .milestoneText{font-style:italic}#mermaid-svg-8RnLY626VSmyS0dd .doneCritText0,#mermaid-svg-8RnLY626VSmyS0dd .doneCritText1,#mermaid-svg-8RnLY626VSmyS0dd .doneCritText2,#mermaid-svg-8RnLY626VSmyS0dd .doneCritText3{fill:#000 !important}#mermaid-svg-8RnLY626VSmyS0dd .activeCritText0,#mermaid-svg-8RnLY626VSmyS0dd .activeCritText1,#mermaid-svg-8RnLY626VSmyS0dd .activeCritText2,#mermaid-svg-8RnLY626VSmyS0dd .activeCritText3{fill:#000 !important}#mermaid-svg-8RnLY626VSmyS0dd .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-8RnLY626VSmyS0dd g.classGroup text .title{font-weight:bolder}#mermaid-svg-8RnLY626VSmyS0dd g.clickable{cursor:pointer}#mermaid-svg-8RnLY626VSmyS0dd g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-8RnLY626VSmyS0dd g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-8RnLY626VSmyS0dd .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-8RnLY626VSmyS0dd .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-8RnLY626VSmyS0dd .dashed-line{stroke-dasharray:3}#mermaid-svg-8RnLY626VSmyS0dd #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd .commit-id,#mermaid-svg-8RnLY626VSmyS0dd .commit-msg,#mermaid-svg-8RnLY626VSmyS0dd .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-8RnLY626VSmyS0dd g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-8RnLY626VSmyS0dd g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-8RnLY626VSmyS0dd g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-8RnLY626VSmyS0dd .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-8RnLY626VSmyS0dd .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-8RnLY626VSmyS0dd .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-8RnLY626VSmyS0dd .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-8RnLY626VSmyS0dd .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-8RnLY626VSmyS0dd .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-8RnLY626VSmyS0dd .edgeLabel text{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-8RnLY626VSmyS0dd .node circle.state-start{fill:black;stroke:black}#mermaid-svg-8RnLY626VSmyS0dd .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-8RnLY626VSmyS0dd #statediagram-barbEnd{fill:#9370db}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-state .divider{stroke:#9370db}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-8RnLY626VSmyS0dd .note-edge{stroke-dasharray:5}#mermaid-svg-8RnLY626VSmyS0dd .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-8RnLY626VSmyS0dd .error-icon{fill:#522}#mermaid-svg-8RnLY626VSmyS0dd .error-text{fill:#522;stroke:#522}#mermaid-svg-8RnLY626VSmyS0dd .edge-thickness-normal{stroke-width:2px}#mermaid-svg-8RnLY626VSmyS0dd .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-8RnLY626VSmyS0dd .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-8RnLY626VSmyS0dd .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-8RnLY626VSmyS0dd .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-8RnLY626VSmyS0dd .marker{fill:#333}#mermaid-svg-8RnLY626VSmyS0dd .marker.cross{stroke:#333}
:root { --mermaid-font-family: "trebuchet ms", verdana, arial;}</style>
<style>#mermaid-svg-8RnLY626VSmyS0dd {
color: rgba(0, 0, 0, 0.75);
font: ;
}</style>
map
Reduce
資料集100T
hadpood101
hadpood102
hadpood103
......很多........
匯總服務器
8.HDFS、YARN和MapReduce三者的關系
9.大資料處理的程序
參考資料：
大資料課程 《Hadoop入門》
