使用 Docker 搭建離線數倉環境

前言

使用版本：

一、Docker安裝

1.1 Centos Docker安裝

# 鏡像比較大, 需要準備一個網路穩定的環境
# 其中--mirror Aliyun代表使用阿里源
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

1.2 Ubuntu Docker安裝【推薦】

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

1.3 MacOs Docker安裝

# 下載安裝包, 拖動安裝即可
https://hub.docker.com/editions/community/docker-ce-desktop-mac/

1.4 Windows Docker安裝【不推薦】

# win10家庭版 【參考】
https://docs.docker.com/docker-for-windows/install-windows-home/

# win10專業版、商業版或教育版 【參考】
https://docs.docker.com/docker-for-windows/install/

二、容器準備

2.1 拉取鏡像

docker pull centos:7

2.2 啟動并創建容器

docker run -itd --privileged --name singleNode -h singleNode \
-p 2222:22 \
-p 3306:3306 \
-p 8020:8020 \
-p 9870:9870 \
-p 19888:19888 \
-p 8088:8088 \
-p 9083:9083 \
-p 10000:10000 \
-p 2181:2181 \
-p 9092:9092 \
-p 8091:8091 \
-p 8080:8080 \
-p 16010:16010 \
-p 4000:4000 \
-p 3000:3000 \
centos:7 /usr/sbin/init

# 其中埠號解釋
2222:22# SSH
3306:3306 #MySQL
8020:8020 # HDFS RPC
9870:9870 # HDFS web UI
19888:19888 # Yarn job history 
8088:8088 # Yarn web UI
9083:9083 # Hive metastore
10000:10000 # HiveServer2
2181:2181 # zk
9092:9092 # kafka
8091:8091 # flink

2.3 進入容器

docker exec -it singleNode /bin/bash

三、環境準備

3.1 安裝必要軟體

yum clean all
yum -y install unzip bzip2-devel vim bashname
yum install kde-l10n-Chinese -y
yum install glibc-common -y
localedef -c -f UTF-8 -i zh_CN zh_CN.utf8
echo "export LANG=zh_CN.UTF-8" >> /etc/locale.conf
echo "LC_ALL zh_CN.UTF-8" >> ~/.bashrc

3.2 配置SSH免密登錄

# 修改root密碼passwd root  # 輸入兩次密碼# 安裝必要SSH服務yum install -y openssh openssh-server openssh-clients openssl openssl-devel # 生成秘鑰ssh-keygen -t rsa -f ~/.ssh/id_rsa -P '' # 配置免密cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys# 方式2: ssh-copy-id# 啟動SSH服務systemctl start sshd

3.3 設定時區

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

3.4 關閉防火墻

yum -y install firewalldsystemctl stop firewalldsystemctl disable firewalld

3.5 時間同步、靜態ip、主機映射

• 由于本次課使用docker進行環境搭建，所以對于靜態ip和主機映射可以不用配置
• 由于本次課搭建的是單節點的偽分布式集群，所以時間同步可以用不設定
• 如果在物理機上搭建多節點的完全分布式集群則必須配置

四、MySQL安裝

4.1 上傳解壓安裝包

cd /opt/software/tar xvf MySQL-5.5.40-1.linux2.6.x86_64.rpm-bundle.tar

4.2 安裝必要依賴

yum -y install libaio perl

4.3 安裝服務端和客戶端

rpm -ivh MySQL-server-5.5.40-1.linux2.6.x86_64.rpmrpm -ivh MySQL-client-5.5.40-1.linux2.6.x86_64.rpm 

4.4 啟動并配置MySQL

方式一

# 啟動服務systemctl start mysql# 修改MySQL密碼/usr/bin/mysqladmin -u root password 'root'# 登陸MySQL設定權限mysql -uroot -proot > update mysql.user set host='%' where host='localhost';> delete from mysql.user where host<>'%' or user='';> flush privileges;

方式二

# 啟動服務systemctl start mysql# 執行MySQL的初始化/usr/bin/mysql_secure_installation# 輸入一次回車, 兩次相同的密碼進行修改密碼# Remove anonymous users? [Y/n] 是否移除掉anonymous用戶  n# Disallow root login remotely? [Y/n] 是否允許root用戶遠程登錄 y# Remove test database and access to it? [Y/n] 是否移除掉test資料庫 n# Reload privilege tables now? [Y/n] 是否現在重繪權限 y# 登陸MySQL設定權限mysql -uroot -proot > update mysql.user set host='%' where host='localhost';> delete from mysql.user where host<>'%' or user='';> flush privileges;

五、安裝JDK

5.1 上傳并解壓

tar zxvf /opt/software/jdk-8u171-linux-x64.tar.gz -C /opt/install/ln -s /opt/install/jdk1.8.0_171 /opt/install/java

5.2 配置環境變數

環境變數配置在 ~/.bashrc 里

vi ~/.bashrc-------------------------------------------export JAVA_HOME=/opt/install/javaexport PATH=$JAVA_HOME/bin:$PATH-------------------------------------------source ~/.bashrc

5.3 查看版本

java -version

六、Hadoop安裝

6.1 上傳并解壓

tar zxvf hadoop-3.2.1.tar.gz -C /opt/install/ln -s /opt/install/hadoop-3.2.1/ /opt/install/hadoop

6.2 修改配置

# 進入路徑cd /opt/install/hadoop/etc/hadoop/

6.2.1 配置core-site.xml

vi core-site.xml-------------------------------------------<configuration>    <property>        <name>fs.defaultFS</name>        <value>hdfs://singleNode:8020</value>    </property>    <property>        <name>hadoop.tmp.dir</name>        <value>/opt/install/hadoop/data</value>    </property>    <property>        <name>hadoop.proxyuser.root.hosts</name>        <value>*</value>    </property>    <property>        <name>hadoop.proxyuser.root.groups</name>        <value>*</value>    </property>    <property>        <name>hadoop.http.staticuser.user</name>        <value>root</value>    </property></configuration>-------------------------------------------

6.2.2 配置hdfs-site.xml

vi hdfs-site.xml-------------------------------------------<configuration>    <property>        <name>dfs.replication</name>        <value>1</value>    </property>    <property>        <name>dfs.namenode.secondary.http-address</name>        <value>singleNode:9868</value>    </property></configuration>-------------------------------------------

6.2.3 配置mapred-site.xml

vi mapred-site.xml-------------------------------------------<configuration>  <property>    <name>mapreduce.framework.name</name>    <value>yarn</value>  </property>  <property>    <name>mapreduce.jobhistory.address</name>    <value>singleNode:10020</value>  </property>  <property>    <name>mapreduce.jobhistory.webapp.address</name>    <value>singleNode:19888</value>  </property></configuration>-------------------------------------------

6.2.4 配置yarn-site.xml

vi yarn-site.xml-------------------------------------------<configuration>    <property>        <name>yarn.nodemanager.aux-services</name>        <value>mapreduce_shuffle</value>    </property>    <property>        <name>yarn.resourcemanager.hostname</name>        <value>singleNode</value>    </property>    <property>        <name>yarn.nodemanager.env-whitelist</name>        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>    </property>    <property>        <name>yarn.scheduler.minimum-allocation-mb</name>        <value>512</value>    </property>    <property>        <name>yarn.scheduler.maximum-allocation-mb</name>        <value>4096</value>    </property>    <property>        <name>yarn.nodemanager.resource.memory-mb</name>        <value>4096</value>    </property>    <property>        <name>yarn.nodemanager.pmem-check-enabled</name>        <value>false</value>    </property>    <property>        <name>yarn.nodemanager.vmem-check-enabled</name>        <value>false</value>    </property>    <property>        <name>yarn.log-aggregation-enable</name>        <value>true</value>    </property>    <property>          <name>yarn.log.server.url</name>          <value>http://${yarn.timeline-service.webapp.address}/applicationhistory/logs</value>    </property>    <property>        <name>yarn.log-aggregation.retain-seconds</name>        <value>604800</value>    </property>    <property>        <name>yarn.timeline-service.enabled</name>        <value>true</value>    </property>    <property>        <name>yarn.timeline-service.hostname</name>        <value>${yarn.resourcemanager.hostname}</value>    </property>    <property>        <name>yarn.timeline-service.http-cross-origin.enabled</name>        <value>true</value>    </property>    <property>        <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>        <value>true</value>    </property></configuration>-------------------------------------------

6.2.5 配置hadoop-env.sh

vi hadoop-env.sh-------------------------------------------export JAVA_HOME=/opt/install/java-------------------------------------------

6.2.6 配置mapred-env.sh

vi mapred-env.sh-------------------------------------------export JAVA_HOME=/opt/install/java-------------------------------------------

6.2.7 配置yarn-env.sh

vi yarn-env.sh-------------------------------------------export JAVA_HOME=/opt/install/java-------------------------------------------

6.2.8 配置works

vi works-------------------------------------------singleNode-------------------------------------------

6.3 添加變數

vi ~/.bashrc------------------------------------------------export HADOOP_HOME=/opt/install/hadoopexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport PATH=$HADOOP_HOME/bin:$PATH------------------------------------------------vi $HADOOP_HOME/sbin/start-dfs.shvi $HADOOP_HOME/sbin/stop-dfs.sh------------------------------------------------HDFS_NAMENODE_USER=root HDFS_DATANODE_USER=root HDFS_SECONDARYNAMENODE_USER=root YARN_RESOURCEMANAGER_USER=root YARN_NODEMANAGER_USER=root------------------------------------------------vi $HADOOP_HOME/sbin/start-yarn.shvi $HADOOP_HOME/sbin/stop-yarn.sh------------------------------------------------YARN_RESOURCEMANAGER_USER=root HADOOP_SECURE_DN_USER=yarn YARN_NODEMANAGER_USER=root------------------------------------------------

6.4 HDFS格式化

hdfs namenode -format

6.5 啟動Hadoop服務

# 啟動HDFS$HADOOP_HOME/sbin/start-dfs.sh# 啟動yarn$HADOOP_HOME/sbin/start-yarn.sh# 啟動歷史服務器mapred --daemon start historyserver

6.6 Web端查看

查看9870埠

查看8088埠

七、Hive安裝

7.1 上傳并解壓

tar zxvf /opt/software/apache-hive-3.1.2-bin.tar.gz -C /opt/install/ln -s /opt/install/apache-hive-3.1.2-bin/ /opt/install/hive

7.2 修改配置

# 進入路徑cd /opt/install/hive/conf/

7.2.1 修改hive-site.xml

cp hive-default.xml.template hive-site.xmlvi hive-site.xml-------------------------------------------<configuration>    <property>        <name>javax.jdo.option.ConnectionURL</name>        <value>jdbc:mysql://singleNode:3306/metastore?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=UTF-8</value>    </property>    <property>        <name>javax.jdo.option.ConnectionDriverName</name>        <value>com.mysql.jdbc.Driver</value>    </property>    <property>        <name>javax.jdo.option.ConnectionUserName</name>        <value>root</value>    </property>    <property>        <name>javax.jdo.option.ConnectionPassword</name>        <value>root</value>    </property>    <property>        <name>hive.metastore.warehouse.dir</name>        <value>/user/hive/warehouse</value>    </property>    <property>        <name>hive.metastore.schema.verification</name>        <value>false</value>    </property>    <property>        <name>hive.metastore.uris</name>        <value>thrift://singleNode:9083</value>    </property>    <property>        <name>hive.server2.thrift.port</name>        <value>10000</value>    </property>    <property>        <name>hive.server2.thrift.bind.host</name>        <value>singleNode</value>    </property>    <property>        <name>hive.metastore.event.db.notification.api.auth</name>        <value>false</value>    </property></configuration>-------------------------------------------

7.2.2 修改hive-env.sh

cp hive-env.sh.template hive-env.shvi hive-env.sh-------------------------------------------HADOOP_HOME=/opt/install/hadoop-------------------------------------------

7.3 添加依賴包

cp /opt/software/mysql-connector-java-5.1.31.jar /opt/install/hive/lib/

7.4 添加環境變數

vi ~/.bashrc------------------------------------------------export HIVE_HOME=/opt/install/hiveexport PATH=$HIVE_HOME/bin:$PATH------------------------------------------------

7.5 啟動服務

# 初始化元資料表schematool -dbType mysql -initSchema# 啟動hiveserver2服務nohup hive --service hiveserver2 &#############報錯 Exception in thread "main" java.lang.NoSuchMethodError ################# jar 包沖突, 需要洗掉低版本包rm -rf /opt/install/hive/lib/guava-19.0.jarcp /opt/install/hadoop/share/hadoop/common/lib/guava-27.0-jre.jar /opt/install/hive/lib/

7.6 Jps查看

八、Sqoop安裝

8.1 上傳并解壓

tar zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/install/ln -s /opt/install/sqoop-1.4.7.bin__hadoop-2.6.0/ /opt/install/sqoop

8.2 修改sqoop-env.sh

cd /opt/install/sqoop/conf/cp sqoop-env-template.sh sqoop-env.shvi sqoop-env.sh-------------------------------------------#Set path to where bin/hadoop is availableexport HADOOP_COMMON_HOME=/opt/install/hadoop#Set path to where hadoop-*-core.jar is availableexport HADOOP_MAPRED_HOME=/opt/install/hadoop#Set the path to where bin/hive is availableexport HIVE_HOME=/opt/install/hive-------------------------------------------

8.3 添加依賴包

cp /opt/software/mysql-connector-java-5.1.31.jar /opt/install/sqoop/lib/cp /opt/software/commons-lang-2.6.jar /opt/install/sqoop/lib/cp /opt/software/java-json.jar /opt/install/sqoop/lib/

8.4 添加環境變數

vi ~/.bashrc------------------------------------------------export SQOOP_HOME=/opt/install/sqoopexport PATH=$SQOOP_HOME/bin:$PATH------------------------------------------------

8.5 查看版本

sqoop version

九、Flume安裝

9.1 上傳并解壓

tar zxvf /opt/software/apache-flume-1.9.0-bin.tar.gz -C /opt/install/cd /opt/install/ln -s apache-flume-1.9.0-bin/ flume

9.2 洗掉依賴

cd flume/rm -rf lib/guava-11.0.2.jar

9.3 添加環境變數

vi ~/.bashrc---------------------export FLUME_HOME=/opt/install/flumeexport PATH=$FLUME_HOME/bin:$PATH---------------------source ~/.bashrc

附錄

本文所使用的安裝包存放地址：
鏈接：https://pan.baidu.com/s/1F2lXHEP3-6_mDPp2L5oEjg
提取碼：vick