最近在將hive中的臨時表資料insert overwrite到ods表中去的時候，設定為lzo的壓縮格式，報錯如下：
Caused by: java.lang.RuntimeException: native-lzo library not available
針對"native-lzo library not available" 例外即lzo安裝的例外，
查了很多帖子，最終整合了解決方法，
1.將 HADOOP-LZO-0.4.20放到 每個機器的 /HADOOP-2.7.2/SHARE/HADOOP/COMMON/ 目錄下
將編譯后得到的hadoop-lzo-0.4.20.jar放到我的三臺機器安裝了hadoop的share/hadoop/common目錄下，
2.修改各個HADOOP節點的HADOOP的組態檔:
cd /opt/module/hadoop-2.7.2/etc/hadoop
vi hadoop-env.sh
# Extra Java CLASSPATH elements. Optional.
export HADOOP_CLASSPATH="<extra_entries>:$HADOOP_CLASSPATH:${HADOOP_HOME}/share/hadoop/common"
#export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/opt/module/hadoop-2.7.2/lib/native:/opt/glibc-2.14/lib
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/opt/module/hadoop-2.7.2/lib/native
3.在 CORE-SITE.XML <-CONFIGURATION-> 添加以下配置
vi core-site.xml
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
4. 在 MAPRED-SITE.XML <-CONFIGURATION-> 添加以下配置
vi mapred-site.xml
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>LD_LIBRARY_PATH=/opt/module/hadoop-2.7.2/lzo/lib</value>
</property>
<property>
<name >mapreduce.reduce.env</name>
<value>LD_LIBRARY_PATH=/opt/module/hadoop-2.7.2/lzo/lib</value>
</property>
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/opt/module/hadoop-2.7.2/lzo/lib</value>
</property>
然后重啟hdfs和yarn，再執行insert overwrite，資料匯入成功，
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/263785.html
標籤：其他
下一篇：四十二、Kafka的介紹