Sqoop(發音：skup)是一款開源的工具，主要用于在Hadoop(Hive)與傳統的資料庫間進行資料的傳遞，可以將一個關系型資料庫（例如 ： MySQL ,Oracle ,Postgres等）中的資料導進到Hadoop的HDFS中，也可以將HDFS的資料導進到關系型資料庫中，
官網：http://sqoop.apache.org/
由于官方不建議將sqoop 2用于生產環境，因為仍然使用sqoop 1
1、準備
下載
找到對應的鏡像地址安裝，復制下載地址
wget http://apache-mirror.8birdsvideo.com/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
解壓
tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
2、安裝
mv sqoop-1.4.7.bin__hadoop-2.6.0 /usr/local/
添加jdbc到sqoop的lib中:
cp mysql-connector-java-5.1.17.jar /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/lib/
3、配置
配置sqoop，加入path
vim /etc/profile
export SQOOP_HOME=/usr/local/sqoop-1.4.7.bin__hadoop-2.6.0export PATH=$SQOOP_HOME/bin:$PATH
配置sqoop-evn.sh
在sqoop的conf目錄下
cp sqoop-env-template.sh sqoop-env.shvim sqoop-env.sh
檔案內容增加
export HADOOP_COMMON_HOME=/lib/hadoop
export HADOOP_MAPRED_HOME=/lib/hadoop-mapreduce
export HIVE_HOME=/search/hive
export HIVE_CONF_DIR=/search/hive/conf
4、測驗
測驗與mysql的連通性
sqoop list-databases --connect jdbc:mysql://${HOST}:3306/test?characterEncoding=UTF-8 --username ${username} --password '$PASSWORD'Warning: /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.Please set $HCAT_HOME to the root of your HCatalog installation.Warning: /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.19/12/12 16:39:52 INFO sqoop.Sqoop: Running Sqoop version: 1.4.719/12/12 16:39:52 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.19/12/12 16:39:53 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.information_schematest
測驗從mysql中匯入到hive中
sqoop import \--connect jdbc:mysql://***:3306/test?characterEncoding=UTF-8 \--username ***\--password '***' \--table testxxx \-m 1 \--warehouse-dir /user/teemo/hive/warehouse/lyj_temp.db \--hive-database lyj_temp \--hive-import \--hive-table testxxx
運行結果:
進入hive，select 一下能看到結果
遇到得問題：
問題1
19/12/12 17:22:29 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/5c81c7e9c9d08472a6c1a1a272e4eeb3/testxxx.jarException in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/InputFormat at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:51) at com.cloudera.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:33) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:506) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252)Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.InputFormat at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 58 more
配置的HIVE_HOME這個變數有問題，應該是hadoop-mapreduce這個目錄
問題2
19/12/12 17:59:34 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50) at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392) at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252)Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44) ... 12 more
將hive的lib里面的拷貝到sqoop的lib目錄下
cp hive-exec-1.2.1.jar /usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/lib/
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/40000.html
標籤：大數據
上一篇：Hadoop學習之路(6)MapReduce自定義磁區實作
下一篇：[求助]pb里能不能壓縮字體