"
mysql table:
mysql> desc s_site_pv_uv;
+---------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+--------------+------+-----+---------+-------+
| thatday | varchar(100) | NO | | NULL | |
| site | varchar(500) | YES | | NULL | |
| site1 | varchar(500) | YES | | NULL | |
| pv | int(11) | YES | | NULL | |
| uv | int(11) | YES | | NULL | |
+---------+--------------+------+-----+---------+-------+
5 rows in set (0.00 sec)
hdfs data describe :
hadoop fs -cat /user/hdfs/test
2015-02-09 aaa bbbb 12 345
2015-02-23 edadfsdf eddd 23 34
under shell:
[hdfs@cdh162 log]$ export --connect jdbc:mysql://172.16.164.162:3306/jrj_log --username root --password jrj123456 --table s_site_pv_uv --export-dir /user/hdfs/test --input-fields-terminated-by '\t' --lines-terminated-by '\n'
bash: export: --: invalid option
export: usage: export [-fn] [name[=value] ...] or export -p
[hdfs@cdh162 log]$ sqoop export --connect jdbc:mysql://172.16.164.162:3306/jrj_log --username root --password jrj123456 --table s_site_pv_uv --export-dir /user/hdfs/test --input-fields-terminated-by '\t' --lines-terminated-by '\n'
Warning: /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/02/16 17:18:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.0
15/02/16 17:18:53 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/02/16 17:18:53 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
15/02/16 17:18:53 INFO tool.CodeGenTool: Beginning code generation
15/02/16 17:18:54 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `s_site_pv_uv` AS t LIMIT 1
15/02/16 17:18:54 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `s_site_pv_uv` AS t LIMIT 1
15/02/16 17:18:54 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
Note: /tmp/sqoop-hdfs/compile/ed405314b1f07f087740b4675f341edc/s_site_pv_uv.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
15/02/16 17:18:55 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hdfs/compile/ed405314b1f07f087740b4675f341edc/s_site_pv_uv.jar
15/02/16 17:18:55 INFO mapreduce.ExportJobBase: Beginning export of s_site_pv_uv
15/02/16 17:18:55 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/02/16 17:18:56 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
15/02/16 17:18:56 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
15/02/16 17:18:56 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/02/16 17:18:56 INFO client.RMProxy: Connecting to ResourceManager at cdh162.com/172.16.164.162:8032
15/02/16 17:18:58 INFO input.FileInputFormat: Total input paths to process : 1
15/02/16 17:18:58 INFO input.FileInputFormat: Total input paths to process : 1
15/02/16 17:18:58 INFO mapreduce.JobSubmitter: number of splits:4
15/02/16 17:18:58 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
15/02/16 17:18:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1423964363172_0047
15/02/16 17:18:58 INFO impl.YarnClientImpl: Submitted application application_1423964363172_0047
15/02/16 17:18:58 INFO mapreduce.Job: The url to track the job: http://cdh162.com:8088/proxy/application_1423964363172_0047/
15/02/16 17:18:58 INFO mapreduce.Job: Running job: job_1423964363172_0047
15/02/16 17:19:11 INFO mapreduce.Job: Job job_1423964363172_0047 running in uber mode : false
15/02/16 17:19:11 INFO mapreduce.Job: map 0% reduce 0%
15/02/16 17:19:20 INFO mapreduce.Job: map 100% reduce 0%
15/02/16 17:19:20 INFO mapreduce.Job: Job job_1423964363172_0047 completed successfully
15/02/16 17:19:20 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=527604
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=697
HDFS: Number of bytes written=0
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Launched map tasks=4
Data-local map tasks=4
Total time spent by all maps in occupied slots (ms)=20994
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=20994
Total vcore-seconds taken by all map tasks=20994
Total megabyte-seconds taken by all map tasks=21497856
Map-Reduce Framework
Map input records=2
Map output records=2
Input split bytes=526
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=139
CPU time spent (ms)=4630
Physical memory (bytes) snapshot=999120896
Virtual memory (bytes) snapshot=6377955328
Total committed heap usage (bytes)=2015363072
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
15/02/16 17:19:20 INFO mapreduce.ExportJobBase: Transferred 697 bytes in 23.8155 seconds (29.2667 bytes/sec)
15/02/16 17:19:20 INFO mapreduce.ExportJobBase: Exported 2 records.
the script is ok
the data is already export into mysql:
mysql> select * from s_site_pv_uv;
+------------+----------+-------+------+------+
| thatday | site | site1 | pv | uv |
+------------+----------+-------+------+------+
| 2015-02-23 | edadfsdf | eddd | 23 | 34 |
| 2015-02-09 | aaa | bbbb | 12 | 345 |
+------------+----------+-------+------+------+
2 rows in set (0.00 sec)
but when I use the hue's job designer to create a sqoop job,it tell me that:
Caused by: java.lang.RuntimeException: Can't parse input data: '2015-02-23 edadfsdf eddd 23 34'
at s_site_pv_uv.__loadFromFields(s_site_pv_uv.java:378)
at s_site_pv_uv.parse(s_site_pv_uv.java:306)
the job log:
2015-02-16 16:19:45,894 INFO [IPC Server handler 2 on 16047] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1423964363172_0046_m_000001_0: Error: java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.RuntimeException: Can't parse input data: '2015-02-09 aaa bbbb 12 345'
at s_site_pv_uv.__loadFromFields(s_site_pv_uv.java:378)
at s_site_pv_uv.parse(s_site_pv_uv.java:306)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:834)
at s_site_pv_uv.__loadFromFields(s_site_pv_uv.java:358)
... 12 more
uj5u.com熱心網友回復：
沒看懂，把前后背景發下uj5u.com熱心網友回復：
通過hue查看stderr日志檔案，是否缺少驅動了uj5u.com熱心網友回復：
是不是目錄沒有權限？轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/76753.html
標籤：數據倉庫