15/05/15 11:45:21 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, ANY, 1327 bytes)
15/05/15 11:45:21 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
15/05/15 11:45:21 INFO HadoopRDD: Input split: hdfs://192.168.105.226:9000/LesterDemoTest/Data/draft/patent.db:0+48437
15/05/15 11:45:22 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
org.elasticsearch.hadoop.rest.EsHadoopTransportException: Invalid target URI fe80:0:0:0:20c:29ff:feac:b7e2%2:9200
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.<init>(CommonsHttpTransport.java:173)
at org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:86)
at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:69)
at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:80)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:81)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:348)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:31)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:34)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:34)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.commons.httpclient.URIException: invalid port number
at org.apache.commons.httpclient.URI.parseAuthority(URI.java:2248)
at org.apache.commons.httpclient.URI.parseUriReference(URI.java:1978)
at org.apache.commons.httpclient.URI.<init>(URI.java:167)
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.<init>(CommonsHttpTransport.java:171)
... 14 more
15/05/15 11:45:22 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, localhost): org.elasticsearch.hadoop.rest.EsHadoopTransportException: Invalid target URI fe80:0:0:0:20c:29ff:feac:b7e2%2:9200
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.<init>(CommonsHttpTransport.java:173)
at org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:86)
at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:69)
at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:80)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:81)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:348)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:31)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:34)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:34)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.commons.httpclient.URIException: invalid port number
at org.apache.commons.httpclient.URI.parseAuthority(URI.java:2248)
at org.apache.commons.httpclient.URI.parseUriReference(URI.java:1978)
at org.apache.commons.httpclient.URI.<init>(URI.java:167)
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.<init>(CommonsHttpTransport.java:171)
... 14 more
15/05/15 11:45:22 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1 times; aborting job
15/05/15 11:45:22 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/05/15 11:45:22 INFO TaskSchedulerImpl: Cancelling stage 1
15/05/15 11:45:22 INFO DAGScheduler: Job 1 failed: runJob at EsSpark.scala:34, took 0.653925 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): org.elasticsearch.hadoop.rest.EsHadoopTransportException: Invalid target URI fe80:0:0:0:20c:29ff:feac:b7e2%2:9200
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.<init>(CommonsHttpTransport.java:173)
at org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:86)
at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:69)
at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:80)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:81)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:348)
at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:31)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:34)
at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:34)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.commons.httpclient.URIException: invalid port number
at org.apache.commons.httpclient.URI.parseAuthority(URI.java:2248)
at org.apache.commons.httpclient.URI.parseUriReference(URI.java:1978)
at org.apache.commons.httpclient.URI.<init>(URI.java:167)
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.<init>(CommonsHttpTransport.java:171)
... 14 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
uj5u.com熱心網友回復：
主要的問題就是：org.elasticsearch.hadoop.rest.EsHadoopTransportException: Invalid target URI fe80:0:0:0:20c:29ff:feac:b7e2%2:9200
在構造sparkConf設定的es.nodes，在spark運行時替換成了fe80:0:0:0:20c:29ff:feac:b7e2%2。
uj5u.com熱心網友回復：
已經解決，禁用掉es所在機器的ipv6就可以了轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/94313.html
標籤：Spark
上一篇：hadoop2.2啟動沒報錯，測驗wordcount就出問題了
下一篇：并行fp-growth圖解