Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 10.10.10.154): org.apache.hadoop.HadoopIllegalArgumentException: The short-circuit local reads feature is enabled but dfs.domain.socket.path is not set.
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: The short-circuit local reads feature is enabled but dfs.domain.socket.path is not set.
求解決辦法
uj5u.com熱心網友回復：
普通的wordcount案例,IntelliJ idea寫的打包成jar 使用bin/spark-submit \
--class WCount \
--master spark://is-nn-01:8888 \
--name wordcountByScala \
--executor-memory 1G \
--total-executor-cores 2 \
/spark_test/WordCount.jar \
hdfs://is-nn-01:8020/spark_test/tst.txt
提交后提示警告
uj5u.com熱心網友回復：
hdfs-site.xml中<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hdfs-sockets/dn</value>
</property>
已經設定過了
但是路徑中的dn檔案無法打開，與這個有關系嗎？
如果有關，那么正常的dn檔案應該是什么內容？長什么樣啊？
uj5u.com熱心網友回復：
worker節點是否配置dfs.domain.socket.pathuj5u.com熱心網友回復：
所有節點都配置好了uj5u.com熱心網友回復：
spark-submit 有個引數 --properties-file File Path to a file from which to load extra properties. If not specified, this will look for conf/spark-defaults.conf. 機器上將這個檔案注釋成temp了，系統無法查看引數，導致出現上述問題轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/78671.html
標籤：Spark