本人剛學Spark，有個最簡單的例子，但是我在spark中運行報以上錯誤，
package example
import com.typesafe.scalalogging.LazyLogging
import org.apache.spark.{SparkConf, SparkContext}
object SparkExample extends LazyLogging {
private val master = "spark://master:7077"
private val appName = "example-spark"
private val stopWords = Set("gif", "png")
def main(args: Array[String]): Unit = {
val conf = new SparkConf()
.setMaster(master)
.setAppName(appName)
val sc = new SparkContext(conf)
val lines = sc.textFile("file:/home/qiujl/access.log")
val wordsCount = WordsCount.count(sc, lines, stopWords)
val counts = wordsCount.collect().mkString("[", ", ", "]")
logger.info(counts)
}
}
報以下錯誤：
[qwwsl@master ~]$ spark-submit --master spark://master:7077 --num-executors 48 --driver-memory 2g --executor-memory 7g --executor-cores 3 /home/qiujl/example-spark-master.jar
18/05/30 10:27:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.ClassNotFoundException: example.SparkExample
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:235)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
uj5u.com熱心網友回復：
java.lang.ClassNotFoundException: example.SparkExample你的代碼沒有打入jar
uj5u.com熱心網友回復：
能具體點嗎？我剛自學，不太清楚uj5u.com熱心網友回復：
打包的時候要把代碼加進去，命令指明哪個class，--class example.SparkExampleuj5u.com熱心網友回復：
請問能具體點嗎？寫在代碼哪里？還是配置中？
uj5u.com熱心網友回復：
看情況是,你打包時,默認指定main class,具體你可以看下如何打包默認指定方法,另外spark可以默認本地執行的,不必非得提交到服務器,可以用val sparkConf = new SparkConf().setAppName("SparkTest").setMaster("local"),就可以在window開發環境下執行;你可以看下你打的包中的內容,能否找到該類,可以用jd-gui查看jar包.uj5u.com熱心網友回復：
submit的時候需要把jar包指定一下，同時把jar包上傳到服務器的spark下面。例如${SPARK_HOME}/bin/spark-submit --master spark://10.70.64.15:8080 --jars ./lib/*.jar --class org.apache.sparkexamples.SparkSQL2 ./jar/first.jar
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/50584.html
標籤：Spark
下一篇：ixgbevf: Unable to start - perhaps the PF Driver isn't up yet