Dataset<Row> dataFrame = sparkSession.sql(“select * from product”);
dataFrame.toJavaRDD().foreachPartition(new VoidFunction<Iterator<Row>>() {
private static final long serialVersionUID = 1L;
@Override
public void call(Iterator<Row> rows) throws Exception {
try {
System.out.print("hello world");
} catch (Exception e) {
}
}
});
執行的時候報錯：
18/11/13 21:31:18 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Task not serializable
org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2101)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:925)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:924)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:924)
at org.apache.spark.api.java.JavaRDDLike$class.foreachPartition(JavaRDDLike.scala:219)
at org.apache.spark.api.java.AbstractJavaRDDLike.foreachPartition(JavaRDDLike.scala:45)
參照其他的帖子，我的函式里面沒有參考其他任何外部變數，為什么還會有這種報錯，百思不得其解，懇請大神指點
uj5u.com熱心網友回復：
這個網址 你自己看下吧,https://blog.csdn.net/achilles12345/article/details/77778431轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/45839.html
標籤：Spark