我正在為 Spark Structured Streaming 進行基于視窗的排序：
val filterWindow: WindowSpec = Window
.partitionBy("key")
.orderBy($"time")
controlDataFrame=controlDataFrame.withColumn("Make Coffee", $"value").
withColumn("datetime", date_trunc("second", current_timestamp())).
withColumn("time", current_timestamp()).
withColumn("temp_rank", rank().over(filterWindow))
.filter(col("temp_rank") === 1)
.drop("temp_rank").
withColumn("digitalTwinId", lit(digitalTwinId)).
withWatermark("datetime", "10 seconds")
我正在獲取timeascurrent_timestamp()并且在模式中我看到它的型別為StructField(time,TimestampType,true)
為什么 Spark 3.0 不允許我基于它進行視窗操作，但有以下例外，因為該檔案顯然是基于時間的？
21/11/22 10:34:03 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.
org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;;
Window [rank(time#163) windowspecdefinition(key#150, time#163 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS temp_rank#171], [key#150], [time#163 ASC NULLS FIRST]
- Project [key#150, value#151, Make Coffee#154, datetime#158, time#163]
uj5u.com熱心網友回復：
所以，根據答案https://issues.apache.org/jira/browse/SPARK-37439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
基于時間的視窗不只是基于時間戳格式列的 Window/WindowSpec。您應該session_window()為時間戳列明確使用函式。
http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#types-of-time-windows
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/369099.html