val creation_timestamp = df.groupBy().agg(min($"userCreation_timestamp").alias("ts")).col("ts")
df.filter(col("userCreation_timestamp").cast("timestamp") >= creation_timestamp).show()
or
df.where(col("userCreation_timestamp").cast("timestamp") >= creation_timestamp).show()
運行上述代碼以獲取顯示資料時，代碼因 org.apache.spark.sql.AnalysisException: Resolved attribute(s) 而失敗。
org.apache.spark.sql.AnalysisException: Resolved attribute(s) ts#1658 missing from id#2,userCreation_timestamp#8,firstname#31 in operator !Filter (cast(userCreation_timestamp#8 as timestamp) >= ts#1658).;;
!Filter (cast(userCreation_timestamp#8 as timestamp) >= ts#1658)
- Relation[id#02,userCreation_timestamp#8, 26 more fields] parquet
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:41)
at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:92)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:293)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:84)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:84)
at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:92)
at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:172)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:178)
at org.apache.spark.sql.Dataset$.apply(Dataset.scala:65)
at org.apache.spark.sql.Dataset.withTypedPlan(Dataset.scala:3306)
at org.apache.spark.sql.Dataset.filter(Dataset.scala:1463)
... 49 elided
df.where(col("userCreation_timestamp").cast("timestamp") >= "2022-03-11 18:36:48").show()
在 where 子句中使用文字值，代碼作業正常，但是當使用資料幀時，它會失敗
uj5u.com熱心網友回復：
您可以先選擇min timestampas 值，然后在where/filter函式中使用該值。請找到以下作業示例：
import org.apache.spark.sql.functions.{col, lit}
import org.apache.spark.sql.types.TimestampType
import org.apache.spark.sql.{SparkSession, functions}
object QuestionStackOverflow extends App {
val spark = SparkSession.builder
.master("local[*]")
.appName("Sample App")
.config("spark.sql.shuffle.partitions", "1")
.getOrCreate()
import spark.sqlContext.implicits._
val df = Seq(
(1, "2022-03-11 18:36:48"),
(2, "2022-03-11 19:00:00"),
(3, "2022-03-11 20:00:00")
).toDF("id", "userCreation_timestamp")
.withColumn("ts", col("userCreation_timestamp").cast(TimestampType))
df.printSchema()
val creation_timestamp = df
.select(functions.min("ts"))
.head().get(0)
df.where(col("ts") > lit(creation_timestamp).cast(TimestampType))
.show()
}
架構是：
root
|-- id: integer (nullable = false)
|-- userCreation_timestamp: string (nullable = true)
|-- ts: timestamp (nullable = true)
輸出：
--- ---------------------- -------------------
| id|userCreation_timestamp| ts|
--- ---------------------- -------------------
| 2| 2022-03-11 19:00:00|2022-03-11 19:00:00|
| 3| 2022-03-11 20:00:00|2022-03-11 20:00:00|
--- ---------------------- -------------------
如果您對類似的問題感興趣Spark，請訪問我的博客：https ://bigdata-etl.com/tag/apache-spark/
uj5u.com熱心網友回復：
您收到此錯誤是因為您沒有ts在過濾條件中傳遞列的值，而是傳遞了列本身。由于資料框中ts不存在該列df，因此您會遇到AnalysisException: Resolved attribute(s) ts#1658 missing例外。
如果要傳遞列的值，則需要檢索聚合資料幀的第一行，然后檢索該行中的時間戳值，最后用于lit將其傳遞給您的條件：
import org.apache.spark.sql.functions.{min, lit, col}
val creation_timestamp = df.agg(min($"userCreation_timestamp")).head().getTimestamp(0)
df.filter(col("userCreation_timestamp").cast("timestamp") >= lit(creation_timestamp)).show()
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/461426.html