我正在嘗試計算 DataFrame 中列的標準偏差，但是在嘗試時我收到如下失敗訊息：
[info] - should return the standard deviation for all columns in a DataFrame *** FAILED *** (51 milliseconds)
[info] org.apache.spark.sql.AnalysisException: cannot resolve '`value_6`' given input columns: [stddev_samp(value_6)];
[info] 'Project ['value_6]
[info] - Aggregate [stddev_samp(value_6#131) AS stddev_samp(value_6)#151]
[info] - Project [coalesce(nanvl(value_6#60, cast(null as double)), cast(0 as double)) AS value_6#131]
[info] - Project [value_6#60]
[info] - Project [_1#39 AS id#54, _2#40 AS value_1#55, _3#41 AS value_2#56, _4#42 AS value_3#57, _5#43 AS value_4#58, _6#44 AS value_5#59, _7#45 AS value_6#60]
[info] - LocalRelation [_1#39, _2#40, _3#41, _4#42, _5#43, _6#44, _7#45]
[info] at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
[info] at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155)
[info] at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152)
[info] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341)
[info] at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
[info] at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:341)
[info] at org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformExpressionsUp$1(QueryPlan.scala:104)
[info] at org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:116)
[info] at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
[info] at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:116)
這是我所擁有的：
def standardDeviationForColumn(df: DataFrame, columnName: String): DataFrame =
df.select(columnName).na.fill(0).agg(stddev(columnName))
我是這樣稱呼它的：
assert(DataFrameUtils.standardDeviationForColumn(randomNumericTestData, "value_6").select("value_6").first().getDouble(0) === 1, "TODO")
我在這里做錯了什么？這是我的資料框：
val randomNumericTestData: DataFrame = Seq(
(1, 1, 10.0, 10.0,10.0,10.0,10.0),
(2, 0, 12.0, 12.0,12.0,12.0,12.0),
(3, 1, 13.0, 13.0,13.0,13.0,13.0),
(4, 1, 14.0, 14.0,14.0,14.0,14.0),
(5, 0, 12.5, 12.5,12.5,12.5,12.5),
(6, 1, 11.5, 11.5,11.5,11.5,11.5),
(7, 0, 17.5, 17.5,17.5,17.5,17.5),
(8, 0, 13.6, 13.6,13.6,13.6,13.6),
(9, 1, 14.2, 14.2,14.2,14.2,14.2)
).toDF("id", "value_1", "value_2", "value_3", "value_4", "value_5", "value_6")
uj5u.com熱心網友回復：
線索在錯誤訊息中：org.apache.spark.sql.AnalysisException: cannot resolve 'value_6' given input columns: [stddev_samp(value_6)];. Spark 找不到名為 的列value_6。
當您呼叫時，您會在輸出....agg(stddev(columnName))中獲得一個名為的新列。您需要重命名聚合列：stddev(columnName)DataFrame
def standardDeviationForColumn(df: DataFrame, columnName: String): DataFrame =
df.select(columnName).na.fill(0).agg(stddev(columnName) as columnName)
轉載請註明出處，本文鏈接：https://www.uj5u.com/net/429920.html
上一篇：用于spark-submit的kubernetes初始化容器
下一篇：更新資料塊上的jar作業