我有一個模式的資料框 -
|-- A: string (nullable = true)
|-- B: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- key: string (nullable = true)
| | |-- x: double (nullable = true)
| | |-- y: double (nullable = true)
| | |-- z: double (nullable = true)
|-- C: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- key: string (nullable = true)
| | |-- x: double (nullable = true)
| | |-- y: double (nullable = true)
我想合并列 B 和 C (array_union)。但是由于這些列的資料型別不同，array_union 不起作用。B & C 的結構除了 z 之外有幾乎相同的列。我不在乎 z - 無論它是否存在 - 在他們的合并輸出中。
什么是實作這一目標的好方法？
uj5u.com熱心網友回復：
當然，將 Z 放入 B 然后 array_join()
new = (df1.withColumn('B',expr("transform(B,s->struct(s.key as key,s.x as x, s.y as y))"))#drop Z
.withColumn('D', array_union(col('B'),col('C')))#array_join
.drop('B','C')#Drop B and C if not needed
).printSchema()
root
|-- A: string (nullable = false)
|-- D: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- key: string (nullable = true)
| | |-- x: double (nullable = true)
| | |-- y: double (nullable = true)
uj5u.com熱心網友回復：
像這樣轉換列'C'并在之后使用array_union：
import pyspark.sql.functions as f
df = (df
.withColumn('z', f.expr("transform(C, element -> cast(1 AS double))"))
.withColumn('C', f.expr("transform(C, (element, idx) -> struct(element_at(C.x, idx 1) AS x, element_at(C.y, idx 1) AS y, element_at(z, idx 1) AS z))"))
.drop('z')
)
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/493065.html
標籤：数据框 斯卡拉 阿帕奇火花 pyspark apache-spark-sql