如何在Sparkscala中將結構陣列添加到結構陣列的結構中

我有下面的例子

val df_temp1 = Seq(
  ("1","Adam","Angra", "Anastasia")
).toDF("id","fname", "mname", "lname")
df_temp1.createOrReplaceTempView("df_temp1")

val df1 = spark.sql("""select id,named_struct('opi1',array(named_struct('data_description','fname','data_details',fname),named_struct('data_description','mname','data_details',mname),named_struct('data_description','lname','data_details',lname))) as pi, array(named_struct('data_description','fname','data_details',fname),named_struct('data_description','mname','data_details',mname), named_struct('data_description','lname','data_details',lname)) as opi2 from df_temp1""")
df1.printSchema
df1.show(false)
df1.createOrReplaceTempView("df1")

這給出了以下輸出模式

root
 |-- id: string (nullable = true)
 |-- pi: struct (nullable = false)
 |    |-- opi1: array (nullable = false)
 |    |    |-- element: struct (containsNull = false)
 |    |    |    |-- data_description: string (nullable = false)
 |    |    |    |-- data_details: string (nullable = true)
 |-- opi2: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- data_description: string (nullable = false)
 |    |    |-- data_details: string (nullable = true)

下面的結果

 --- ----------------------------------------------------- --------------------------------------------------- 
|id |pi                                                   |opi2                                               |
 --- ----------------------------------------------------- --------------------------------------------------- 
|1  |{[{fname, Adam}, {mname, Angra}, {lname, Anastasia}]}|[{fname, Adam}, {mname, Angra}, {lname, Anastasia}]|
 --- ----------------------------------------------------- --------------------------------------------------- 

我希望將 opi2 與 opi1 一起包含在 pi 中，因此預期的架構應該如下所示

root
 |-- id: string (nullable = true)
 |-- pi: struct (nullable = false)
 |    |-- opi1: array (nullable = false)
 |    |    |-- element: struct (containsNull = false)
 |    |    |    |-- data_description: string (nullable = false)
 |    |    |    |-- data_details: string (nullable = true)
 |----|-- opi2: array (nullable = false)
 |    |    |-- element: struct (containsNull = false)
 |    |    |--   |--data_description: string (nullable = false)
 |    |    |---- |--data_details: string (nullable = true)

預期輸出將是 pi 內的兩個陣列 opi1 和 opi2 ，如下所示

 --- ----------------------------------------------------- --------------------------------------------------- 
|id |pi                                                                                                |
 --- ----------------------------------------------------- --------------------------------------------------- 
|1  |{[{fname, Adam}, {mname, Angra}, {lname, Anastasia}],[{fname, Adam}, {mname, Angra}, {lname, Anastasia}]}|
 --- ----------------------------------------------------- --------------------------------------------------- 

所以基本上將現有列添加到結構中（順便說一下，我使用的是 Spark 2.3，因此無法使用 Spark 2.4 中的任何函式）

uj5u.com熱心網友回復：

只需從pi.opi1and創建一個新結構opi2

val df2 = spark.sql("select id, named_struct('opi1',pi.opi1, 'opi2', opi2) as pi from df1")

df2.show(false)
df2.printSchema

 --- ---------------------------------------------------------------------------------------------------------- 
|id |pi                                                                                                        |
 --- ---------------------------------------------------------------------------------------------------------- 
|1  |{[{fname, Adam}, {mname, Angra}, {lname, Anastasia}], [{fname, Adam}, {mname, Angra}, {lname, Anastasia}]}|
 --- ---------------------------------------------------------------------------------------------------------- 

root
 |-- id: string (nullable = true)
 |-- pi: struct (nullable = false)
 |    |-- opi1: array (nullable = false)
 |    |    |-- element: struct (containsNull = false)
 |    |    |    |-- data_description: string (nullable = false)
 |    |    |    |-- data_details: string (nullable = true)
 |    |-- opi2: array (nullable = false)
 |    |    |-- element: struct (containsNull = false)
 |    |    |    |-- data_description: string (nullable = false)
 |    |    |    |-- data_details: string (nullable = true)

標籤：斯卡拉 阿帕奇火花 apache-spark-sql

上一篇：無法使用PySpark在Databricks上與apachespark函式to_timestamp()連接并添加一列

下一篇：使用括號內的點來達到Jinja2中的可變深度JSON，如['object.subobject']？