我在從 hdfs hive 背景關系中對資料框進行排序時遇到問題。嘗試對與此類似結構的資料框進行排序：
--- -------------- ---------------
| id|parameter_name|parameter_value
--- -------------- ---------------
|id1| name_en | value a
|id1| name_il | value b
|id1| address_en| value c
|id1| address_il| value d
|id2| name_il | value f
|id2| name_en | value e
|id2| address_il| value h
|id1| address_en| value g
--- -------------- ---------------
我試圖以 id 排序的方式對這個資料幀進行排序，每個 id 的 df 中的 parameter_name 序列如下：
name_en
name_il
address_en
address_il
請注意，在示例中情況并非如此，id 之間的名稱和地址被翻轉。
嘗試使用 df.sort(["id","parameter_name"]) 會產生混合結果，進一步混合資料幀并將 id 拆分為：
id1, name_en
id1, name_il
id2, name_il
id2, name_en
id1, address_en
id1, address_il
id2, address_il
id2, address_en
uj5u.com熱心網友回復：
我創建了您的資料框，但為其分配了隨機值，parameter_value因此訂單不再相關。
from random import random
data = [
{"id": "id1", "parameter_name": "name_en", "parameter_value": random()},
{"id": "id1", "parameter_name": "name_il", "parameter_value": random()},
{"id": "id1", "parameter_name": "address_en", "parameter_value": random()},
{"id": "id1", "parameter_name": "address_il", "parameter_value": random()},
{"id": "id2", "parameter_name": "name_il", "parameter_value": random()},
{"id": "id2", "parameter_name": "name_en", "parameter_value": random()},
{"id": "id2", "parameter_name": "address_il", "parameter_value": random()},
{"id": "id2", "parameter_name": "address_en", "parameter_value": random()},
]
df = spark.createDataFrame(data)
df.show()
--- -------------- -------------------
| id|parameter_name| parameter_value|
--- -------------- -------------------
|id1| address_il|0.11850447351294957|
|id2| name_en|0.18902815459657452|
|id2| address_il| 0.294998203578158|
|id1| address_en|0.48741740190944827|
|id2| name_il| 0.5651073044407224|
|id2| address_en| 0.6530661784882391|
|id1| name_il| 0.6797674631659714|
|id1| name_en| 0.9887386653580036|
--- -------------- -------------------
然后，我需要創建一個排序列來維護您需要的人工訂單：
from pyspark.sql import functions as F
ordering_col = (
F.when(F.col("parameter_name") == "name_en", 1)
.when(F.col("parameter_name") == "name_il", 2)
.when(F.col("parameter_name") == "address_en", 3)
.when(F.col("parameter_name") == "address_il", 4)
)
df.orderBy("id", ordering_col).show()
--- -------------- -------------------
| id|parameter_name| parameter_value|
--- -------------- -------------------
|id1| name_en| 0.9887386653580036|
|id1| name_il| 0.6797674631659714|
|id1| address_en|0.48741740190944827|
|id1| address_il|0.11850447351294957|
|id2| name_en|0.18902815459657452|
|id2| name_il| 0.5651073044407224|
|id2| address_en| 0.6530661784882391|
|id2| address_il| 0.294998203578158|
--- -------------- -------------------
uj5u.com熱心網友回復：
只需將 Pyspark 資料幀轉換為 Pandas 資料幀并執行排序操作。您可以sparkContext.createDataFrame(panda_df)像這樣將資料框轉換回：
panda_df = dataframe.toPandas().sort_values(["id", "parameter_name"], ascending=(True, False))
sorted_df = sparkContext.createDataFrame(panda_df)
sorted_df.show()
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/364017.html