我想測驗列中的值是否存在于常規 python dict 或 when().otherwise() 代碼塊中的 pyspark 映射中，但無法找出正確的語法。將有多個使用“計數”列排列的 when() 子句，因此需要類似于“if/elif/else”的內容。字典/地圖會很大，不會是資料框中的一列
from pyspark.sql import SparkSession, Row
import pyspark.sql.functions as F
from pyspark.sql.types import *
from itertools import chain
spark = (SparkSession
.builder
.getOrCreate())
data = [('Category A', 100, "This is category A"),
('Category B', 120, "This is category B"),
('Category C', 150, None)]
schema = StructType([
StructField('Category', StringType(), True),
StructField('Count', IntegerType(), True),
StructField('Description', StringType(), True)
])
rdd = spark.sparkContext.parallelize(data)
df = spark.createDataFrame(rdd, schema)
### Can either match regular python dict, or pyspark map ###
sec_lookup = {120: "new_category"}
sec_lookup_map = F.create_map(*[F.lit(x) for x in chain(*sec_lookup.items())])
user_df = df.withColumn(
"new_col",
F.when(
df["Count"].value in sec_lookup.keys(), <--- WHAT IS CORRECT SYNTAX?
F.concat(F.col("Category"), F.lit("_add"))
).when(
...
...
)
.otherwise(
F.concat(F.col("Category"), F.lit("_old"))
)
)
uj5u.com熱心網友回復：
我isin相信您不需要 if/elif/else 而 if/else 應該沒問題，因為您只需檢查字典鍵上的成員資格：
sec_lookup = {120: "new_category"}
df = df.withColumn("new",F.when(F.col("Count").isin([*sec_lookup.keys()])
,F.concat("Category",F.lit("_new"))).otherwise(
F.concat("Category",F.lit("_old"))))
df.show()
---------- ----- ------------------ --------------
| Category|Count| Description| new|
---------- ----- ------------------ --------------
|Category A| 100|This is category A|Category A_old|
|Category B| 120|This is category B|Category B_new|
|Category C| 150| null|Category C_old|
---------- ----- ------------------ --------------
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/461169.html