我有這樣的mydata：
data = [("110125","James","2021-12-05","NY","PA",60000),("110125","James","2021-12-07","NY","PA",3000),("110125","James","2021-12-07","NY","AT",3000),
("5225","Michael","2021-12-25","LA","AT",60000),("5225","Michael","2021-12-17","LA","PA",15000),("5225","Michael","2021-12-17","LA","PA",65000)]
columns = ["id","Name","Date","Local","Office","salary"]
df = spark.createDataFrame(data = data, schema = columns)
輸入：
-------- -------- ---------- ----- ------ ------
| id |Name |Date |Local|Office|salary|
-------- -------- ---------- ----- ------ ------
| 110125| James |2021-12-05|NY |PA | 60000|
| 110125| James |2021-12-07|NY |PA | 3000 |
| 110125| James |2021-12-07|NY |AT | 3000 |
| 5225 | Michael|2021-12-25|LA |AT | 60000|
| 5225 | Michael|2021-12-17|LA |PA | 15000|
| 5225 | Michael|2021-12-17|LA |PA | 65000|
-------- -------- ---------- ----- ------ ------
我想要一個新列' Check'，如果 4 個值Date, Local; Offfice; Salary中的一個與以前的值不同并且相同id, name，那么 True。
輸出：
-------- -------- ---------- ----- ------ ------ -----
| id |Name |Date |Local|Office|salary|Check|
-------- -------- ---------- ----- ------ ------ -----
| 110125| James |2021-12-05|NY |PA | 60000| |
| 110125| James |2021-12-07|NY |PA | 3000 | True|
| 110125| James |2021-12-07|NY |AT | 3000 | True|
| 5225 | Michael|2021-12-25|LA |AT | 60000| |
| 5225 | Michael|2021-12-17|LA |PA | 15000| True|
| 5225 | Michael|2021-12-17|LA |PA | 65000| True|
-------- -------- ---------- ----- ------ ------ -----
我的代碼 PySpark：
df.groupby("ID", "Name").withColumn("Check", F.when((F.col('Local') == F.lag('Local')) |(F.col('Office') == F.lag('Office'))|
(F.col('Date') == F.lag('Date'))|(F.col('salary') == F.lag('salary')), False ).otherwise(True))
AttributeError：“GroupedData”物件沒有屬性“withColumn”
uj5u.com熱心網友回復：
你想使用視窗：
from pyspark.sql import Window, functions as F
w = Window.partitionBy("id", "name").orderBy("Date")
df = df.withColumn(
"Check",
~((F.col('Local') == F.lag('Local').over(w))
& (F.col('Office') == F.lag('Office').over(w))
& (F.col('Date') == F.lag('Date').over(w))
& (F.col('salary') == F.lag('salary').over(w))
)
)
df.show()
# ------ ------- ---------- ----- ------ ------ -----
#| id| Name| Date|Local|Office|salary|Check|
# ------ ------- ---------- ----- ------ ------ -----
#|110125| James|2021-12-05| NY| PA| 60000| null|
#|110125| James|2021-12-07| NY| PA| 3000| true|
#|110125| James|2021-12-07| NY| AT| 3000| true|
#| 5225|Michael|2021-12-17| LA| PA| 15000| null|
#| 5225|Michael|2021-12-17| LA| PA| 65000| true|
#| 5225|Michael|2021-12-25| LA| AT| 60000| true|
# ------ ------- ---------- ----- ------ ------ -----
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/474823.html