我有資料框：
columns = ['id', 'text']
vals = [
(1, 'I am good or'),
(2, 'You are okey'),
(3, 'She is fine in')
]
df = spark.createDataFrame(vals, columns)
--- --------------
| id| text|
--- --------------
| 1| I am good or|
| 2| You are okey|
| 3|She is fine in|
--- --------------
我只想在長度小于 3 的情況下洗掉最后一個單詞。我嘗試這樣做，但我不明白如何強加一個條件，以便長度檢查僅針對最后一個標記。我想設定一個非固定長度的條件。
df.withColumn('split_text', F.split(F.col("text")," "))\
.withColumn("split_text", F.expr("filter(split_text, x -> not(length(x) < 3))"))
我期待這個結果：
--- ------------
| id| text|
--- ------------
| 1| I am good |
| 2|You are okey|
| 3|She is fine |
--- ------------
uj5u.com熱心網友回復：
df.withColumn(
'split_text',
f.split(f.col("text")," "))\
.withColumn(
"split_text",
f.expr("if (length( split_text[size(split_text) - 1])<3, slice( split_text, 1 ,size(split_text) -1 ) ,split_text)")).show()
| id| text| split_text|
--- -------------- ----------------
| 1| I am good or| [I, am, good]|
| 2| You are okey|[You, are, okey]|
| 3|She is fine in| [She, is, fine]|
--- -------------- ----------------
expr explained in detail:
if (
length( split_text[size(split_text) - 1]) < 3, #conditional (Does the last item in the array have a length less than 3?
slice( # true condition
split_text, # array to splice
1 , #int - start of splice to keep
size(split_text) - 1 ) , #int - end of splice to keep
split_text #false condition
)
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/494564.html
標籤：阿帕奇火花 pyspark apache-spark-sql
上一篇：避免加載到空的表資料框中