我知道您可以使用提供的輔助函式從訊息段中檢索子欄位，例如
val nameDf = df.select(segment_field("PID", 4).alias("name"))
有沒有一種方法可以提取整個訊息段（“PID”），而不必為每個子欄位輸入索引？
PID|||d40726da-9b7a-49eb-9eeb-e406708bbb60||Heller^Keneth||||||140 Pacocha Way Suite 52^^Northampton^Massachusetts^^USA
uj5u.com熱心網友回復：
segment_field這里實作的是一個輔助方法，用于提取給定索引的單個值。
/**
* Extracts a field from a message segment.
*
* @param segment The ID of the segment to extract.
* @param field The index of the field to extract.
* @param segmentColumn The name of the column containing message segments.
* Defaults to "segments".
* @return Yields a new column containing the field of a message segment.
*
* @note If there are multiple segments with the same ID, this function will
* select the field from one of the segments. Order is undefined.
*/
def segment_field(segment: String,
field: Int,
segmentColumn: Column = col("segments")): Column = {
filter(segmentColumn, s => s("id") === lit(segment))
.getItem(0)
.getField("fields")
.getItem(field)
}
但是，如果您有興趣提取所有欄位值而不考慮索引，您可以將此行為復制為
import org.apache.spark.sql.Column
import org.apache.spark.sql.functions._
def segment_fields(segment: String,
segmentColumn: Column = col("segments")): Column = {
filter(segmentColumn, s => s("id") === lit(segment))
.getItem(0)
.getField("fields")
}
并使用
val nameDf = df.select(segment_fields("PID").alias("values"))
然后，您可以根據需要提取或轉換資料。
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/372253.html