這是我的 dataset
DataFrame[column1: double, column2: double, column3: int, column4: int, column5: int, ... , column300: int]
我想要的是
DataFrame[column1: double, column2: double, column3: double, column4: double, column5: double, ... , column300: double]
我做了什么
dataset.withColumn("column3", datalabel.column3.cast(DoubleType()))
這太手動了，你能告訴我怎么做嗎？
uj5u.com熱心網友回復：
您可以使用串列推導來構造轉換后的欄位串列。
import pyspark.sql.functions as F
...
cols = [F.col(field[0]).cast('double') if field[1] == 'int' else F.col(field[0]) for field in df.dtypes]
df = df.select(cols)
df.printSchema()
uj5u.com熱心網友回復：
您首先需要int從可用架構中篩選出您的列型別。
然后與reduce結合使用，您可以遍歷DataFrame以將它們轉換為您的選擇
reduce 是一個非常重要且有用的功能，通常可用于導航 Spark 中的任何迭代用例
資料準備
df = pd.DataFrame({
'id':[f'id{i}' for i in range(0,10)],
'col1': [i for i in range(80,90)],
'col2': [i for i in range(5,15)],
'col3': [6,7,5,3,4,2,9,12,4,10]
})
sparkDF = sql.createDataFrame(df)
sparkDF.printSchema()
root
|-- id: string (nullable = true)
|-- col1: long (nullable = true)
|-- col2: long (nullable = true)
|-- col3: long (nullable = true)
鑒別
sparkDF.dtypes
## [('id', 'string'), ('col1', 'bigint'), ('col2', 'bigint'), ('col3', 'bigint')]
long_double_list = [ col for col,dtyp in sparkDF.dtypes if dtyp == 'bigint' ]
long_double_list
## ['col1', 'col2', 'col3']
降低
sparkDF = reduce(lambda df,c: df.withColumn(c,F.col(c).cast(DoubleType()))
,long_double_list
,sparkDF
)
sparkDF.printSchema()
root
|-- id: string (nullable = true)
|-- col1: double (nullable = true)
|-- col2: double (nullable = true)
|-- col3: double (nullable = true)
轉載請註明出處，本文鏈接：https://www.uj5u.com/qukuanlian/345669.html