我一直在使用 Pyspark（我在我的機器上本地設定）來查詢一個大資料集，到目前為止效果很好。但是，我想為某些資料創建熊貓資料框。
SummaryStatList = []
for i in dataColumns:
SummaryStatList.append(df.groupby('Status').agg(func.min(df[i]).alias(i ' Min'),
func.max(df[i]).alias(i ' Max'),
func.mean(df[i]).alias(i ' Mean'),
func.stddev(df[i]).alias(i ' Variance'),
func.percentile_approx(i,0.5).alias(i ' Median')))
dfSummaryStat = pd.DataFrame(SummaryStatList)
但是在創建熊貓資料框時，我收到此錯誤：
raceback (most recent call last):
File "f:\Homework\UniCS\CS Year 3\Big Data\.env_BigData\BigDataAssingment.py", line 6, in <module>
from pyspark.sql.pandas._typing import PandasDataFrame
ImportError: cannot import name 'PandasDataFrame' from 'pyspark.sql.pandas._typing' (unknown location)
這些是匯入的庫：
import os
import findspark
import pandas as pd
import matplotlib
from pandas.core.frame import DataFrame
from pyspark.sql.pandas._typing import PandasDataFrame
import seaborn
import numpy as np
import sklearn
from pyspark.ml.feature import Imputer
from pyspark import SparkContext
from pyspark.sql import SQLContext
sc = SparkContext("local", "Big Data Assignment")
sql = SQLContext(sc)
import pyspark.sql.functions as func
import pyspark
from pyspark.sql import SparkSession
現在收到此錯誤：
f:\Homework\UniCS\CS Year 3\Big Data\.env_BigData\lib\site-packages\pyspark\sql\context.py:77: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
warnings.warn(
Traceback (most recent call last):
File "f:\Homework\UniCS\CS Year 3\Big Data\.env_BigData\lib\site-packages\findspark.py", line 143, in init
py4j = glob(os.path.join(spark_python, "lib", "py4j-*.zip"))[0]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "f:\Homework\UniCS\CS Year 3\Big Data\.env_BigData\BigDataAssingment.py", line 20, in <module>
findspark.init()
File "f:\Homework\UniCS\CS Year 3\Big Data\.env_BigData\lib\site-packages\findspark.py", line 145, in init
raise Exception(
Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly
不知道為什么因為它以前有效？
uj5u.com熱心網友回復：
試試這個
wget -q https://downloads.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz
或手動下載，然后解壓，然后將以下行添加到您的 python 檔案中
import os
os.environ["SPARK_HOME"] = "/path/to/file/spark-3.0.1-bin-hadoop3.2"
uj5u.com熱心網友回復：
只需使用 .toPandas()
pandas_df = spark_df.toPandas()
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/383926.html
上一篇：pyspark中不存在忽略路徑