有一個簡單的程式，如下所示
import pyspark
builder = (
pyspark.sql.SparkSession.builder.appName("MyApp")
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.config(
"spark.sql.catalog.spark_catalog",
"org.apache.spark.sql.delta.catalog.DeltaCatalog",
)
)
spark = builder.getOrCreate()
spark._jsc.hadoopConfiguration().set(
"fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"
)
spark._jsc.hadoopConfiguration().set(
"fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
)
df = spark.read.format("delta").load(
"gs://org/delta/bronze/mongodb/registration/audits"
)
print(df.show())
使用以下 Dockerfile 將其打包到容器中
FROM varunmallya/spark-pi:3.2.1
USER root
ADD gcs-connector-hadoop2-latest.jar $SPARK_HOME/jars
WORKDIR /app
COPY main.py .
我認為這里的基本影像datamechanics/spark:3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17是關鍵。向把它放在一起的人的道具！
資料來源：https ://towardsdatascience.com/optimized-docker-images-for-apache-spark-now-public-on-dockerhub-1f9f8fed1665
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/483633.html
標籤：爪哇 斯卡拉 阿帕奇火花 Kubernetes pyspark