在 Kubernetes 上運行 Spark，給 3 個 Spark 作業執行緒中的每一個分配 8 個內核和 8G 記憶體，結果是
Executor app-xxx-xx/0 finished with state KILLED exitStatus 143
似乎無論計算多么簡單或我傳遞給什么標志spark-submit。
例如，
kubectl run -n redacted spark-client --rm -it --restart='Never' \
--image docker.io/bitnami/spark:3.2.0-debian-10-r2 \
-- run-example \
--name my-pi-calc-example-2 \
--master spark://spark-master-svc:7077 \
--deploy-mode cluster \
--driver-memory 4g \
--executor-memory 1g \
--driver-cores 4 \
--executor-cores 4 \
--verbose \
SparkPi 3
給我以下日志spark-worker-0：
21/11/15 22:07:42 INFO DriverRunner: Launch Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx4096M" "-Dspark.master=spark://spark-master-svc:7077" "-Dspark.driver.cores=4" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.driver.memory=4g" "-Dspark.executor.memory=4g" "-Dspark.submit.pyFiles=" "-Dspark.jars=file:///opt/bitnami/spark/examples/jars/scopt_2.12-3.7.1.jar,file:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.2.0.jar,file:/opt/bitnami/spark/examples/jars/spark-examples_2.12-3.2.0.jar" "-Dspark.rpc.askTimeout=10s" "-Dspark.app.name=my-pi-calc-example-2" "-Dspark.executor.cores=4" "org.apache.spark.deploy.worker.DriverWrapper" "spark://[email protected]:34637" "/opt/bitnami/spark/work/driver-20211115220742-0006/spark-examples_2.12-3.2.0.jar" "org.apache.spark.examples.SparkPi" "3" "--verbose"
21/11/15 22:07:44 INFO Worker: Asked to launch executor app-20211115220744-0006/4 for Spark Pi
21/11/15 22:07:44 INFO SecurityManager: Changing view acls to: spark
21/11/15 22:07:44 INFO SecurityManager: Changing modify acls to: spark
21/11/15 22:07:44 INFO SecurityManager: Changing view acls groups to:
21/11/15 22:07:44 INFO SecurityManager: Changing modify acls groups to:
21/11/15 22:07:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set()
21/11/15 22:07:44 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx4096M" "-Dspark.driver.port=42013" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@spark-worker-0.spark-headless.redacted.svc.cluster.local:42013" "--executor-id" "4" "--hostname" "xx.xx.19.190" "--cores" "4" "--app-id" "app-20211115220744-0006" "--worker-url" "spark://[email protected]:34637"
21/11/15 22:07:48 INFO Worker: Asked to kill executor app-20211115220744-0006/4
21/11/15 22:07:48 INFO ExecutorRunner: Runner thread for executor app-20211115220744-0006/4 interrupted
21/11/15 22:07:48 INFO ExecutorRunner: Killing process!
21/11/15 22:07:48 INFO Worker: Executor app-20211115220744-0006/4 finished with state KILLED exitStatus 143
21/11/15 22:07:48 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 4
21/11/15 22:07:48 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20211115220744-0006, execId=4)
21/11/15 22:07:48 INFO ExternalShuffleBlockResolver: Application app-20211115220744-0006 removed, cleanupLocalDirs = true
21/11/15 22:07:48 INFO Worker: Cleaning up local directories for application app-20211115220744-0006
21/11/15 22:07:48 INFO Worker: Driver driver-20211115220742-0006 exited successfully
我可以洗掉、更改或修改run-example或spark-submit標志。它似乎沒有效果，即使對于像SparkPi 3; 執行者被殺死并退出代碼 143，幾乎沒有關于他們被殺死的原因的資訊。
資源限制在這里不應該是任何問題。這是一個由 3 個 AWS m5.4xlarge 作業節點、16 個 vCPU 和 64GiB RAM 組成的 Kubernetes 集群，實際上幾乎沒有其他部署。我沒有將 Kubernetes 設定spec.resources為limits或requests。Spark集群部署如下：
argocd app create spark \
--repo https://charts.bitnami.com/bitnami \
--helm-chart spark \
--dest-server https://kubernetes.default.svc \
--insecure \
--helm-set 'worker.replicaCount=3' \
--dest-namespace redacted \
--revision '5.7.9' \
--helm-set worker.coreLimit=8 \
--helm-set worker.memoryLimit=8G \
--helm-set worker.daemonMemoryLimit=4G \
--helm-set master.daemonMemoryLimit=4G
argocd app sync spark
This uses the Spark Bitnami Helm chart and ArgoCD/Helm to deploy.
The cluster deploys just fine; I can see, for example, Starting Spark worker xxx.xx.xx.xx:46105 with 8 cores, 8.0 GiB RAM and that all 3 workers have joined.
What am I missing here? How can I better debug this and figure out what the resource constraint is?
Funny enough, I can even run SparkPi locally. If I e.g. kubectl exec -it spark-worker-0 -- bash:
$ ./bin/run-example SparkPi 3
...
21/11/15 22:22:09 INFO SparkContext: Running Spark version 3.2.0
...
21/11/15 22:22:11 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 0.634538 s
Pi is roughly 3.1437838126127087
Then I can add two arguments to run in cluster mode, and boom, executors get killed:
$ ./bin/run-example \
--master spark://spark-master-svc:7077 \
--deploy-mode cluster SparkPi
# Executor app-20211115222530-0008/2 finished with state KILLED exitStatus 143
uj5u.com熱心網友回復：
在這里學到了一些東西。第一個是 143 KILLED 似乎實際上并不表示失敗，而是執行器在作業完成后收到關閉信號。因此，在日志中發現時似乎很嚴厲，但事實并非如此。
讓我感到困惑的是，我在 stdout/stderr 上沒有看到任何“Pi 大約為 3.1475357376786883”的文本。這讓我相信計算從來沒有那么遠，這是不正確的。
這里的問題是我在這種情況下實際上更有意義--deploy-mode cluster時使用的--deploy-mode client。那是因為我正在運行一個臨時容器，kubectl run它不是現有部署的一部分。這更符合客戶端模式的定義，因為提交不是來自現有的 Spark worker。在 中運行時--deploy-mode=cluster，您實際上永遠不會看到 stdout，因為應用程式的輸入/輸出未連接到控制臺。
更改--deploy-mode為 后client，我還需要--conf spark.driver.host按照此處和此處的檔案進行添加，以便 pod 能夠決議回呼叫主機。
kubectl run -n redacted spark-client --rm -it --restart='Never' \
--image docker.io/bitnami/spark:3.2.0-debian-10-r2 \
-- /bin/bash -c '
run-example \
--name my-pi-calc-example \
--master spark://spark-master-svc:7077 \
--deploy-mode client \
--conf spark.driver.host=$(hostname -i) \
SparkPi 10'
輸出：
21/11/15 23:22:16 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
21/11/15 23:22:16 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 2.961188 s
Pi is roughly 3.140959140959141
21/11/15 23:22:16 INFO SparkUI: Stopped Spark web UI at http://xx.xx.xx.xx:4040
21/11/15 23:22:16 INFO StandaloneSchedulerBackend: Shutting down all executors
21/11/15 23:22:16 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
有趣的是app-20211115232213-0024，它在 Spark Master UI 中仍然顯示每個工人為 KILLED 143 - 強化了這是一個“正常”關閉信號的結論。
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/360763.html
標籤：java apache-spark kubernetes