例如,創建如下所示的作業:
apiVersion: batch/v1
kind: Job
metadata:
name: test-job-sebas
spec:
template:
spec:
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: Never
backoffLimit: 4
導致創建作業資源但未觀察到 pod 或事件。Pod 狀態如下:
Pods Statuses: 1 Running / 0 Succeeded / 0 Failed
唯一可見的事件是創建成功 pod 的通知。問題是該訊息僅在完全明顯的沉默 30 分鐘后才出現。
Normal SuccessfulCreate 21m job-controller Created pod: test-job-sebas-882bh
從我們可以觀察到kube-apiserver日志允許 Job 資源的“創建”動詞時,我們無法在任何其他 pod(控制器/調度程式/apiserver)中發現任何其他日志,其中包含文本“test-job-sebas ",直到大約 30 分鐘后kube-controller-manager記錄以下內容。
Event occurred" object="test-namespace/job-test-01" kind="Job" apiVersion="batch/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: test-job-sebas-882bh"
這個集群中的任何 Job 都會發生這種情況,無論是哪個命名空間或 Job 的性質,如果它來自 CronJob 或者它是否像此處示例中提供的那樣顯式創建。
Looking at the code does not throw any obvious suspect for us that points to what could be happening: https://github.com/kubernetes/kubernetes/blob/b5b0cc8bb88fb678c9b065c8da4f4c06a155a628/pkg/controller/job/job_controller.go
edit: We currently have ~15.000 jobs in the cluster where it seems that most of them are active, from only one namespace. This would lead us to think that we are hitting some limit or making some sort of saturation...but we can't confirm this by any of the visible data.
uj5u.com熱心網友回復:
這聽起來與我在遇到行為不端的 webhook 時遇到的非常相似。
如果您有大量作業都顯示為活動,但沒有出現 pod,或者 pod 需要很長時間才能出現,那么這表明 admission webhook 干擾了 pod 創建。如果它是受影響的 cronjob,您將獲得“雪球”效果:
撰寫:https ://blenderfox.com/2020/08/07/the-snowball-effect-in-kubernetes/
Kubernetes 問題:https ://github.com/kubernetes/kubernetes/issues/93783
至于解決您的問題,您需要找出干擾創建的原因(在我們的例子中,我們有一個 up9 webhook 行為不端。禁用它允許創建 pod)
轉載請註明出處,本文鏈接:https://www.uj5u.com/shujuku/438939.html
標籤:kubernetes
