Overview
A pod has been deleted, and remains in a status of Terminated for more than a few seconds.
This can happen because:
the pod has a finalizer associated with it that is not completing, or
the pod is not responding to termination signals
Check RunBook Match
This runbook matches if pods have been deleted and remain in a Terminated state for a long time, or a time longer than is expected.
When running a kubectl get pods command, you will see a line like this in the output for your pod:
NAME READY STATUS RESTARTS AGE
nginx-7ef9efa7cd-qasd2 1/1 Terminating 0 1h
Initial Steps Overview
Detailed Steps
1) Gather information
kubectl get pod -n [NAMESPACE] -p [POD_NAME] -o yaml
2) Check for finalizers
First we check to see whether the pod has any finalizers. If it does, their failure to complete may be the root cause.
Get the pod’s configuration:
kubectl get pod -n [NAMESPACE] -p [POD_NAME] -o yaml > /tmp/runbooks_pod_configuration.txt
and look for a finalizers section under metadata. If any finalizers are present, then go to Solution A).
3) Check the status of the node
It is possible that the node your pod(s) is/are running on has failed in some way.
If you see from the /tmp/runbooks_pod_configuration.txt file that all pods on the same node are in a Terminating state on a specific node, then this may be the issue.
4) Delete the pod
The pod may not be terminating due to a process that is not responding to a signal. The exact reason will be context-specific and application dependent. Common causes include:
A tight loop in userspace code that does not allow for interrupt signals
A maintenance process (eg garbage collection) on the application runtime
In these cases, Solution B) may resolve the issue.
5) Restart kubelet
If nothing else works, it may be worth trying to restart the kubelet on the node the pod was trying to run on. See the output of
See solution C)
Solutions List
Solutions Detail
A) Remove finalizers
To remove any finalizers from the pod, run:
kubectl patch pod [POD_NAME] -p '{"metadata":{"finalizers":null}}'
B) Force-delete the pod
Please note that this is more of a workaround than a solution, and should be done with care to ensure that it won’t result in further problems. See also here for information pertaining to StatefulSets.
To force-delete the pod, run:
kubectl delete pod --grace-period=0 --force --namespace [NAMESPACE] [POD_NAME]
If this [does not work]{#check-resolution}, then return to the previous step.
C) Restart kubelet
If you can, SSH to the node and restart the kubelet process. If you do not have access or permission, this may require an administrator to get involved.
Before you do this (and if you have access), check the kubelet logs to see whether there are any issues in the kubelet logs.
Check Resolution
If the specific pod no longer shows up when running kubectl get pods
$ kubectl get pod -n mynamespace -p nginx-7ef9efa7cd-qasd2
NAME READY STATUS RESTARTS AGE
then the issue has been resolved.
Further Steps
If the issue recurs (or not), you may want to:
[Check whether the finalizer’s work needs to still be done]{#further-steps-1}
[Determine the root cause]{#further-steps-2}
1) Check whether the finalizer’s work needs to still be done
This will vary depending on what the finalizer did.
See further information for guidance on finalizers.
Common cases of finalizers not completing include:
- Volume
2) Determine the root cause
This will vary depending on what the finalizer did, and will require context-specific knowledge.
Some tips:
- If you have access, check the kubelet logs. Controllers can log useful information there.
Further Information
Unofficial Kubernetes Pod Termination
