Jay Taylor's notes

back to listing index

What will happen to evicted pods in kubernetes? - Stack Overflow

[web search]
Original source (stackoverflow.com)
Tags: kubernetes quick-fix stackoverflow.com
Clipped on: 2020-02-06

Make your voice heard. Take the 2020 Developer Survey now.
Asked 2 years, 4 months ago
Active 16 days ago
Viewed 48k times
51

I just saw some of my pods got evicted by kubernetes. What will happen to them? just hanging around like that or I have to delete them manually?

asked Sep 26 '17 at 6:21
Image (Asset 1/17) alt=
Witnessing the same behavior, I have a pod that has been in Evicted state for 13 days now. Looks like evicted pods don't get removed (or maybe it is just a bug). – Elouan Keryell-Even Oct 23 '17 at 12:30
  • podgc controller will reclaim those Failed/Succeeded pods when a configurable threshold reached. – zhb Aug 7 '19 at 21:09
  • My Pods are evicted and there is a total of 40. So will I be charged per month for those evicted pods too? – Anant Sep 24 '19 at 2:42
    • 57

    A quick workaround I use, is to delete all evicted pods manually after an incident. You can use this command:

    kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | "kubectl delete pods \(.metadata.name) -n \(.metadata.namespace)"' | xargs -n 1 bash -c
    22

    To delete pods in Failed state in namespace default

    kubectl -n default delete pods --field-selector=status.phase=Failed
    answered Mar 25 '19 at 14:28
    Image (Asset 3/17) alt=
    Strangely, this doesn't show any when status.phase=Evicted. I managed to do this by kubectl -n default delete pods --field-selector=status.phase!=Running. But be careful, this would delete everything – n3o May 16 '19 at 8:29
  • 1
    I guess it can be useful to run kubectl -n default get pods --field-selector=status.phase=Failed at first. – username Aug 12 '19 at 9:43
    • 9

    Depending on if a soft or hard eviction threshold that has been met, the Containers in the Pod will be terminated with or without grace period, the PodPhase will be marked as Failed and the Pod deleted. If your Application runs as part of e.g. a Deployment, there will be another Pod created and scheduled by Kubernetes - probably on another Node not exceeding its eviction thresholds.

    Be aware that eviction does not necessarily have to be caused by thresholds but can also be invoked via kubectl drain to empty a node or manually via the Kubernetes API.

    answered Sep 26 '17 at 8:39
    Image (Asset 4/17) alt=
    yeah, my pod is from a deployment, and i do see another pod running on another node, but those previous evicted pods are also there – reachlin Sep 27 '17 at 3:12
  • How do you determine that are they "also there"? Which command exactly shows that to you? – Simon Tesar Sep 27 '17 at 4:39
  • just kubectl get pods -n mynamespace – reachlin Sep 28 '17 at 4:51
  • In which state? What does kubectl describe pod <pod> say? – Simon Tesar Sep 28 '17 at 14:41
  • 1
    Not OP but I have this issue. kubectl describe says "Status: Failed Reason: Evicted Message: Pod The node was low on resource: [MemoryPressure]." – Bryan Dec 15 '17 at 14:39
    • 9

    Evicted pods should be manually deleted. You can use following command to delete all pods in Error state.

    kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f -
    answered Feb 12 '19 at 11:18
    Image (Asset 5/17) alt=

    In case you have pods with a Completed status that you want to keep around:

    kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f -
    Image (Asset 6/17) alt=

    OpenShift equivalent of Kalvin's command to delete all 'Evicted' pods:

    eval "$(oc get pods --all-namespaces -o json | jq -r '.items[] | select(.status.phase == "Failed" and .status.reason == "Evicted") | "oc delete pod --namespace " + .metadata.namespace + " " + .metadata.name')"
    answered Mar 19 '18 at 12:22
    Image (Asset 7/17) alt=

    Just in the case someone wants to automatically delete all evicted pods for all namespaces:

    • Powershell
        Foreach( $x in (kubectl get po --all-namespaces --field-selector=status.phase=Failed --no-headers -o custom-columns=:metadata.name)) {kubectl delete po $x --all-namespaces }
    • Bash
    kubectl get po --all-namespaces --field-selector=status.phase=Failed --no-headers -o custom-columns=:metadata.name | xargs kubectl delete po --all-namespaces
    answered Jul 30 '19 at 6:35
    Image (Asset 8/17) alt=

    One more bash command to delete evicted pods

    kubectl get pods | grep Evicted | awk '{print $1}' | xargs kubectl delete pod
    answered Oct 11 '19 at 12:45
    Image (Asset 9/17) alt=

    Kube-controller-manager exists by default with a working K8s installation. It appears that the default is a max of 12500 terminated pods before GC kicks in.

    Directly from the K8s documentation: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/#kube-controller-manager

    --terminated-pod-gc-threshold int32     Default: 12500

    Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If <= 0, the terminated pod garbage collector is disabled.

    Image (Asset 10/17) alt=
    I have the kube-controller-manager pods on my master nodes. But how should I modify this flag? If I want to use kubectl edit pod kube-controller-manager-<master_name> -n kube-system it gives me pod is invalid error after saving the config file. – Ali Tou Aug 8 '19 at 19:27
    0

    Here is the 'official' guide for how to hard code the threshold(if you do not want to see too many evicted pods): kube-controll-manager

    But a known problem is how to have kube-controll-manager installed...

    Image (Asset 11/17) alt=
    Please advise on how the mentioned installation may be achieved, if it is troublesome. – MandyShaw Jul 30 '18 at 18:58
  • I do not know the answer either that is why I mentioned it. And OP did not mention the system he using and I do not know if he would have the same issue. BTW, downvote is SUPER NICE. – tikael Jul 30 '18 at 19:09
  • You would I think have done better adding your idea as a comment since it doesn't fully answer the question (which is why I downvoted it - sorry but it happens to us all, including me just now). – MandyShaw Jul 30 '18 at 19:13
  • check all the other answers above, OP asked what happened and how many of them did answer that and how many of them provide a way to delete the eviction pod? – tikael Jul 30 '18 at 19:15
    • -1

    below command will get all evicted pods from the default namespace and delete them

    kubectl get pods | grep Evicted | awk '{print$1}' | xargs -I {} kubectl delete pods/{}

    Image (Asset 12/17) alt=
    Code only answers are not encouraged, as it will not help like an answer with explained context that helps the community in long run. – Arun Vinoth Dec 20 '19 at 22:35

    Your Answer

    Community wiki

    Not the answer you're looking for? Browse other questions tagged or ask your own question.

    Hot Network Questions