New Containers Cannot Start After a Node DiskPressure¶

Problem Description¶

When the root disk usage reaches more than 85% and the system is offline, the node goes in DiskPressure state and new containers cannot start.

Root Cause¶

A bug in MetalK8s makes kubelet delete some container images that are mandatory when the root disk usage reaches more than 85%. Kubelet considers the pause image as not mandatory and deletes it, but this image is required to run new containers.

Note

If your system is online and the node is no longer in DiskPressure (the root disk usage reaches less than 80%), the pause image will be downloaded from the Internet automatically.

Investigation Steps¶

Check the node status to know if the node is in DiskPressure, Make sure to replace the <node_name> placeholder.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf describe node <node_name>

[...]
Events:
    Type     Reason                 Age                From     Message
    ----     ------                 ----               ----     -------
    Normal   NodeHasDiskPressure    19m                kubelet  Node <node_name> status is now: NodeHasDiskPressure
    Warning  EvictionThresholdMet   18m (x4 over 19m)  kubelet  Attempting to reclaim ephemeral-storage
    Normal   NodeHasNoDiskPressure  13m (x2 over 41h)  kubelet  Node <node_name> status is now: NodeHasNoDiskPressure

Confirm that the pause image is not available.

[artesca-os@bootstrap ~]$

sudo crictl images | grep "pause"

Problem Resolution¶

Redeploy the pause image from MetalK8s using salt.

[artesca-os@bootstrap ~]$

sudo salt-call state.sls metalk8s.container-engine

Confirm that the pause image is now available.

[artesca-os@bootstrap ~]$

sudo crictl images | grep "pause"

k8s.gcr.io/pause    3.2     80d28bedfe5de    686kB