Resolve Unschedulable Pods Due to Reserved Names

Problem Description

You notice that the pods are in a CreateContainerError status.

[artesca-os@bootstrap ~]$

sudo kubectl get pods -n kube-system \
--kubeconfig /etc/kubernetes/admin.conf

repositories-<NODENAME>            0/1     CreateContainerError
salt-master-<NODENAME>             0/2     CreateContainerError

Investigation Step

Using the command below, you can observe the following error messages:

[artesca-os@bootstrap ~]$

sudo journalctl --no-pager -u containerd | grep 'is reserved for'

Jun 06 07:57:13 <HOSTNAME> containerd[1886]: time="2024-06-06T07:57:13.283267810+02:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:salt-master-<NODENAME>,Uid:4deeb9d6fcd47817dc2f50a81d4100aa,Namespace:kube-system,Attempt:1,} failed, error" error="failed to reserve sandbox name \"salt-master-<NODENAME>_kube-system_4deeb9d6fcd47817dc2f50a81d4100aa_1\": name \"salt-master-<NODENAME>_kube-system_4deeb9d6fcd47817dc2f50a81d4100aa_1\" is reserved for \"5ef28bab21f20cd44df447c83a58675530690646cc14f20d33c8a91c8f1f56f0\""
Jun 06 07:57:13 <HOSTNAME> containerd[1886]: time="2024-06-06T07:57:13.377801308+02:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:repositories-<NODENAME>,Uid:b74c27c475feb29c8f89855865154f9a,Namespace:kube-system,Attempt:1,} failed, error" error="failed to reserve sandbox name \"repositories-<NODENAME>_kube-system_b74c27c475feb29c8f89855865154f9a_1\": name \"repositories-<NODENAME>_kube-system_b74c27c475feb29c8f89855865154f9a_1\" is reserved for \"28f48fbb4eb9e1062061805fa2c3b9276b54555094ca6cd571f2f433a4430f2c\""

Root Cause

This problem is caused by containerd not being able to reserve the pod name. The name is reserved for another pod, usually due to a time synchronization issue on the server.

Problem Resolution

You must delete the resources using the reserved names:

[artesca-os@bootstrap ~]$

sudo crictl rmp --all

Removed sandbox 28f48fbb4eb9e1062061805fa2c3b9276b54555094ca6cd571f2f433a4430f2c
Removed sandbox 5ef28bab21f20cd44df447c83a58675530690646cc14f20d33c8a91c8f1f56f0
pod sandbox "184899aa0290d61059ba2c6f662901dff3d3db63922ba37febc8ca05e66f2e2e" is running, please stop it first

The containers are then removed and the pods are rescheduled.

Note

The --all flag (without --force) is used to remove all non-running pods as seen on the command above.