Recover Single-Node ETCD Cluster from WAL Corruption¶

Warning

This procedure is only applicable to single-node clusters.

Problem Description¶

When a sudden poweroff of the node occurs, the etcd Write-Ahead Log (WAL) can become corrupted. Since there are no other nodes in the cluster, there is no way to recover the cluster from the corrupted WAL. This can prevent etcd from starting, which in turn prevents the Kubernetes API server and the Kubernetes control plane from starting.

Root Cause¶

Etcd’s WAL is a file-based log of all changes to the etcd database. It is used to make sure that all changes to the database are durable and can be recovered in case of a crash. When a sudden poweroff occurs, the WAL can become corrupted because the last write to the WAL may not have been fully flushed to a disk. This can cause the etcd database to become corrupted, which prevents etcd from starting.

Investigation Steps¶

Check that etcd is not running.

[artesca-os@bootstrap ~]$

sudo crictl ps -q --label io.kubernetes.container.name=etcd --label io.kubernetes.pod.namespace=kube-system --state Running


No output.

Check the logs from the latest etcd pod to see if there are any errors related to WAL corruption.

[artesca-os@bootstrap ~]$

sudo crictl logs $(sudo crictl ps -qa --label io.kubernetes.container.name=etcd --label io.kubernetes.pod.namespace=kube-system --state Exited | head -n 1)

{"level":"warn","ts":"2023-10-19T08:00:47.960Z","caller":"fileutil/fileutil.go:53","msg":"check file permission","error":"directory \"/var/lib/etcd\" exist, but the permission is \"drwxr-x---\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
{"level":"info","ts":"2023-10-19T08:00:47.964Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"4.171107ms"}
{"level":"warn","ts":"2023-10-19T08:00:47.964Z","caller":"wal/util.go:90","msg":"ignored file in WAL directory","path":"0000000000000000-0000000000000000.wal.broken"}
{"level":"info","ts":"2023-10-19T08:00:47.965Z","caller":"etcdserver/server.go:530","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"warn","ts":"2023-10-19T08:00:47.965Z","caller":"wal/util.go:90","msg":"ignored file in WAL directory","path":"0000000000000000-0000000000000000.wal.broken"}
{"level":"info","ts":"2023-10-19T08:00:47.965Z","caller":"etcdserver/raft.go:529","msg":"restarting local member","cluster-id":"0","local-member-id":"0","commit-index":0}
panic: cannot use none as id

Problem Resolution¶

Extract the latest backup to a temporary directory.

[artesca-os@<NODE_NAME> ~]$

last_backup=$(ls -1 /var/lib/metalk8s/backups/*.tar.gz | tail -n 1)
mkdir -p /tmp/recovery-workspace
tar -xzf $last_backup -C /tmp/recovery-workspace

Back up the current etcd Pod manifest.

[artesca-os@<NODE_NAME> ~]$

sudo cp /etc/kubernetes/manifests/etcd.yaml /tmp/recovery-workspace/etcd.yaml

Create a static pod to restore the etcd backup.

[artesca-os@<NODE_NAME> ~]$

cat <<EOF > /tmp/recovery-workspace/etcd-restore.yaml
apiVersion: v1
kind: Pod
metadata:
  name: etcd-restore-snap
  namespace: "kube-system"
spec:
  restartPolicy: Never
  containers:
    - name: etcd-restore-snap
      image: metalk8s-registry-from-config.invalid/metalk8s-133.0.16/metalk8s-utils:133.0.16
      imagePullPolicy: IfNotPresent
      command: ["sh", "-c"]
      args:
      - |
        rm -rf /var/lib/etcd/* && rm -rf /var/lib/etcd/.* 2>/dev/null
        ETCDCTL_API=3 etcdctl --cert /etc/kubernetes/pki/etcd/salt-master-etcd-client.crt --key /etc/kubernetes/pki/etcd/salt-master-etcd-client.key --cacert /etc/kubernetes/pki/etcd/ca.crt snapshot restore /tmp/recovery-workspace/etcd_snapshot --data-dir /var/lib/etcd && touch /tmp/recovery-workspace/success
      volumeMounts:
        - mountPath: /var/lib/etcd
          name: etcd-data
          readOnly: False
        - mountPath: /etc/kubernetes/pki/etcd
          name: etcd-certs
          readOnly: True
        - mountPath: /tmp/recovery-workspace
          name: recovery-workspace
          readOnly: False
  hostNetwork: true
  volumes:
     - hostPath:
         path: /var/lib/etcd
         type: DirectoryOrCreate
       name: etcd-data
     - hostPath:
         path: /etc/kubernetes/pki/etcd
         type: Directory
       name: etcd-certs
     - hostPath:
         path: /tmp/recovery-workspace
         type: Directory
       name: recovery-workspace
EOF
sudo cp -f /tmp/recovery-workspace/etcd-restore.yaml /etc/kubernetes/manifests/etcd.yaml

Restart kubelet to pick up the new static pod.

[artesca-os@<NODE_NAME> ~]$

sudo systemctl restart kubelet

Wait for the restore pod to finish restoring.

[artesca-os@<NODE_NAME> ~]$

timeout 120s sh -c "until [ -f /tmp/recovery-workspace/success ]; do sleep 1s; done" \
   && echo "Restore succeeded" || echo "Restore failed"

Restore etcd.

[artesca-os@<NODE_NAME> ~]$

sudo cp -f /tmp/recovery-workspace/etcd.yaml /etc/kubernetes/manifests/etcd.yaml

Restart kubelet to pick up the static pod.

[artesca-os@<NODE_NAME> ~]$

sudo systemctl restart kubelet