Platform Upgrade Did Not Complete

Problem Description

The ARTESCA upgrade, or the OS update that precedes it, stops during the pre-checks. Several messages lead to this page, listed below. On the OS update path, the prefix reads 'platform' os-update pre-checks instead of 'platform' upgrade pre-checks, and the rest is identical.

A platform upgrade was interrupted towards a version this installer does not deploy:

Error: Failure while running 'platform' upgrade pre-checks: The platform
upgrade to metalk8s 133.0.12 did not complete: the cluster is committed to
133.0.12 while ARTESCA records 132.0.2. This installer deploys metalk8s
134.0.1 and will not upgrade over an unfinished platform upgrade.

ARTESCA records a platform version ahead of the one the cluster is committed to, so its bookkeeping no longer describes the cluster:

Error: Failure while running 'platform' upgrade pre-checks: ARTESCA records
metalk8s 133.0.12 while the cluster is only committed to 132.0.2, so the
'artesca-version' ConfigMap does not describe this cluster. Check what the
nodes actually run before upgrading.

Two of the platform versions cannot be compared, one of them not being a version number at all:

Error: Failure while running 'platform' upgrade pre-checks: Unable to
compare the platform versions 133.0.12 and unknown, so whether the upgrade
can proceed cannot be established.

One of the two version sources cannot be read at all:

Error: Failure while running 'platform' upgrade pre-checks: Unable to read
the platform version the cluster is committed to: annotation
'metalk8s.scality.com/cluster-version' on the kube-system namespace is
missing or empty.

The upgrade was started without the platform step, on a cluster whose platform upgrade never completed:

Error: Failure while running 'platform' upgrade pre-checks: The platform
upgrade to metalk8s 133.0.12 did not complete: the cluster is already
committed to 133.0.12 while ARTESCA records 132.0.2. This run does not
complete it, and would act on the cluster regardless. Include the platform
step, or run 'sudo ./artesca-installer upgrade'.

Root Cause

ARTESCA records the deployed platform version in the artesca-version ConfigMap, and only once a platform upgrade has completed. The cluster itself is annotated with the platform version it is committed to, before the first node is upgraded.

A platform upgrade interrupted midway therefore leaves the two out of sync: the cluster runs a platform version ARTESCA does not know about. The installer reports it instead of starting a second platform upgrade on top of an unfinished one.

Important

Do not reboot the nodes and do not restart containerd until the platform upgrade has been completed. An interrupted upgrade can leave /etc/containerd/config.toml out of sync with the running container engine, in which case a restart breaks every image pull on the node.

Problem Resolution

Note

This procedure applies when the installer deploys a platform version different from the one the cluster is committed to, which is what the first error message above reports.

When the installer deploys the very version the cluster is committed to, no manual step is needed: run sudo ./artesca-installer upgrade, the pre-checks report the situation and the run completes the interrupted platform upgrade on its own. Only a run that leaves out the platform step is refused, since it would upgrade the application components on top of an unfinished platform.

Complete the interrupted platform upgrade first. In the commands below, replace <version> with the version the cluster is committed to, as named in the error message.

1. Complete the platform upgrade, on the bootstrap node.

   [artesca-os@bootstrap ~]$
   

   sudo /srv/scality/metalk8s-<version>/upgrade.sh
   

   The platform upgrade is idempotent: no node is skipped, and the steps that already succeeded are simply applied again.

2. Record the platform version, only if the previous command reported a success.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf patch \
   --namespace kube-system configmap artesca-version \
   --patch '{"data": {"metalk8s": "<version>"}}'
   

   Warning

   If upgrade.sh failed, do not run this command. ARTESCA would then consider the platform upgraded, skip the platform upgrade entirely and continue with the application components on an incomplete platform.

3. Resume the Upgrade process.

If upgrade.sh cannot be completed, or if the error message states that ARTESCA records a platform version ahead of the one the cluster is committed to, that the two versions cannot be compared, or that one of them cannot be read, collect the versions the nodes actually run:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes \
--label-columns metalk8s.scality.com/version

Then Gather System Information Using the sosreport Tool and share the report file with the Scality Support team via a ticket.