Upgrade Failed Because MongoDB Feature Compatibility Version Is Not Valid¶

Problem Description¶

The ARTESCA upgrade failed and one of the following messages is displayed:

MongoDB is in a partially upgraded or downgraded state.

MongoDB feature compatibility version is not set to X.Y. Got A.B.

ARTESCA may upgrade MongoDB’s version when upgrading the Data Services. The MongoDB feature compatibility version is automatically upgraded to match the upgraded MongoDB version.

To ensure a successful upgrade, ARTESCA first ensures that the MongoDB feature compatibility version is set to the expected value. If the version is partially set or different from the expected value, the upgrade fails. In this situation, a manual operation must be performed to correct the issue.

Root Cause¶

MongoDB holds a different feature compatibility version than expected in the following scenarios:

The disk where MongoDB is running was replaced, or lost, and maintenance operations were done.
A previous upgrade failed, and MongoDB is not fully upgraded.

Problem Resolution¶

Before proceeding, ensure that all MongoDB instances are running properly.

Collect the status of the MongoDB instances:

[artesca-os@bootstrap ~]$

SHARDS_BASE_NAME="data-db-mongodb-sharded-shard"
shard_count=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts -n zenko | grep "$SHARDS_BASE_NAME" | wc -l)

for ((i=0; i<$shard_count; i++)); do
   SHARD_NAME="$SHARDS_BASE_NAME$i-data"
   replica_count=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts $SHARD_NAME -n zenko -o jsonpath='{.spec.replicas}')

   for ((j=0; j<$replica_count; j++)); do
      echo "Checking pod $SHARD_NAME-$j in namespace zenko"

      sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec $SHARD_NAME-$j \
      -n zenko -c mongodb -- bash -c \
      "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval \
      'printjson(rs.config()); printjson(rs.status()); printjson(rs.printReplicationInfo()); printjson(db.stats())'"

      echo "Completed checking pod $SHARD_NAME-$j"
      echo "------------------------------------------------"
   done
done

Ensure that all instances are either PRIMARY or SECONDARY, and that the stateStr field is not RECOVERING.

Collect the status of the Mongos instances:

[artesca-os@bootstrap ~]$

shard_server_instances=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts/data-db-mongodb-sharded-mongos -n zenko -o jsonpath={.spec.replicas})
for i in $(seq 0 $((shard_server_instances - 1))); do
   echo "Shard server $i:"
   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-mongos-$i -n zenko -c mongos -- \
      bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'sh.status()'"
done

Ensure that the number of active mongoses is equal to the number of ARTESCA nodes.

Check the status of the MongoDB Config Servers instances:

[artesca-os@bootstrap ~]$

CONFIGSVR_BASE_NAME="data-db-mongodb-sharded-configsvr"

replica_count=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts $CONFIGSVR_BASE_NAME -n zenko -o jsonpath='{.spec.replicas}')

for ((i=0; i<$replica_count; i++)); do
   echo "Checking StatefulSet pod $CONFIGSVR_BASE_NAME-$i in namespace zenko"

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec $CONFIGSVR_BASE_NAME-$i \
   -n zenko -c mongodb -- bash -c \
   "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval \
   'printjson(rs.config()); printjson(rs.status()); printjson(rs.printReplicationInfo()); printjson(db.stats())'"

   echo "Completed checking pod $CONFIGSVR_BASE_NAME-$i"
   echo "------------------------------------------------"
done

Ensure that all instances are either PRIMARY or SECONDARY, and that the stateStr field is not RECOVERING.

Pay special attention to any instance that is in a desynchronized state, as it may need additional troubleshooting. In some cases, you may have to Full Sync a MongoDB Member.

Once you have confirmed that all MongoDB instances are running properly, proceed with the steps below.

Get the expected MongoDB feature version:

[artesca-os@bootstrap ~]$

for shard in $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts -n zenko | grep "$SHARDS_BASE_NAME" | awk '{print $1}'); do
  echo "Checking expected version for shard: $shard"
  sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts/$shard -n zenko -o jsonpath='{.spec.template.spec.containers[0].image}'
done

The output will be similar to:

Expected Output¶

metalk8s-registry-from-config.invalid/zenko-base-X.Y.Z/mongodb-sharded:5.0.18-debian-11-r15

Here, the expected version is 5.0, and only includes the major and minor versions.

Ensure all MongoDB replicas are running.

Before proceeding, ensure all MongoDB replica pods are running with the same image version.
[artesca-os@bootstrap ~]$
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts -n zenko
The output will be similar to:
Expected Output¶
```
NAME                                  READY   AGE
data-db-mongodb-sharded-shard0-data   3/3     155d
```
If the number of READY pods is different from the number of replicas defined in the StatefulSet, wait for the pods to be running. If one or more pods are not running, you must troubleshoot the issue before proceeding.

Get the current MongoDB feature compatibility version:

[artesca-os@bootstrap ~]$

 for shard in $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts -n zenko | grep "$SHARDS_BASE_NAME" | awk '{print $1}'); do
   replica_count=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts/$shard -n zenko -o jsonpath='{.spec.replicas}')

   for ((j=0; j<$replica_count; j++)); do
      echo "Checking feature compatibility version for pod $shard-$j"

      sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec $shard-$j -n zenko -c mongodb -- bash -c \
      "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ).featureCompatibilityVersion'"
   done
done

The output will be similar to:

Expected Output¶

Shard server 0:
{ "version" : "4.4" }
Shard server 1:
{ "version" : "4.4" }
Shard server 2:
{ "version" : "4.2" }

Proceed with the procedure that matches your situation.

Current version is too old

The MongoDB version is too old (i.e you notice one or more reported version below the minimum required version). In that case, you must upgrade the feature compatibility version.

Define the feature compatibility version to be set. Refer to the MongoDB documentation for more information.
Replace the major and minor versions, and export them:

Note

For more information, refer to the Get the expected MongoDB feature version step.
[artesca-os@bootstrap ~]$
export MONGODB_FEATURE_VERSION="X.Y"

Apply the changes.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-mongos-0 -n zenko -c mongos -- \
bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval \
'db.adminCommand({ setFeatureCompatibilityVersion: \"${MONGODB_FEATURE_VERSION}\", confirm: true })'"

The output will be similar to:

Expected Output¶

{
    "ok" : 1,
    "operationTime" : Timestamp(1719316543, 2825),
    "$clusterTime" : {
        ...
    }
}

Retry the upgrade.

Current version has targetVersion

The setFeatureCompatibilityVersion command can fail and include the targetVersion field in two cases:

If a sharded cluster has a shard replica set that is read only when you run the setFeatureCompatibilityVersion command against the mongos.
If a replica set becomes read only while the setFeatureCompatibilityVersion command is running.

In any case, ensure that MongoDB is healthy and accepts writes. Check the MongoDB logs for more information or refer to the Mongodb documentation.