Add a Data Services MongoDB Member Back to a Replica Set¶

Problem Description¶

The data upgrade prechecks fail with the following error message:

Error: Failure while running 'data' upgrade pre-checks: Command:
kubectl --kubeconfig=/etc/kubernetes/admin.conf exec data-db-mongodb-sharded-shard0-data-1 \
  --namespace zenko -- bash -c "mongosh --quiet $MONGODB_APP_DATABASE \
  -u root -p $MONGODB_ROOT_PASSWORD --authenticationDatabase admin \
  --eval 'JSON.stringify(rs.status())'"
failed with retcode 1:
MongoServerError: Authentication failed.
command terminated with exit code 1

Root Cause¶

A MongoDB shard member exists as a Kubernetes pod, but is missing from the MongoDB replica set configuration. The upgrade is not the root cause in this case. The precheck detects that the MongoDB cluster is already unhealthy and blocks the upgrade.

Investigation Steps¶

Confirm that the failing MongoDB pod is missing from the replica set configuration before applying the resolution. If another issue is found, do not apply this procedure.

Identify the failing MongoDB pod from the upgrade precheck error.

In the following example, the failing pod is data-db-mongodb-sharded-shard0-data-1:
```
kubectl --kubeconfig=/etc/kubernetes/admin.conf exec data-db-mongodb-sharded-shard0-data-1 \
  --namespace zenko -- bash -c "mongosh ..."
```

Save the failing pod in a variable and derive the shard StatefulSet name:

[artesca-os@bootstrap ~]$

export NAMESPACE=zenko
export KUBECONFIG=/etc/kubernetes/admin.conf
export FAILING_POD=data-db-mongodb-sharded-shard0-data-1
export SHARD_STS=${FAILING_POD%-*}

Confirm that the MongoDB connection fails in the affected pod:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig "$KUBECONFIG" exec "$FAILING_POD" -n "$NAMESPACE" -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'JSON.stringify(rs.status())'"

The command should return MongoServerError: Authentication failed.

Find the primary member of the same shard:

[artesca-os@bootstrap ~]$

export PRIMARY_POD=$(for pod in $(sudo kubectl --kubeconfig "$KUBECONFIG" get pods -n "$NAMESPACE" -o name | sed 's#pod/##' | grep "^${SHARD_STS}-" | grep -v "^${FAILING_POD}$"); do
   if sudo kubectl --kubeconfig "$KUBECONFIG" exec "$pod" -n "$NAMESPACE" -c mongodb -- \
      bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'db.hello().isWritablePrimary ? print(\"PRIMARY\") : print(\"not primary\")'" | grep -q PRIMARY; then
      echo "$pod"
      break
   fi
done)

if [ -z "$PRIMARY_POD" ]; then
   echo "ERROR: No primary member found. This procedure does not apply."
else
   echo "$PRIMARY_POD"
fi

If the command reports that no primary member was found, the replica set has no primary and this procedure does not apply. Contact Scality Support.

Confirm that the failing pod is missing from the replica set configuration:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig "$KUBECONFIG" exec "$PRIMARY_POD" -n "$NAMESPACE" -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval '
     const cfg = rs.conf();
     const missingHost = cfg.members.find(m => m.host.startsWith(\"${FAILING_POD}.\"));
     if (missingHost) {
       throw new Error(\"Pod is already present in replica set config: ${FAILING_POD}\");
     }
     print(\"Pod is missing from replica set config: ${FAILING_POD}\");
   '"

Continue only if the command confirms that the pod is missing from rs.conf().

Problem Resolution¶

Apply this procedure only when exactly one shard member is missing from the replica set configuration and the replica set still has a primary.

If the member cannot become SECONDARY after being added back, follow Full Sync a MongoDB Member instead.

Build the MongoDB host string to add back:

[artesca-os@bootstrap ~]$

export MISSING_HOST=$(sudo kubectl --kubeconfig "$KUBECONFIG" exec "$PRIMARY_POD" -n "$NAMESPACE" -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval '
     print(db.hello().me.replace(/${SHARD_STS}-[0-9]+/, \"${FAILING_POD}\"));
   '")

case "$MISSING_HOST" in
   *"$FAILING_POD"*":27017") echo "$MISSING_HOST" ;;
   *) echo "ERROR: Unexpected MongoDB host: $MISSING_HOST" ;;
esac

Add the member back with no votes first.

Add the member first with priority: 0 and votes: 0, then restore the values after it reaches SECONDARY.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig "$KUBECONFIG" exec "$PRIMARY_POD" -n "$NAMESPACE" -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval '
     rs.add({ host: \"${MISSING_HOST}\", priority: 0, votes: 0 });
     rs.conf().members.forEach(m => print(m.host + \" priority=\" + m.priority + \" votes=\" + m.votes));
   '"

Check the member state until it becomes SECONDARY:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig "$KUBECONFIG" exec "$PRIMARY_POD" -n "$NAMESPACE" -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval '
     rs.status().members.forEach(m => print(m.name + \" \" + m.stateStr));
   '"

Run the command again until the member becomes SECONDARY.

If the member stays in STARTUP2 or RECOVERING, or if it logs that it cannot sync from another member, follow Full Sync a MongoDB Member instead.

Restore the member votes and priority:

Note

This command applies to a default ARTESCA MongoDB deployment. If the MongoDB replica set was customized, adjust priority and votes to match the expected values for the deployment.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig "$KUBECONFIG" exec "$PRIMARY_POD" -n "$NAMESPACE" -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval '
     const cfg = rs.conf();
     const member = cfg.members.find(m => m.host === \"${MISSING_HOST}\");

     if (!member) {
       throw new Error(\"Missing member not found in replica set config: ${MISSING_HOST}\");
     }

     member.priority = 1;
     member.votes = 1;
     rs.reconfig(cfg);
   '"

Verify that the MongoDB connection now succeeds:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig "$KUBECONFIG" exec "$FAILING_POD" -n "$NAMESPACE" -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'JSON.stringify(rs.status())'"

Check that the member is part of rs.conf().members and that all replica set members are in PRIMARY or SECONDARY state:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig "$KUBECONFIG" exec "$PRIMARY_POD" -n "$NAMESPACE" -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval '
     const cfg = rs.conf();
     const status = rs.status();
     if (!cfg.members.some(m => m.host === \"${MISSING_HOST}\")) {
       throw new Error(\"Member is still missing from rs.conf(): ${MISSING_HOST}\");
     }
     const unhealthy = status.members.filter(m => ![\"PRIMARY\", \"SECONDARY\"].includes(m.stateStr));
     if (unhealthy.length > 0) {
       throw new Error(\"Unexpected replica set member state: \" + JSON.stringify(unhealthy));
     }
     status.members.forEach(m => print(m.name + \" \" + m.stateStr));
   '"

MongoDB is fixed when the command succeeds and all members are in PRIMARY or SECONDARY state. You can relaunch the upgrade. If the verification fails, contact Scality Support.