Full Sync a MongoDB Member¶

Problem Description¶

To maintain up-to-date copies of the shared data set, secondary MongoDB members of a replica set automatically replicate data from a source member.

Root Cause¶

After a prolonged downtime period, or elevated load on ARTESCA, a MongoDB shard or config server member can fall behind, and cannot remain in the active cluster. In this case, the replica set becomes “stale”.

Before reaching this state, the ReplicationLagWarning alert may be created. This indicates the MongoDB instance at risk.

The member must be completely resynchronized with the rest of the replica set, by removing its data and performing an initial sync.

In this case, the replica member can be:

Running, but not active in the cluster.
In a CrashLoopBack state.

Apply the procedure in these situations:

The MongoDbRSNotSynced with MongoDbDegraded alerts are active.
A failing MongoDB instance reports the We are too stale to use candidate as a sync source log.
An ARTESCA upgrade fails with MongoServerError: Authentication failed.
The service disk replacement when the full RAID1 array was lost. In this case, both the shard server, and the config servers must be resynchronized.

Warning

Do not apply this procedure if you have a single server.

Problem Resolution¶

MongoDB Config Server Initial Sync

Warning

To prevent changing the write quorum, never rotate more than one configuration.

Locate the MongoDB member that is unable to restart.

[artesca-os@bootstrap ~]$

echo "### Starting health check for MongoDB Config Servers (configsvr)..."

# Get all StatefulSet names for the config servers
CONFIG_STS_NAMES=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get sts -n zenko -l app.kubernetes.io/instance=data-db,app.kubernetes.io/component=configsvr -o jsonpath='{.items[*].metadata.name}')

if [ -z "$CONFIG_STS_NAMES" ]; then
   echo "Warning: No StatefulSets found for config servers (configsvr)."
else
   # Loop through each config server StatefulSet
   for STS_NAME in $CONFIG_STS_NAMES; do
      echo "--- Checking StatefulSet: $STS_NAME ---"
      # Get the number of replicas for the current StatefulSet
      REPLICA_COUNT=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts "$STS_NAME" -n zenko -o jsonpath='{.spec.replicas}')

      # Loop through each pod in the StatefulSet
      for ((i=0; i<$REPLICA_COUNT; i++)); do
      POD_NAME="$STS_NAME-$i"
      echo "Checking pod '$POD_NAME'..."

      # Check if the pod is ready before executing the command
      IS_READY=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get pod -n zenko "$POD_NAME" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)

      if [ "$IS_READY" != "True" ]; then
         echo "  Skipping pod '$POD_NAME' as it is not in a 'Ready' state."
         continue
      fi

      # Execute the replica set status check inside the pod
      sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf exec -n zenko "$POD_NAME" -c mongodb -- \
         bash -c "mongosh --quiet --eval \"
            var status = rs.status();
            var unhealthyMembers = [];
            if (status && status.members) {
            status.members.forEach(function(m) {
               if (m.stateStr !== 'PRIMARY' && m.stateStr !== 'SECONDARY') {
                  unhealthyMembers.push('  ' + m.name + ' (state: ' + m.stateStr + ')');
               }
            });
            }
            if (unhealthyMembers.length > 0) {
            print('Unhealthy Replica Set Members Found:');
            unhealthyMembers.forEach(function(info) { print(info); });
            } else {
            print('All replica set members are healthy.');
            }
         \" -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin \$MONGODB_APP_DATABASE"
      done
      echo "------------------------------------------------"
   done
fi

echo "### Finished health check for MongoDB Config Servers."

Important

If you see All replica set members are healthy, the remainder of this procedure is not required.

Save the index in a variable for later use:

export MONGODB_REPLICA_ID=<MONGODB_REPLICA_ID>

Put a write lock on the MongoDB instance to prevent any new writes:

Note

Perform this step only if the pod is not in CrashLoopBack state. Otherwise the command will not work. This is expected and there is no impact to skipping this step.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-configsvr-$MONGODB_REPLICA_ID -n zenko -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'db.fsyncLock()'"

Delete all the data in the MongoDB instance:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko delete pod \
  data-db-mongodb-sharded-configsvr-$MONGODB_REPLICA_ID && \
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko wait \
  --for=condition=Ready pod/data-db-mongodb-sharded-configsvr-$MONGODB_REPLICA_ID --timeout=300s && \
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko exec \
  data-db-mongodb-sharded-configsvr-$MONGODB_REPLICA_ID -c mongodb -- \
  bash -c 'if [ "'"$MONGODB_REPLICA_ID"'" -eq 0 ]; then rm -rf /bitnami/mongodb/data/db/* && touch /bitnami/mongodb/data/db/delete_me; else rm -rf /bitnami/mongodb/data/db/*; fi'

Restart the MongoDB instance to initiate the resynchronization process:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf delete pod data-db-mongodb-sharded-configsvr-$MONGODB_REPLICA_ID -n zenko

Delete the temporary file created in the previous steps (only when MONGODB_REPLICA_ID is equal to 0):

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-configsvr-$MONGODB_REPLICA_ID -n zenko -c mongodb -- \
   bash -c 'rm -f /bitnami/mongodb/data/db/delete_me'

Check that the member has rejoined the replica set and completed the initial sync, after several minutes:

Note

The MongoDB instance may take several minutes before restarting, this is normal.

During the initial sync process, the MongoDB instance might be in the STARTUP2 state.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-configsvr-$MONGODB_REPLICA_ID -n zenko -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'rs.status()'"

The MongoDB instance must now be in the PRIMARY or SECONDARY state and fully synced with the replica set.

MongoDB Shard Server Initial Sync

Warning

To prevent changing the write quorum, never rotate more than one replica set member at a time.

Locate the MongoDB member that is unable to restart.

[artesca-os@bootstrap ~]$

echo "### Starting health check for MongoDB Shard Servers (shardsvr)..."

# Get all StatefulSet names for the shard servers
SHARD_STS_NAMES=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get sts -n zenko -l app.kubernetes.io/instance=data-db,app.kubernetes.io/component=shardsvr -o jsonpath='{.items[*].metadata.name}')

if [ -z "$SHARD_STS_NAMES" ]; then
   echo "Warning: No StatefulSets found for shard servers (shardsvr)."
else
   # Loop through each shard server StatefulSet
   for STS_NAME in $SHARD_STS_NAMES; do
      echo "--- Checking StatefulSet: $STS_NAME ---"
      # Get the number of replicas for the current StatefulSet
      REPLICA_COUNT=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get sts "$STS_NAME" -n zenko -o jsonpath='{.spec.replicas}')

      # Loop through each pod in the StatefulSet
      for ((i=0; i<$REPLICA_COUNT; i++)); do
      POD_NAME="$STS_NAME-$i"
      echo "Checking pod '$POD_NAME'..."

      # Check if the pod is ready before executing the command
      IS_READY=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get pod -n zenko "$POD_NAME" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)

      if [ "$IS_READY" != "True" ]; then
         echo "  Skipping pod '$POD_NAME' as it is not in a 'Ready' state."
         continue
      fi

      # Execute the replica set status check inside the pod
      sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf exec -n zenko "$POD_NAME" -c mongodb -- \
         bash -c "mongosh --quiet --eval \"
            var status = rs.status();
            var unhealthyMembers = [];
            if (status && status.members) {
            status.members.forEach(function(m) {
               if (m.stateStr !== 'PRIMARY' && m.stateStr !== 'SECONDARY') {
                  unhealthyMembers.push('  ' + m.name + ' (state: ' + m.stateStr + ')');
               }
            });
            }
            if (unhealthyMembers.length > 0) {
            print('Unhealthy Replica Set Members Found:');
            unhealthyMembers.forEach(function(info) { print(info); });
            } else {
            print('All replica set members are healthy.');
            }
         \" -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin \$MONGODB_APP_DATABASE"
      done
      echo "------------------------------------------------"
   done
fi

echo "### Finished health check for MongoDB Shard Servers."

Important

If you see All replica set members are healthy, the remainder of this procedure is not required.

Save the index in a variable for later use:

export MONGODB_REPLICA_ID=<MONGODB_REPLICA_ID>

Put a write lock on the MongoDB instance to prevent any new writes:

Note

Perform this step only if the pod is not in CrashLoopBack state. Otherwise the command will not work. This is expected and there is no impact to skipping this step.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-shard0-data-$MONGODB_REPLICA_ID -n zenko -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'db.fsyncLock()'"

Delete all the data in the MongoDB instance:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko delete pod \
  data-db-mongodb-sharded-shard0-data-$MONGODB_REPLICA_ID && \
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko wait \
  --for=condition=Ready pod/data-db-mongodb-sharded-shard0-data-$MONGODB_REPLICA_ID --timeout=300s && \
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko exec \
  data-db-mongodb-sharded-shard0-data-$MONGODB_REPLICA_ID -c mongodb -- \
  bash -c 'if [ "'"$MONGODB_REPLICA_ID"'" -eq 0 ]; then rm -rf /bitnami/mongodb/data/db/* && touch /bitnami/mongodb/data/db/delete_me; else rm -rf /bitnami/mongodb/data/db/*; fi'

Restart the MongoDB instance to initiate the resynchronization process:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf delete pod data-db-mongodb-sharded-shard0-data-$MONGODB_REPLICA_ID -n zenko

Delete the temporary file created in the previous steps (only when MONGODB_REPLICA_ID is equal to 0):

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-shard0-data-$MONGODB_REPLICA_ID -n zenko -c mongodb -- \
   bash -c 'rm -f /bitnami/mongodb/data/db/delete_me'

Check that the member has rejoined the replica set and completed the initial sync, after several minutes:

Note

The MongoDB instance may take several minutes before restarting, this is normal.

During the initial sync process, the MongoDB instance might be in the STARTUP2 state.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-shard0-data-$MONGODB_REPLICA_ID -n zenko -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'rs.status()'"

The MongoDB instance must now be in the PRIMARY or SECONDARY state and fully synced with the replica set.