Data Services MongoDB Is Not Ready After a Node Restart

Problem Description

After restarting one or several nodes in the MongoDB cluster, the MongoDB statefulsets are not ready.

Relevant Alerts and Errors

• Alert message:

  MongoDbDegraded: Less than 100% of MongoDb zenko/data-db-mongodb-sharded-shard0-data instances are up and healthy
  
  UnhealthyMemberWarning: Member data-db-mongodb-sharded-shardsrv-? (SECONDARY) is not healthy
  

• Error message:

  Rollback failed with unrecoverable error: UnrecoverableRollbackError: not
  willing to roll back more than 86400 seconds of data. Have: X seconds.
  

Root Cause

The issue is caused by a non-clean prolonged shutdown of the MongoDB cluster, while ARTESCA tolerates a maximum of one day of downtime, in this case.

Investigation Steps

1. Check the status of the mongodb pods.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -n zenko -l app.kubernetes.io/component=shardsvr,app.kubernetes.io/instance=data-db,app.kubernetes.io/name=mongodb-sharded
   
   Output Example

   NAME                                                              READY
   data-db-mongodb-sharded-configsvr-0                               2/2
   data-db-mongodb-sharded-configsvr-1                               2/2
   data-db-mongodb-sharded-configsvr-2                               2/2
   data-db-mongodb-sharded-mongos-0                                  2/2
   data-db-mongodb-sharded-mongos-1                                  2/2
   data-db-mongodb-sharded-mongos-2                                  2/2
   data-db-mongodb-sharded-shard0-data-0                             2/2
   data-db-mongodb-sharded-shard0-data-1                             1/2
   data-db-mongodb-sharded-shard0-data-2                             2/2
   

2. If the instances are not all completed (i.e. READY 2/2) , check their logs.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko logs data-db-mongodb-sharded-shard0-data-1 -c mongodb | grep UnrecoverableRollbackError
   

   If you get a relevant alert or error message, proceed with the following steps.

Problem Resolution

Warning

This procedure is a workaround that affects, during several minutes, the availability of the whole ARTESCA cluster. It is recommended to perform this procedure during a maintenance window.

1. Save the shard index of the failing MongoDB instance:

   export MONGODB_SHARD_ID=<MONGODB_SHARD_ID>
   

   For example, if the MongoDB pod name is data-db-mongodb-sharded-shard1-data-0, the shard index is 1.

2. Adjust the MongoDB configuration temporarily to allow for a successful restart.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf set env statefulset/data-db-mongodb-sharded-configsvr -n zenko MONGODB_EXTRA_FLAGS="--setParameter rollbackTimeLimitSecs=8640000" && \
   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf set env statefulset/data-db-mongodb-sharded-shard$MONGODB_SHARD_ID-data -n zenko MONGODB_EXTRA_FLAGS="--setParameter rollbackTimeLimitSecs=8640000" \
   && \
   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf delete pods -n zenko -l app.kubernetes.io/component=shardsvr,app.kubernetes.io/instance=data-db,app.kubernetes.io/name=mongodb-sharded,shard=$MONGODB_SHARD_ID && \
   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf delete pods -n zenko -l app.kubernetes.io/component=configsvr,app.kubernetes.io/instance=data-db,app.kubernetes.io/name=mongodb-sharded && \
   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf rollout status statefulset/data-db-mongodb-sharded-shard$MONGODB_SHARD_ID-data -n zenko --timeout=20m && \
   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf rollout status statefulset/data-db-mongodb-sharded-configsvr -n zenko --timeout=20m
   

3. Check that the MongoDB statefulsets are ready.

   Tip

   In the following command, replace <id> with the ID of the MongoDB pod. The default value is 0.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec data-db-mongodb-sharded-shard$MONGODB_SHARD_ID-data-<id> -n zenko -c mongodb -- \
   bash -c "mongosh --quiet \$MONGODB_APP_DATABASE -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin --eval 'rs.status()'"
   

   Ensure all members states are either PRIMARY or SECONDARY.

4. When the MongoDB statefulsets are ready, revert the configuration back to its original state.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf set env statefulset/data-db-mongodb-sharded-shard$MONGODB_SHARD_ID-data -n zenko MONGODB_EXTRA_FLAGS- \
   && \
   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf set env statefulset/data-db-mongodb-sharded-configsvr -n zenko MONGODB_EXTRA_FLAGS-