Node and Cluster Maintenance

The following procedure explains how to power off a single-node or an entire cluster for maintenance depending on your ARTESCA configuration.

Single-node

For a single-node configuration, this procedure allows you to power off the node or to stop services depending on your needs.

Power Off the Node

Shut down the server.

[artesca-os@<NODE_NAME> ~]$ sudo systemctl poweroff

Stop Services

1. Disable Kubernetes services.

   [artesca-os@bootstrap ~]$
   

   sudo systemctl disable kubelet \
   && sudo systemctl disable containerd
   

2. Reboot the server.

   [artesca-os@bootstrap ~]$
   

   sudo systemctl reboot
   

3. Perform the maintenance operation on the server.

4. Enable Kubernetes services.

   [artesca-os@bootstrap ~]$
   

   sudo systemctl enable kubelet \
   && sudo systemctl enable containerd
   

5. Reboot the server.

   [artesca-os@bootstrap ~]$
   

   sudo systemctl reboot
   

Multi-node

For a multi-node configuration, this procedure allows you to power off a single-node, the entire cluster, or to stop services depending on your needs.

Power Off a Node

1. Relocate the node workload before powering it off so that all Pods running on top of it can be safely re-scheduled elsewhere.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
       drain --ignore-daemonsets --force \
       --delete-emptydir-data <NODE_NAME>
   

2. Power off the node.

   [artesca-os@<NODE_NAME> ~]$ sudo systemctl poweroff
   

3. Uncordon the node once it is up again.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
     uncordon <NODE_NAME>
   

Power Off the Entire Cluster

1. Ensure that the Data Services are switched to read-only mode.

   [artesca-os@bootstrap ~]$
   

   export MONGODB_SHARDSVR_PODS=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf \
   get pod -n zenko -l app.kubernetes.io/name=mongodb-sharded,app.kubernetes.io/instance=data-db,app.kubernetes.io/component=shardsvr \
   -o jsonpath='{.items[*].metadata.name}') && \
   export MONGODB_CONFIGSVR_PODS=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf \
   get pod -n zenko -l app.kubernetes.io/name=mongodb-sharded,app.kubernetes.io/instance=data-db,app.kubernetes.io/component=configsvr \
   -o jsonpath='{.items[*].metadata.name}') && \
   echo "Preparing to lock data services..." && \
   all_success=true && \
   for shard_pod in $MONGODB_SHARDSVR_PODS; do \
   response=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf exec -n zenko $shard_pod -c mongodb -- \
   bash -c "mongosh --quiet --eval 'db.fsyncLock()' -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin \$MONGODB_APP_DATABASE"); \
   ok_status=$(echo "$response" | grep -oP 'ok.*?([0-9]+(\.[0-9]+)?),?' | awk -F': ' '{print $2}' | awk -F',' '{print $1}'); \
   if [ "$ok_status" == "1" ]; then \
      echo "Lock successful for $shard_pod"; \
   else \
      echo "Lock failed for $shard_pod"; \
      all_success=false; \
   fi; \
   done && \
   for config_pod in $MONGODB_CONFIGSVR_PODS; do \
   response=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf exec -n zenko $config_pod -c mongodb -- \
   bash -c "mongosh --quiet --eval 'db.fsyncLock()' -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin \$MONGODB_APP_DATABASE"); \
   ok_status=$(echo "$response" | grep -oP 'ok.*?([0-9]+(\.[0-9]+)?),?' | awk -F': ' '{print $2}' | awk -F',' '{print $1}'); \
   if [ "$ok_status" == "1" ]; then \
      echo "Lock successful for $config_pod"; \
   else \
      echo "Lock failed for $config_pod"; \
      all_success=false; \
   fi; \
   done && \
   if $all_success; then \
   echo "Artesca may now be stopped."; \
   else \
   echo "An error occurred, please make sure Artesca is healthy before initiating a shutdown."; \
   fi
   

2. Power off all of the cluster’s nodes.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl exec -n kube-system -c salt-master \
     --kubeconfig /etc/kubernetes/admin.conf \
     $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
       get pods -n kube-system -l app=salt-master -o name) \
     -- salt '*' system.shutdown 1
   

   Note

   Upon restarting the cluster, the read-only mode is automatically deactivated.

   To revert changes, use the following command:

   [artesca-os@bootstrap ~]$
   

   export MONGODB_SHARDSVR_PODS=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf \
   get pod -n zenko -l app.kubernetes.io/name=mongodb-sharded,app.kubernetes.io/instance=data-db,app.kubernetes.io/component=shardsvr \
   -o jsonpath='{.items[*].metadata.name}') && \
   export MONGODB_CONFIGSVR_PODS=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf \
   get pod -n zenko -l app.kubernetes.io/name=mongodb-sharded,app.kubernetes.io/instance=data-db,app.kubernetes.io/component=configsvr \
   -o jsonpath='{.items[*].metadata.name}') && \
   echo "Preparing to unlock data services..." && \
   all_success=true && \
   for shard_pod in $MONGODB_SHARDSVR_PODS; do \
   response=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf exec -n zenko $shard_pod -c mongodb -- \
   bash -c "mongosh --quiet --eval 'db.fsyncUnlock()' -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin \$MONGODB_APP_DATABASE"); \
   ok_status=$(echo "$response" | grep -oP 'ok.*?([0-9]+(\.[0-9]+)?),?' | awk -F': ' '{print $2}' | awk -F',' '{print $1}'); \
   if [ "$ok_status" == "1" ]; then \
      echo "Unlock successful for $shard_pod"; \
   else \
      echo "Unlock failed for $shard_pod"; \
      all_success=false; \
   fi; \
   done && \
   for config_pod in $MONGODB_CONFIGSVR_PODS; do \
   response=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf exec -n zenko $config_pod -c mongodb -- \
   bash -c "mongosh --quiet --eval 'db.fsyncUnlock()' -u root -p \$MONGODB_ROOT_PASSWORD --authenticationDatabase admin \$MONGODB_APP_DATABASE"); \
   ok_status=$(echo "$response" | grep -oP 'ok.*?([0-9]+(\.[0-9]+)?),?' | awk -F': ' '{print $2}' | awk -F',' '{print $1}'); \
   if [ "$ok_status" == "1" ]; then \
      echo "Unlock successful for $config_pod"; \
   else \
      echo "Unlock failed for $config_pod"; \
      all_success=false; \
   fi; \
   done && \
   if $all_success; then \
   echo "Artesca data services are ready to accept writes."; \
   else \
   echo "An error occurred."; \
   fi
   

Relocate Workload and Stop Services

1. Relocate the node workload.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
       drain --ignore-daemonsets --force \
       --delete-emptydir-data <NODE_NAME>
   

2. Stop services.

   1. Disable Kubernetes services.

      [artesca-os@bootstrap ~]$
      

      sudo systemctl disable kubelet \
      && sudo systemctl disable containerd
      

   2. Reboot the server.

      [artesca-os@bootstrap ~]$
      

      sudo systemctl reboot
      

   3. Perform the maintenance operation on the server.

   4. Enable Kubernetes services.

      [artesca-os@bootstrap ~]$
      

      sudo systemctl enable kubelet \
      && sudo systemctl enable containerd
      

   5. Reboot the server.

      [artesca-os@bootstrap ~]$
      

      sudo systemctl reboot
      

3. Once the node is up again, uncordon it.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
     uncordon <NODE_NAME>