Replication Topic Lag Alert

When a storage location is deleted, any Kafka consumer groups associated with replication to that location become inactive but remain in the system, potentially causing monitoring alerts due to accumulated lag.

Note

This cleanup is only necessary if the storage location was actively used in replication workflows. If no replication workflows were configured to use this location, you can skip this section.

1. Verify the replication Kafka consumer group exists.

   Important

   In the following commands, replace <LOCATION_NAME> with the name of the deleted storage location.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf run -i --tty --rm debug \
      --image=$(sudo crictl images | grep 'kafka ' | grep zenko | sort | tail -1 | awk '{print $1":"$2}') \
      --restart=Never -n zenko -- bash -c '/opt/kafka/bin/kafka-consumer-groups.sh \
      --bootstrap-server artesca-data-base-queue-0:9092 --list | grep "backbeat-replication-group-<LOCATION_NAME>"'
   

   This should return a single consumer group with the format: <INSTANCE_ID>.backbeat-replication-group-<LOCATION_NAME>

2. Wait for the consumer group to become empty.

   Important

   In the following commands, replace <CONSUMER_GROUP_NAME> with the full name of the consumer group obtained from the previous command.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf run -i --tty --rm debug \
      --image=$(sudo crictl images | grep 'kafka ' | grep zenko | sort | tail -1 | awk '{print $1":"$2}') \
      --restart=Never -n zenko -- bash -c '
      TIMEOUT=600  # 10 minutes timeout
      ELAPSED=0
   
      while [ $ELAPSED -lt $TIMEOUT ]; do
      MEMBERS=$(/opt/kafka/bin/kafka-consumer-groups.sh \
      --bootstrap-server artesca-data-base-queue-0:9092 \
      --describe --group <CONSUMER_GROUP_NAME> \
      --state 2>/dev/null | tail -n +3 | awk "{print \$NF}")
   
      if [ "$MEMBERS" = "0" ]; then
         echo "Consumer group is empty and ready for deletion"
         exit 0
      else
         echo "Consumer group state: $MEMBERS. Waiting 10 seconds... (${ELAPSED}s/${TIMEOUT}s elapsed)"
         sleep 10
         ELAPSED=$((ELAPSED + 10))
      fi
      done
   
      echo "Timeout reached after ${TIMEOUT} seconds. Consumer group may still have active members."
      echo "You may need to wait longer or manually retry the deletion command later."
      exit 1'
   

3. Delete the replication-related Kafka consumer group.

   Now that the consumer group is empty, you can safely delete it.

   Important

   Again, replace <CONSUMER_GROUP_NAME> with the full name of the consumer group.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf run -i --tty --rm debug \
      --image=$(sudo crictl images | grep 'kafka ' | grep zenko | sort | tail -1 | awk '{print $1":"$2}') \
      --restart=Never -n zenko -- bash -c '/opt/kafka/bin/kafka-consumer-groups.sh \
      --bootstrap-server artesca-data-base-queue-0:9092 --delete \
      --group <CONSUMER_GROUP_NAME>'
   

Important

Only delete consumer groups that are specifically related to the deleted storage location. Deleting the wrong consumer groups can disrupt active replication workflows to other locations.