Internal Cloudserver Is Not Properly Scaled

Problem Description

The following issues are noticed:

• Important lag on the {ZENKO_ID}.backbeat-lifecycle-bucket-tasks topic.

• ListObject ECONNRESET errors in the bucket processor logs.

• MongoNetworkError errors in the internal Cloudserver.

Problem Resolution

Increase the Internal Cloudserver Number of Replicas

1. Edit the Zenko CR.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf edit zenko artesca-data -n zenko
   

2. In the Spec section, search for internalCloudserver: and edit the number of replicas depending on the platform and the size of the buckets.

   In the following example, there are 12 replicas.

   Spec:
     internalCloudserver:
       replicas: 12
   

Reset the Bucket Processor Kafka Offsets

This manual procedure enables to return to a stable state and to decrease the lag on the {ZENKO_ID}.backbeat-lifecycle-bucket-tasks topic.

1. Get the current number of replicas and save that number for later.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko \
   get deploy artesca-data-backbeat-lifecycle-bucket-processor \
   -o=jsonpath='{.status.replicas}'
   

2. Scale down the number of replicas to 0.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n \
   zenko scale deploy artesca-data-backbeat-lifecycle-bucket-processor --replicas=0
   

3. Get the Kafka image inside the cluster.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko \
   get kafkacluster -o jsonpath="{...image}" | grep -E -o "[^ ]*kafka:.*"
   

4. Spin a debug pod containing all Kafka scripts. Replace the <KAFKA_IMAGE> variable with the Kafka image from the previous step.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf run -i --tty \
   --rm debug-kafka --image=<KAFKA_IMAGE> --restart=Never -n zenko -- bash
   

5. In the debug pod, get the name of the topic.

   kafka-topics.sh --bootstrap-server artesca-data-base-queue-0:9092 --list | grep backbeat-lifecycle-bucket-tasks
   

   Example output

   c19d1d32-7c40-44c8-8fdd-babd5ab960ad.backbeat-lifecycle-bucket-tasks
   

   Note

   If you have multiple Kafka brokers, and not all of them are accessible, you can regroup all brokers using the following command:

   [artesca-os@bootstrap ~]$
   

   --bootstrap-server artesca-data-base-queue-0:9092,artesca-data-base-queue-1:9092,artesca-data-base-queue-2:9092
   

6. In the debug pod, get the name of the consumer group.

   kafka-consumer-groups.sh --bootstrap-server artesca-data-base-queue-0:9092 --list | grep backbeat-lifecycle-bucket-processor-group
   

   Example output

   c19d1d32-7c40-44c8-8fdd-babd5ab960ad.backbeat-lifecycle-bucket-processor-group
   

7. Reset the offsets. Replace the <GROUP> and <TOPIC> variables with the consumer group and topic names from the previous steps.

   kafka-consumer-groups.sh --bootstrap-server artesca-data-base-queue-0:9092 \
   --reset-offsets --to-latest --group <GROUP> --topic <TOPIC> --execute
   

8. Exit the debug pod.

9. Restart the bucket processor. Replace the <ORIGINAL_REPLICAS> variable with the number of replicas from step 1.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf -n zenko \
   scale deploy artesca-data-backbeat-lifecycle-bucket-processor --replicas=<ORIGINAL_REPLICAS>
   

10. Check in the Kafka dashboard that the lag on the {ZENKO_ID}.backbeat-lifecycle-bucket-tasks topic is now 0.

Note

Because Lifecycle Transition also uses the bucket processor, it can face the same issue.