Scale Async Metadata Updates

Follow this procedure to scale up the async metadata updates feature, allowing it to keep up with the volume of operations on RING S3C.

This procedure involves increasing the number of partitions of the Kafka topic related to the feature.

Important

This increase is permanent and cannot be decreased. If you need to increase the number of partitions by a large amount, it is recommended to do so in smaller increments until the desired number of partitions is reached.

1. Define the desired number of partitions and replicas.

   [artesca-os@bootstrap ~]$
   

   export DESIRED_INGESTION_PARTITIONS=<number of partitions>
   export DESIRED_INGESTION_PROCESSOR_REPLICAS=<number of replicas>
   

   Note

   It is common practice to set the number of DESIRED_INGESTION_PROCESSOR_REPLICAS equal to the number of DESIRED_INGESTION_PARTITIONS, as Kafka automatically assigns partitions based on the available replicas.

2. Increase the number of partitions for the topic used by async metadata updates.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf -n zenko patch \
   kafkatopic/artesca-data-backbeat-ingestion --type=merge \
   -p '{ "spec": { "partitions": '"${DESIRED_INGESTION_PARTITIONS}"' } }'
   

3. Increase the number of replicas for ingestion-processor deployment.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf -n zenko patch \
   deployment/artesca-data-backbeat-ingestion-processor --type=merge \
   -p '{ "spec": { "replicas": '"${DESIRED_INGESTION_PROCESSOR_REPLICAS}"' } }'