Scaling Replication Services

Scaling the replication service enables it to keep up with the volume of operations on S3.

A ConsumerLagWarning alert on the replication topic will get triggered when lag on the replication queues exceeds a certain threshold for too long. This alert does not necessarily mean there is a scaling issue, it should only be used as a signal. Replication topic name is <UID>.backbeat-replication.

The Overview section of the replication dashboard can be used to further diagnose a not properly scaled replication service:

• Replication backlog size can be used to monitor lag on the replication queues.

• Replication Rate can be used to monitor the rate at which objects are being replicated.

If Objects are still being replicated but the lag is not decreasing, it is a telltale sign that the replication service is not scaled properly.

Start with a vertical scaling, and if the replication lag remains too high, proceed with a horizontal scaling.

Vertical Scaling

Vertical scaling aims at increasing the concurrency factors of the replication processors to enable the processing of more objects at a time.

You can incrementally increase concurrency factors until no more improvement is seen in the replication lag.

The following parameters can be adjusted to increase the concurrency of the replication processors:

• concurrency: The number of objects that can be processed in parallel by a single processor.

• mpuPartsConcurrency: The number of parts of a single multipart upload that can be replicated in parallel by a single processor.

Adjust these parameters.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf -n zenko patch \
zenko/artesca-data --type=merge \
-p '{"spec":{"backbeat":{"replicationDataProcessor":{"concurrency":<new_concurrency>,"mpuPartsConcurrency":<new_mpuPartsConcurrency>}}}}'

The default value for both concurrency and mpuPartsConcurrency is 10.

When increasing the concurrency factors, it is important to monitor the internal Cloudserver’s performance as it can increase the load on the S3 side.

If the internal Cloudserver cannot keep up with the increased load, you can experience high latencies and errors, which can trigger retries on the replication side.

You can monitor latencies and error rates in the Latency and Response codes sections of the S3 Service Dashboard.

If needed, you can scale the Internal Cloudserver by increasing the number of replicas:

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf -n zenko patch \
zenko/artesca-data --type=merge \
-p '{"spec":{"internalCloudserver":{"replicas":<internal_cloudserver_replicas>}}}'

Horizontal Scaling

Important

Horizontal scaling cannot apply to the ARTESCA+ Veeam Unified Software Appliance.

Horizontal scaling aims at increasing the number of replicas and the number of partitions of the Kafka topic used by the replication processors.

Horizontal scaling does not help decreasing the current replication lag, it only prevents it from building up further at the same rate. This is because each partition is consumed by at most one consumer, and when increasing the number of partitions, previous unread messages are not redistributed across the new partitions.

Kafka uses message keys to determine on which partition a message should be sent. Messages with the same key are sent to the same partition. This ensures that messages related to the same object are processed in the right order.

When new partitions are added, the distribution of message keys across the partitions can change. This can lead to temporarily undesirable effects such as the disruption of the message ordering.

It is recommended to increase the number of replicas gradually until the replication lag stabilizes to a low value. Setting a number of replicas higher than the number of partitions is ineffective, because the extra replicas will not be used.

1. Define the desired number of partitions and replicas.

   [artesca-os@bootstrap ~]$
   

   export DESIRED_REPLICATION_PARTITIONS=<number_of_partitions>
   export DESIRED_REPLICATION_PROCESSOR_REPLICAS=<number_of_replicas>
   

   It is recommended to slightly over-provision the number of partitions to limit the impact of the redistribution of new messages across the partition range.

   The default number of partitions is 5 and the default number of replicas is 1 per node.

2. Increase the number of partitions of the topic used by the replication processors.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf -n zenko patch \
   kafkatopic/artesca-data-backbeat-replication --type=merge \
   -p '{"spec":{"partitions":'"${DESIRED_REPLICATION_PARTITIONS}"'}}'
   

3. Increase the number of replicas of the replication processors.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf -n zenko patch \
   zenko/artesca-data --type=merge \
   -p '{"spec":{"backbeat":{"replicationDataProcessor":{"replicas":'"${DESIRED_REPLICATION_PROCESSOR_REPLICAS}"'}}}}'