Performance Tuning

Optimize High Number of Operations During Incremental Backups

When immutability is enabled, VBR extends the retention period of objects required for the backup chain during incremental runs. This is achieved using the PUT Object Retention API call on the bucket. An excessive number of these requests can overload the storage servers, potentially degrading overall backup performance.

The following settings can help control the number of bucket operations. Configuration options vary depending on whether you are using VMs or Veeam Agents to back up the buckets.

Note

VBR exposes two distinct settings:

  • Concurrent repository tasks: the number of backup tasks VBR runs in parallel against the repository. One task processes one VM disk stream.

  • S3 connection threads (S3ConcurrentTaskLimit registry value): the number of parallel S3 connections that each repository task opens to ARTESCA. The default is 64 threads per task.

Set the number of concurrent tasks in the repository settings in VBR so that the resulting number of active threads stays within the capacity of the cluster.

Best practices are to:

  • Keep the number of active threads targeting a single ARTESCA node below 512.

  • Not exceed 8 concurrent repository tasks per ARTESCA node.

  • Optionally, reduce the maximum number of active threads per repository task by setting the S3ConcurrentTaskLimit parameter to a value between 16 and 64, and adjust the number of concurrent tasks accordingly.

Important

Do not modify this setting unless instructed by Veeam support.

The following registry setting can be set on the VBR console server, however it will apply the setting to all gateways in the infrastructure. The registry setting can also be applied on each gateway separately as well.

Tip

Keep the value between 16 and 64, and size it so that the total number of active threads targeting a single ARTESCA node stays below 512.

reg add "HKLM\SOFTWARE\Veeam\Veeam Backup and Replication" /v
S3ConcurrentTaskLimit /t REG_DWORD /d 16 /f

Note

Setting for concurrent tasks in the repository is ignored if Veeam agents are the ones backing up to the repository.

Optimize Multi-Object Delete API Requests from VBR

Important

Do not modify this setting unless instructed by Veeam support.

During incremental backups, VBR concurrently writes new objects, extends retention for required objects, and deletes those no longer needed.

The Multi-Object Delete API calls can remove up to 1000 objects per request, which can significantly load the S3 metadata layer.

The following setting defines the number of objects per Multi-Object Delete call to optimize request performance. Apply these settings to the Veeam Backup Server, the Object Storage Gateway (if applicable), and the Veeam Data Mover/Proxy.

Set the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
S3MultiObjectDeleteLimit (DWORD)
1000

In /etc/VeeamAgentConfig file, set S3MultiObjectDeleteLimit = 1000.

Note

1000 is the maximum defined by S3 standard, decreasing it will cause more S3 operations, as the same amount of objects need to be deleted in more S3 operations. If you set the value to 1, then 1000x S3 operations are sent to the object storage to achieve the same.

Configure Immutability Settings on VBR

The immutability feature is designed to protect the most recent restore points. Extending immutability periods beyond the recommended window increases object interdependence, leading to a significantly higher volume of PUT Object Retention API calls.

The immutability period must be set at the repository level to match the retention of the daily backups, typically 14 to 30 days, and no more than 60 days.

Setting longer immutability periods at the repository level will lead to additional space consumption due to the interdependence of the objects. For more information, refer to this documentation.

The GFS restore points will remain immutable until its retention expires, no matter the immutability period set at the repository level. For example if you keep 3 yearly backups, these will remain immutable for 3 years.

If the immutability period needs to be set over 30 days, use a copy job from the performance tier to the capacity tier, and set a GFS retention policy.

For more information on GFS and immutability refer to How Immutability Works - User Guide for Microsoft Hyper-V.

Move Backup Files to Different Buckets

VBR’s SOBR (Scale Out Backup Repository) can be used to allow multiple repositories (buckets) used for a backup plan.

Below are the best practices to use multiple buckets using SOBR in Veeam Backup & Replication.

Note

These actions require using VBR that has been used to create the backups. It cannot be performed by platform administrators who have access to the ARTESCA servers.