Replication Monitoring

Access the Replication Dashboards

In Grafana, click Dashboards. Open the Data Services folder and click Replication.

Monitoring Components

Variables

Use the Location variable to filter the dashboard in showing only the data relevant to a specific replication location/site.

Use the Queue processor variable to select which queue processor to include in metrics/graphs reported in the Queue Processor section. By default, data from all processors is aggregated, and you can view the information related to the Data Processor or Replay Processor metrics.

Overview Section

Metric	Description
Queue Populator	Number of producers that are up and running.
Data Processor	Number of data processors that are up and running.
Replay Processor	Number of replay processors that are up and running.
Status Processor	Number of status processors that are up and running.
Zookeeper Quorum	Number of ZooKeeper quorum members (generally one per node).
Kafka brokers	Number of available Kafka brokers (generally one per node).
Offline partitions	Number of offline Kafka partitions. Should be 0 unless the Kafka cluster is not properly up and running.
Underreplicated partitions	Number of under-replicated Kafka partitions (less copies than expected in the Kafka cluster).
Replication Rate	Number of objects being replicated per second.
Replication Speed	Number of bytes being replicated per second.
Success rate	Proportion of objects that failed replication.
Replays	Number of objects being replayed per second.
Errors	Number of objects failing replication per second.
Replication backlog size	Number of objects pending replication (in replication topic, before they get picked up by the Data Processor).
Avg RPO	Average time for an object update/creation to be picked for replication by the Data Processor.
Avg latency	Average time taken for an object to replicate successfully to the destination.
Replication Rate by location	Number of objects being replicated to each location per second.
Replication Speed by location	Number of bytes being replicated to each location per second.
Replication RPO - 99 percentile	99 percentile of the time taken for an object update/creation to be picked for replication by the Data Processor.
Replication Latency - 99 percentile	99 percentile of the time taken for an object to replicate successfully to the destination.

Queue Populator Section

Metric	Description
Replication populator lag	Delay between the time an object is updated and the time it is picked up by Queue Populator. The precision is affected however by Prometheus polling interval (60 seconds), and delays up to this interval are expected.
Kafka injection rate	Number of messages posted to Kafka replication topic per second.
Queued objects count	Number of objects added to the Kafka replication topic.
Queued objects size	Cumulated size of objects added to the Kafka replication topic.
Queued objects count/s	Number of objects added to the Kafka replication topic per second.
Queued objects size/s	Cumulated size of objects added to the Kafka replication topic per second.

Queue Processor Section

Metric	Description
Data processor lag	Number of messages from Kafka replication topic which have not yet been processed, by location.
Replay processor lag	Number of messages from Kafka replication replay topic which have not yet been processed, by location.
Replication read rate by location	Number of objects being read by replication, by target location.
Replication write rate by location	Number of objects being written by replication, by target location.
Replication read speed by location	Number of bytes being read by replication, by target location.
Replication write speed by location	Number of bytes being written by replication of data, by target location.
Replication metadata write speed by location	Number of bytes being written by replication of metadata, by target location.
Replication completed by location	Number of objects which were successfully replicated per second, for each location.
Distribution by location	Repartition of replication between the locations. Helps to identify which location contributes the most activity.
Replication failed by location	Number of objects which failed replication per second, for each location. Note that this is the raw rate of errors, in each data-processor: before objects get through further replication replay. The overall rate of errors should be much lower.
Distribution of errors	Repartition of replication errors between the locations. Helps to identify is specific locations are not working properly.
Time in read stage	Time spent in read stage, for each operation.
Time in data write stage	Time spent in data write stage, for each operation.
Time in metadata write stage	Time spent in metadata write stage, for each operation.
Avg read stage duration	Average time spent in read stage, for each operation.
Avg data write stage duration	Average time spent in data write stage, for each operation.
Avg metadata write stage duration	Average time spent in metadata write stage, for each operation.

Replay Section

Metric	Description
Replay rate by location	Number of objects being replayed (i.e. not replicated on initial attempt) by second, for each location.
Replay attempts by location	Number of replay attempts per second, for each location.
Replay success by location	Number of objects for which replication was successful after replay, for each location.
Replay failed by location	Number of objects for which replication failed after replay, for each location.
Replay count distribution	Number of attempts, for each object.
Replay distribution by size	Size of objects being replayed.

Status Processor Section

Metric	Description
Status processor lag	Number of messages from Kafka replication topic which have not yet been processed.
Status processor ops rate	Number of status update operations per second.
Average latency	Average duration of status update operation.