Resolve Issues with Oplog Populator and Kafka Connect

Problem Description

The following issues are noticed:

  • ChangeStreamHistoryLost on the artesca-data-base-queue-connector-XXX pods.

  • /var/lib/containerd size filling up on the host.

Problem Resolution

Important

If this procedure is applied on a system with Lifecycle Transition, Lifecycle Replication, or bucket notification, some event will be skipped and the respective features can not work properly.

This procedure enables to reset the change stream tokens and to remove the log files.

  1. Execute into one of the artesca-data-base-queue-connector-XXX pods.

    [artesca-os@bootstrap ~]$

    sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec -ti -n zenko \
artesca-data-queue-connector-0 -- bash

  2. List the connectors one by one.

    (inside)# curl 0:8083/connectors
["backbeat-oplog-source-connector-XXXX", "backbeat-oplog-source-connector-YYYY", ...]

  3. Delete the connectors one by one.

    (inside)# curl -XDELETE 0:8083/connectors/backbeat-oplog-source-connector-XXXX
(inside)# curl -XDELETE 0:8083/connectors/backbeat-oplog-source-connector-YYYY

  4. Restart all Kafka connect pods.

    [artesca-os@bootstrap ~]$

    sudo kubectl rollout restart -n zenko deploy/artesca-data-base-queue-connector

  5. Restart Oplog Populator.

    [artesca-os@bootstrap ~]$

    sudo kubectl rollout restart -n zenko deploy/artesca-data-backbeat-oplog-populator