Add Bucket Notification Destination¶

Bucket notification destinations can be added by creating a custom resource in the ARTESCA Kubernetes cluster. The custom resource will contain the target Kafka’s configuration.

Note

You must be connected to the ARTESCA cluster to perform the following procedures.

Retrieve the zenko instance name.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get zenko -n zenko --no-headers -o custom-columns=":metadata.name"

Save the destination configuration into a YAML file. Refer to the Configuration Examples for more information.

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  # unique destination name
  name: <DESTINATION_NAME>
  labels:
    # active zenko instance name
    app.kubernetes.io/instance: <ZENKO_INSTANCE_NAME>
  namespace: zenko
spec:
  # only 'kafka' type is supported for now
  type: kafka
  # kafka host
  host: <DESTINATION_KAFKA_HOST>
  # kafka port
  port: <DESTINATION_KAFKA_PORT>
  # kafka topic
  destinationTopic: <DESTINATION_KAFKA_TOPIC>
  # optional parameter if no auth required
  # 'kerberos', 'plain', and 'scram' are supported
  auth: <AUTH_TYPE>
  # This section is required if auth is set to 'kerberos'
  kerberos:
    # Name of secret containing the keytab to use
    keyTab: <KEYTAB_SECRET_NAME>
    # Name of entry to use as keytab, in the secret
    # If not specified, the entry must be named "cluster.keytab"
    keyTabEntry: <KEYTAB_SECRET_ENTRY>
    # Principal to authenticate in kerberos
    principal: <KERBEROS_PRINCIPAL>
    # use 'SASL_SSL' as protocol for SSL
    # otherwise use 'SASL_PLAINTEXT'
    protocol: <PROTOCOL_NAME>
    # Kerberos service name configured in kafka server
    serviceName: <KERBEROS_SERVICE_NAME>
  # This section is required if auth is set to 'plain'
  plain:
    # Username used for SASL/PLAIN authentication
    username: <PLAIN_USERNAME>
    # Password used for SASL/PLAIN authentication
    password: <PLAIN_PASSWORD>
  # This section is required if auth is set to 'scram'
  scram:
    # Username used for SASL/SCRAM authentication
    username: <SCRAM_USERNAME>
    # Password used for SASL/SCRAM authentication
    password: <SCRAM_PASSWORD>
    # SCRAM mechanism: 'SHA-256' or 'SHA-512'
    mechanism: <SCRAM_MECHANISM>
  # Specifying ssl will enable TLS when connecting to the destination.
  # To use TLS without a client certificate or custom CA, specify
  # an empty object: `ssl: {}`
  ssl:
    # Name of a kubernetes.io/tls secret with the TLS client certificate.
    # The secret must contain `tls.crt` and `tls.key`, and may
    # optionally include `ca.crt` for a custom CA.
    # Specifying a secret with only `ca.crt` (without `tls.crt`
    # and `tls.key`) is not supported.
    certificate: <SSL_CERTIFICATE_SECRET>
    # Password for TLS client key (optional)
    password: <TLS_KEY_PASSWORD>

Create the custom resource using the previously created YAML file.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f <FILE_NAME> -n zenko

The cluster takes a few minutes to reconcile the changes and create resources to handle the additional destinations.

To wait for cluster reconciliation:

Export the notification destination name as a variable.

[artesca-os@bootstrap ~]$

export DESTINATION_NAME=<DESTINATION_NAME>

Wait for Bucket Notification to be ready.

[artesca-os@bootstrap ~]$

timeout 5m bash -c 'until sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf -n zenko get deploy artesca-data-backbeat-notification-processor-${DESTINATION_NAME} &>/dev/null ; do sleep 1; done' \
&& sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf wait deployment -n zenko artesca-data-backbeat-notification-processor-${DESTINATION_NAME} --for condition=Available=True --timeout=5m >/dev/null \
&& echo -e "\nBucket Notification Ready for target \"${DESTINATION_NAME}\"" \
|| echo -e "\nTimed out waiting for Bucket Notification to be ready for target \"${DESTINATION_NAME}\""

On success, the Bucket Notification Ready for target "<DESTINATION_NAME>" message is displayed.

Configure Azure Event Hub as a Notification Destination¶

Azure Event Hub offers a Kafka-compatible interface, which can be used to receive bucket notifications from ARTESCA.

To configure an Azure Event Hub as a notification destination:

Create an Event Hub namespace and an Event Hub inside it, using the official Azure documentation for more information.

Tip

Use the Standard pricing tier or higher (not Basic).
In your Event Hubs namespace, go to Settings > Shared access policies, and create a new Shared access policy with Send permissions. Copy the Primary connection string value.
Use the Azure Event Hub destination example to create the ZenkoNotificationTarget resource. Replace the Event Hub namespace host name with the host value, the Event Hub name with the destinationTopic value, and the Primary connection string with the password value.

Configure AWS MSK as a Notification Destination¶

Amazon Managed Streaming for Apache Kafka (MSK) supports SASL/SCRAM authentication, which can be used to receive bucket notifications from ARTESCA.

To configure an AWS MSK cluster as a notification destination:

Create an MSK cluster using the official AWS documentation.
Under Security settings, enable SASL/SCRAM authentication.
In AWS KMS, create a custom symmetric encryption key. This key will be used to encrypt the secret (the default aws/secretsmanager key cannot be used with MSK).
In AWS Secrets Manager, create a secret of type Other type of secret, encrypted with the KMS key created in the previous step. The secret name must start with the AmazonMSK_ prefix, and its value must be in the following JSON format:
```
{"username": "msk-user", "password": "msk-secret-password"}
```
Associate the secret with the MSK cluster using the Associate secrets action in the MSK console, or the BatchAssociateScramSecret API.
Use the AWS MSK destination with SASL/SCRAM authentication example to create the ZenkoNotificationTarget resource. Replace the MSK bootstrap server with the host value, the topic name with the destinationTopic value, and the Secrets Manager credentials with the username and password values.

Configuration Examples¶

Single destination, no authentication¶

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  name: target1
  labels:
    app.kubernetes.io/instance: artesca-data
  namespace: zenko
spec:
  type: kafka
  host: 10.20.14.2
  port: 9092
  destinationTopic: notification-topic

Multiple destinations, no authentication¶

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  name: target1
  labels:
    app.kubernetes.io/instance: artesca-data
  namespace: zenko
spec:
  type: kafka
  host: 10.20.14.2
  port: 9092
  destinationTopic: notification-topic

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  name: target2
  labels:
    app.kubernetes.io/instance: artesca-data
  namespace: zenko
spec:
  type: kafka
  host: 10.64.1.22
  port: 9092
  destinationTopic: notification-topic

Single destination with Kerberos authentication and TLS with a TLS client certificate¶

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  name: target1
  labels:
    app.kubernetes.io/instance: artesca-data
  namespace: zenko
spec:
  type: kafka
  host: 10.64.50.10
  port: 9092
  destinationTopic: notification-topic
  auth: 'kerberos'
  kerberos:
    keyTab: kerberosSecret
    keyTabEntry: 'kafka.keytab'
    principal: kafka
    protocol: SASL_SSL
    serviceName: kafka/host-name@kerberos.scality.com
  ssl:
    certificate: sslCertSecret
    password: securePassword

Single destination with SASL/PLAIN authentication and TLS (without a TLS client certificate)¶

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  name: target1
  labels:
    app.kubernetes.io/instance: artesca-data
  namespace: zenko
spec:
  type: kafka
  host: 10.64.50.10
  port: 9092
  destinationTopic: notification-topic
  auth: 'plain'
  plain:
    username: notification-user
    password: notification-secret-password
  ssl: {}

Azure Event Hub destination¶

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  name: target1
  labels:
    app.kubernetes.io/instance: artesca-data
  namespace: zenko
spec:
  type: kafka
  host: myeventhubnamespace.servicebus.windows.net
  port: 9093
  destinationTopic: myeventhub
  auth: 'plain'
  plain:
    username: '$ConnectionString'
    password: 'Endpoint=sb://myeventhubnamespace.servicebus.windows.net/;SharedAccessKeyName=MySharedAccessKey;SharedAccessKey=secret+replace+me='
  ssl: {}

Single destination with SASL/SCRAM authentication and TLS (without a TLS client certificate)¶

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  name: target1
  labels:
    app.kubernetes.io/instance: artesca-data
  namespace: zenko
spec:
  type: kafka
  host: 10.64.50.10
  port: 9092
  destinationTopic: notification-topic
  auth: 'scram'
  scram:
    username: notification-user
    password: notification-secret-password
    mechanism: SHA-512
  ssl: {}

AWS MSK destination with SASL/SCRAM authentication¶

apiVersion: zenko.io/v1alpha2
kind: ZenkoNotificationTarget
metadata:
  name: target1
  labels:
    app.kubernetes.io/instance: artesca-data
  namespace: zenko
spec:
  type: kafka
  host: b-1.mymskcluster.abcdef.c1.kafka.us-east-1.amazonaws.com
  port: 9096
  destinationTopic: notification-topic
  auth: 'scram'
  scram:
    username: msk-user
    password: msk-secret-password
    mechanism: SHA-512
  ssl: {}