Configuring Splunk¶

Splunk with HEC Token

Collect the following information from your Splunk administrator:
- Host: The hostname or IP address of your Splunk HEC (HTTP Event Collector) endpoint (e.g. splunk.example.com).
- Port: The HEC port, typically 8088.
- Index: The Splunk index where audit logs should be stored (e.g. artesca_audit).
- HEC Token: A valid HEC token with permission to send events to the target index. In Splunk Web, go to Settings → Data Inputs → HTTP Event Collector to create or retrieve a token. A HEC token is required even when using mTLS — the token routes events to the correct index/sourcetype while mTLS provides an additional layer of client certificate verification.
- TLS certificates: Download the following files from your Splunk administrator:
  - splunk_ca.crt — the CA certificate that signed the Splunk HEC server certificate.

Edit the fluent-bit configuration

[artesca-os@bootstrap ~]$

export SPLUNK_HOST="<splunk-host>"
export SPLUNK_PORT="8088"
export SPLUNK_INDEX="<splunk-index>"
export SPLUNK_TOKEN="<splunk-token>"

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f - << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: metalk8s-fluent-bit-config
  namespace: metalk8s-logging
data:
  config.yaml: |-
    apiVersion: addons.metalk8s.scality.com
    kind: FluentBitConfig
    spec:
      config:
        output:
          - Name: splunk
            Match: audit.*
            Host: "$SPLUNK_HOST"
            Port: "$SPLUNK_PORT"
            Splunk_Token: "$SPLUNK_TOKEN"
            TLS: On
            TLS.Verify: On
            TLS.ca_file: /fluent_bit/etc/tls/splunk_ca.crt
            event_key: \$log
            event_index: "$SPLUNK_INDEX"
            event_sourcetype: linux:audit
EOF

Add the certificates to Fluent Bit
Note

Before running the command below, copy the certificate files to the home directory of the bootstrap node (/home/artesca-os/) with the following exact names:
- splunk_ca.crt
Once the command has completed, delete them from the node.
[artesca-os@bootstrap ~]$
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ patch secret -n metalk8s-logging fluent-bit-certs -p \ '{"data":{ "splunk_ca.crt":"'"$(base64 -w 0 splunk_ca.crt)"'" }}'

Update bootstrap configuration

[artesca-os@bootstrap ~]$

sudo salt-call --local state.single file.serialize \
  /etc/metalk8s/bootstrap.yaml \
  dataset="{ \
    'addons': { \
      'fluent-bit': { \
        'enabled': True \
      }, \
    } \
  }" \
  serializer=yaml merge_if_exists=True show_changes=True

Export the MetalK8s version you are using as an environment variable:

[artesca-os@bootstrap ~]$

export MK8SVERSION=`sudo kubectl \
    --kubeconfig=/etc/kubernetes/admin.conf \
    get cm -n kube-system artesca-version \
    -o jsonpath="{.data.metalk8s}"`

Apply the changes

[artesca-os@bootstrap ~]$

sudo kubectl exec --kubeconfig /etc/kubernetes/admin.conf \
    $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -l app=salt-master -o name) \
    -n kube-system -c salt-master -- \
    salt-run state.sls metalk8s.addons.logging.deployed \
    saltenv=metalk8s-$MK8SVERSION

Splunk with mTLS

Collect the following information from your Splunk administrator:
- Host: The hostname or IP address of your Splunk HEC (HTTP Event Collector) endpoint (e.g. splunk.example.com).
- Port: The HEC port, typically 8088.
- Index: The Splunk index where audit logs should be stored (e.g. artesca_audit).
- HEC Token: A valid HEC token with permission to send events to the target index. In Splunk Web, go to Settings → Data Inputs → HTTP Event Collector to create or retrieve a token. A HEC token is required even when using mTLS — the token routes events to the correct index/sourcetype while mTLS provides an additional layer of client certificate verification.
- TLS certificates: Download the following files from your Splunk administrator:
  - splunk_ca.crt — the CA certificate that signed the Splunk HEC server certificate.
  - splunk_client.crt and splunk_client.key — a client certificate/key pair for mutual TLS authentication.

Edit the fluent-bit configuration

[artesca-os@bootstrap ~]$

export SPLUNK_HOST="<splunk-host>"
export SPLUNK_PORT="8088"
export SPLUNK_INDEX="<splunk-index>"
export SPLUNK_TOKEN="<splunk-token>"

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f - << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: metalk8s-fluent-bit-config
  namespace: metalk8s-logging
data:
  config.yaml: |-
    apiVersion: addons.metalk8s.scality.com
    kind: FluentBitConfig
    spec:
      config:
        output:
          - Name: splunk
            Match: audit.*
            Host: "$SPLUNK_HOST"
            Port: "$SPLUNK_PORT"
            Splunk_Token: "$SPLUNK_TOKEN"
            TLS: On
            TLS.Verify: On
            TLS.ca_file: /fluent_bit/etc/tls/splunk_ca.crt
            TLS.crt_file: /fluent_bit/etc/tls/splunk_client.crt
            TLS.key_file: /fluent_bit/etc/tls/splunk_client.key
            event_key: \$log
            event_index: "$SPLUNK_INDEX"
            event_sourcetype: linux:audit
EOF

Add the certificates to Fluent Bit

Note

Before running the command below, copy the certificate files to the home directory of the bootstrap node (/home/artesca-os/) with the following exact names:

splunk_ca.crt
splunk_client.crt
splunk_client.key

Once the command has completed, delete them from the node.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
   patch secret -n metalk8s-logging fluent-bit-certs -p \
   '{"data":{
     "splunk_ca.crt":"'"$(base64 -w 0 splunk_ca.crt)"'",
     "splunk_client.crt":"'"$(base64 -w 0 splunk_client.crt)"'",
     "splunk_client.key":"'"$(base64 -w 0 splunk_client.key)"'"
   }}'

Update bootstrap configuration

[artesca-os@bootstrap ~]$

sudo salt-call --local state.single file.serialize \
  /etc/metalk8s/bootstrap.yaml \
  dataset="{ \
    'addons': { \
      'fluent-bit': { \
        'enabled': True \
      }, \
    } \
  }" \
  serializer=yaml merge_if_exists=True show_changes=True

Export the MetalK8s version you are using as an environment variable:

[artesca-os@bootstrap ~]$

export MK8SVERSION=`sudo kubectl \
    --kubeconfig=/etc/kubernetes/admin.conf \
    get cm -n kube-system artesca-version \
    -o jsonpath="{.data.metalk8s}"`

Apply the changes

[artesca-os@bootstrap ~]$

sudo kubectl exec --kubeconfig /etc/kubernetes/admin.conf \
    $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -l app=salt-master -o name) \
    -n kube-system -c salt-master -- \
    salt-run state.sls metalk8s.addons.logging.deployed \
    saltenv=metalk8s-$MK8SVERSION