Configuring Elasticsearch¶

Elasticsearch with HTTP auth

Collect the following information from your Elasticsearch administrator:
- Host: The hostname or IP address of your Elasticsearch instance (e.g. elastic.example.com).
- Port: The Elasticsearch port, typically 9200.
- Index: The index name where audit logs should be stored (e.g. linux-audit).
- TLS certificates: Download the following files from your Elasticsearch administrator:
  - elastic_ca.crt — the CA certificate that signed the Elasticsearch server certificate.
- HTTP credentials: A username and password with write access to the target index.

Edit the fluent-bit configuration

[artesca-os@bootstrap ~]$

export ELASTIC_HOST="<elastic-host>"
export ELASTIC_PORT="9200"
export ELASTIC_INDEX="<elastic-index>"
export ELASTIC_USER="<elastic-user>"
export ELASTIC_PASSWORD="<elastic-password>"

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f - << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: metalk8s-fluent-bit-config
  namespace: metalk8s-logging
data:
  config.yaml: |-
    apiVersion: addons.metalk8s.scality.com
    kind: FluentBitConfig
    spec:
      config:
        output:
          - Name: es
            Match: audit.*
            Host: "$ELASTIC_HOST"
            Port: "$ELASTIC_PORT"
            Index: "$ELASTIC_INDEX"
            Type: _doc
            Suppress_Type_Name: On
            TLS: On
            TLS.Verify: On
            TLS.ca_file: /fluent-bit/etc/tls/elastic_ca.crt
            HTTP_User: "$ELASTIC_USER"
            HTTP_Passwd: "$ELASTIC_PASSWORD"
EOF

Add the certificates to Fluent Bit
Note

Before running the command below, copy the certificate files to the home directory of the bootstrap node (/home/artesca-os/) with the following exact names:
- elastic_ca.crt
Once the command has completed, delete them from the node.
[artesca-os@bootstrap ~]$
sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ patch secret -n metalk8s-logging fluent-bit-certs -p \ '{"data":{ "elastic_ca.crt":"'"$(base64 -w 0 elastic_ca.crt)"'" }}'

Update bootstrap configuration

[artesca-os@bootstrap ~]$

sudo salt-call --local state.single file.serialize \
  /etc/metalk8s/bootstrap.yaml \
  dataset="{ \
    'addons': { \
      'fluent-bit': { \
        'enabled': True \
      }, \
    } \
  }" \
  serializer=yaml merge_if_exists=True show_changes=True

Export the MetalK8s version you are using as an environment variable:

[artesca-os@bootstrap ~]$

export MK8SVERSION=`sudo kubectl \
    --kubeconfig=/etc/kubernetes/admin.conf \
    get cm -n kube-system artesca-version \
    -o jsonpath="{.data.metalk8s}"`

Apply the changes

[artesca-os@bootstrap ~]$

sudo kubectl exec --kubeconfig /etc/kubernetes/admin.conf \
    $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -l app=salt-master -o name) \
    -n kube-system -c salt-master -- \
    salt-run state.sls metalk8s.addons.logging.deployed \
    saltenv=metalk8s-$MK8SVERSION

Elasticsearch with mTLS

Collect the following information from your Elasticsearch administrator:
- Host: The hostname or IP address of your Elasticsearch instance (e.g. elastic.example.com).
- Port: The Elasticsearch port, typically 9200.
- Index: The index name where audit logs should be stored (e.g. linux-audit).
- TLS certificates: Download the following files from your Elasticsearch administrator:
  - elastic_ca.crt — the CA certificate that signed the Elasticsearch server certificate.
  - elastic_client.crt and elastic_client.key — a client certificate/key pair for mutual TLS authentication.
Note

Elasticsearch must be configured to use the PKI realm so it can authenticate clients via their certificate without requiring a username and password. In your Elasticsearch configuration, enable the PKI realm and create a role mapping that grants write access to your audit index for the distinguished name of your client certificate.

Edit the fluent-bit configuration

[artesca-os@bootstrap ~]$

export ELASTIC_HOST="<elastic-host>"
export ELASTIC_PORT="9200"
export ELASTIC_INDEX="<elastic-index>"

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f - << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: metalk8s-fluent-bit-config
  namespace: metalk8s-logging
data:
  config.yaml: |-
    apiVersion: addons.metalk8s.scality.com
    kind: FluentBitConfig
    spec:
      config:
        output:
          - Name: es
            Match: audit.*
            Host: "$ELASTIC_HOST"
            Port: "$ELASTIC_PORT"
            Index: "$ELASTIC_INDEX"
            Type: _doc
            Suppress_Type_Name: On
            TLS: On
            TLS.Verify: On
            TLS.ca_file: /fluent-bit/etc/tls/elastic_ca.crt
            TLS.crt_file: /fluent-bit/etc/tls/elastic_client.crt
            TLS.key_file: /fluent-bit/etc/tls/elastic_client.key
EOF

Add the certificates to Fluent Bit

Note

Before running the command below, copy the certificate files to the home directory of the bootstrap node (/home/artesca-os/) with the following exact names:

elastic_ca.crt
elastic_client.crt
elastic_client.key

Once the command has completed, delete them from the node.

[artesca-os@bootstrap ~]$

sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \
   patch secret -n metalk8s-logging fluent-bit-certs -p \
   '{"data":{
     "elastic_ca.crt":"'"$(base64 -w 0 elastic_ca.crt)"'",
     "elastic_client.crt":"'"$(base64 -w 0 elastic_client.crt)"'",
     "elastic_client.key":"'"$(base64 -w 0 elastic_client.key)"'"
   }}'

Update bootstrap configuration

[artesca-os@bootstrap ~]$

sudo salt-call --local state.single file.serialize \
  /etc/metalk8s/bootstrap.yaml \
  dataset="{ \
    'addons': { \
      'fluent-bit': { \
        'enabled': True \
      }, \
    } \
  }" \
  serializer=yaml merge_if_exists=True show_changes=True

Export the MetalK8s version you are using as an environment variable:

[artesca-os@bootstrap ~]$

export MK8SVERSION=`sudo kubectl \
    --kubeconfig=/etc/kubernetes/admin.conf \
    get cm -n kube-system artesca-version \
    -o jsonpath="{.data.metalk8s}"`

Apply the changes

[artesca-os@bootstrap ~]$

sudo kubectl exec --kubeconfig /etc/kubernetes/admin.conf \
    $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -l app=salt-master -o name) \
    -n kube-system -c salt-master -- \
    salt-run state.sls metalk8s.addons.logging.deployed \
    saltenv=metalk8s-$MK8SVERSION