Managing Loki (Log Aggregation Tool)

Loki is a log aggregation tool integrated with Grafana to visualize cluster logs. It eases the debugging process.

Loki depends on Fluent Bit to collect logs. Enabling Loki automatically enables Fluent Bit.

Warning

By default, Loki is disabled after installation or upgrade due to the CPU/LOAD high resource usage that might cause performance or stability issues. If you activate Loki, make sure it is for a limited amount of time. Closely monitor the Loki partition and when the partition is close to full, disable Loki.

Prerequisites

These elements must be strictly under the following thresholds:

• Memory (%): < 90%

• System Load: < 100

Note

These values can be monitored in the Dashboard Monitoring.

Enable Loki

1. Update bootstrap configuration

   [artesca-os@bootstrap ~]$
   

   sudo salt-call --local state.single file.serialize \
     /etc/metalk8s/bootstrap.yaml \
     dataset="{ \
       'addons': { \
         'fluent-bit': { \
           'enabled': True \
         }, \
         'loki': { \
           'enabled': True \
         } \
       } \
     }" \
     serializer=yaml merge_if_exists=True show_changes=True
   

2. Export the MetalK8s version you are using as an environment variable:

   [artesca-os@bootstrap ~]$
   

   export MK8SVERSION=`sudo kubectl \
       --kubeconfig=/etc/kubernetes/admin.conf \
       get cm -n kube-system artesca-version \
       -o jsonpath="{.data.metalk8s}"`
   

3. Apply the changes

   [artesca-os@bootstrap ~]$
   

   sudo kubectl exec --kubeconfig /etc/kubernetes/admin.conf \
       $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -l app=salt-master -o name) \
       -n kube-system -c salt-master -- \
       salt-run state.sls metalk8s.addons.logging.deployed,metalk8s.addons.ui.deployed.ingress \
       saltenv=metalk8s-$MK8SVERSION
   

4. Ensure that the pods are up and running.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl  --kubeconfig /etc/kubernetes/admin.conf get pods -n metalk8s-logging
   

5. Closely monitor the Loki partition in the ARTESCA UI.

Disable Loki

1. Update bootstrap configuration

   [artesca-os@bootstrap ~]$
   

   sudo salt-call --local state.single file.serialize \
     /etc/metalk8s/bootstrap.yaml \
     dataset="{ \
       'addons': { \
         'loki': { \
           'enabled': False \
         } \
       } \
     }" \
     serializer=yaml merge_if_exists=True show_changes=True
   

2. Export the MetalK8s version you are using as an environment variable:

   [artesca-os@bootstrap ~]$
   

   export MK8SVERSION=`sudo kubectl \
       --kubeconfig=/etc/kubernetes/admin.conf \
       get cm -n kube-system artesca-version \
       -o jsonpath="{.data.metalk8s}"`
   

3. Apply the changes

   [artesca-os@bootstrap ~]$
   

   sudo kubectl exec --kubeconfig /etc/kubernetes/admin.conf \
       $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -l app=salt-master -o name) \
       -n kube-system -c salt-master -- \
       salt-run state.sls metalk8s.addons.logging.deployed,metalk8s.addons.ui.deployed.ingress \
       saltenv=metalk8s-$MK8SVERSION
   

Note

After disabling Loki, Fluent Bit remains running. If you are also not using SIEM log forwarding, you can save resources by Disabling Fluent Bit.