Fluent Bit is the DKP choice of open-source log collection and forwarding tool.

On the Management cluster, Fluentbit is disabled by default. The amount of admin logs ingested to Loki requires additional disk space to be configured on the grafana-loki-minio Minio Tenant. Enabling admin logs may use around 2GB/day per node. See Release Notes 2.3.0 | Configuring-the-Grafana-Loki-Minio-Tenant for more details on how to configure the Minio Tenant.

Audit Log Collection

Auditing in Kubernetes provides a way to chronologically document the actions taken on a cluster. On Kommander, by default, audit logs are collected and stored for quick indexing. Viewing and accessing can be done via the Grafana logging UI.

To adjust the default Audit Policy log backend configuration, you must modify the log retention settings by Configuring the Control Plane before creating the cluster. This needs to be done prior to creating the cluster since it cannot be edited after creation.

Collecting systemd logs from a non-default path

By default, Fluent Bit pods are configured to collect systemd logs from the /var/log/journal/ path on cluster nodes.

If systemd-journald running as a part of the OS on the nodes uses a different path for writing logs, you will need to override configuration of the fluent-bit AppDeployment to make Fluent Bit collect systemd logs.

To configure the Fluent Bit AppDeployment to collect systemd logs from a non-default path, follow these steps (all kubectl and dkp invocations refer to the management cluster):

  1. Execute the following command to get the namespace of the workspace in which you would like to configure Fluent Bit:

    dkp get workspaces
    CODE

    And copy the value under the NAMESPACE column for your workspace.

  2. Set the WORKSPACE_NAMESPACE variable to the namespace copied in the previous step:

    export WORKSPACE_NAMESPACE=<WORKSPACE_NAMESPACE>
    CODE
  3. Identify the systemd-journald log data storage path on the nodes of the clusters in the workspace by using the OS documentation and examining the systemd configuration.

    Usually it will be either /var/log/journal (typically used when systemd-journald is configured to store logs permanently; in this case the default Fluent Bit configuration should work) or /run/log/journal (typically used when systemd-journald is configured to use a volatile storage).

  4. Extract the default Helm values used by the Fluent Bit App:

    kubectl get -n ${WORKSPACE_NAMESPACE} configmaps fluent-bit-0.19.21-d2iq-defaults -o=jsonpath='{.data.values\.yaml}' > fluent-bit-values.yaml
    CODE
  5. Edit the resulting file fluent-bit-values.yaml by removing all sections except for extraVolumes, extraVolumeMounts and config.inputs. The result should look similarly to this:

    extraVolumes:
    # we create this to have a persistent tail-db directory an all nodes
    # otherwise a restarted fluent-bit would rescrape all tails
    - name: tail-db
      hostPath:
        path: /var/log/tail-db
        type: DirectoryOrCreate
    # we create this to get rid of error messages that would appear on non control-plane nodes
    - name: kubernetes-audit
      hostPath:
        path: /var/log/kubernetes/audit
        type: DirectoryOrCreate
    # needed for kmsg input plugin
    - name: uptime
      hostPath:
        path: /proc/uptime
        type: File
    - name: kmsg
      hostPath:
        path: /dev/kmsg
        type: CharDevice
    
    extraVolumeMounts:
    - name: tail-db
      mountPath: /tail-db
    - name: kubernetes-audit
      mountPath: /var/log/kubernetes/audit
    - name: uptime
      mountPath: /proc/uptime
    - name: kmsg
      mountPath: /dev/kmsg
    
    config:
      inputs: |
        # Collect audit logs, systemd logs, and kernel logs.
        # Pod logs are collected by the fluent-bit deployment managed by logging-operator.
        [INPUT]
            Name tail
            Alias kubernetes_audit
            Path /var/log/kubernetes/audit/*.log
            Parser kubernetes-audit
            DB /tail-db/audit.db
            Tag audit.*
            Refresh_Interval 10
            Rotate_Wait 5
            Mem_Buf_Limit 135MB
            Buffer_Chunk_Size 5MB
            Buffer_Max_Size 20MB
            Skip_Long_Lines Off
        [INPUT]
            Name systemd
            Alias kubernetes_host
            DB /tail-db/journal.db
            Tag host.*
            Max_Entries 1000
            Read_From_Tail On
            Strip_Underscores On
        [INPUT]
            Name kmsg
            Alias kubernetes_host_kernel
            Tag kernel
    CODE
  6. Add the following item to the list under the extraVolumes key:

    - name: kubernetes-host
      hostPath:
        path: <path to systemd logs on the node>
        type: Directory
    CODE
  7. Add the following item to the list under the extraVolumeMounts key:

    - name: kubernetes-host
      mountPath: <path to systemd logs on the node>
    CODE

    These items will make Kubernetes mount systemd logs into Fluent Bit pods.

  8. Add the following line into the [INPUT] entry identified by Name systemd and Alias kubernetes_host.

    Path <path to systemd logs on the node>
    CODE

    This is needed to make Fluent Bit actually collect the mounted logs

  9. Assuming that the path to systemd logs on the node is /run/log/journal, the result will look similarly to this:

    extraVolumes:
    # we create this to have a persistent tail-db directory an all nodes
    # otherwise a restarted fluent-bit would rescrape all tails
    - name: tail-db
      hostPath:
        path: /var/log/tail-db
        type: DirectoryOrCreate
    # we create this to get rid of error messages that would appear on non control-plane nodes
    - name: kubernetes-audit
      hostPath:
        path: /var/log/kubernetes/audit
        type: DirectoryOrCreate
    # needed for kmsg input plugin
    - name: uptime
      hostPath:
        path: /proc/uptime
        type: File
    - name: kmsg
      hostPath:
        path: /dev/kmsg
        type: CharDevice
    - name: kubernetes-host
      hostPath:
        path: /run/log/journal
        type: Directory
    
    extraVolumeMounts:
    - name: tail-db
      mountPath: /tail-db
    - name: kubernetes-audit
      mountPath: /var/log/kubernetes/audit
    - name: uptime
      mountPath: /proc/uptime
    - name: kmsg
      mountPath: /dev/kmsg
    - name: kubernetes-host
      mountPath: /run/log/journal
    
    config:
      inputs: |
        # Collect audit logs, systemd logs, and kernel logs.
        # Pod logs are collected by the fluent-bit deployment managed by logging-operator.
        [INPUT]
            Name tail
            Alias kubernetes_audit
            Path /var/log/kubernetes/audit/*.log
            Parser kubernetes-audit
            DB /tail-db/audit.db
            Tag audit.*
            Refresh_Interval 10
            Rotate_Wait 5
            Mem_Buf_Limit 135MB
            Buffer_Chunk_Size 5MB
            Buffer_Max_Size 20MB
            Skip_Long_Lines Off
        [INPUT]
            Name systemd
            Alias kubernetes_host
            Path /run/log/journal
            DB /tail-db/journal.db
            Tag host.*
            Max_Entries 1000
            Read_From_Tail On
            Strip_Underscores On
        [INPUT]
            Name kmsg
            Alias kubernetes_host_kernel
            Tag kernel
    CODE
  10. Create a ConfigMap manifest with override values from fluent-bit-values.yaml:

    cat <<EOF >fluent-bit-overrides.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      namespace: ${WORKSPACE_NAMESPACE}
      name: fluent-bit-overrides
    data:
      values.yaml: |
    $(cat fluent-bit-values.yaml | sed 's/^/    /g')
    EOF
    CODE
  11. Create a ConfigMap from the manifest above:

    kubectl apply -f fluent-bit-overrides.yaml
    CODE
  12. Edit the fluent-bit AppDeployment to set the value of spec.configOverrides.name to the name of the created ConfigMap. (You can use the steps in the procedure, Deploy an Application with a Custom Configuration as a guide.)

    dkp edit appdeployment -n ${WORKSPACE_NAMESPACE} fluent-bit
    CODE

    After your editing is complete, the AppDeployment resembles this example:

    apiVersion: apps.kommander.d2iq.io/v1alpha3
    kind: AppDeployment
    metadata:
      name: fluent-bit
      namespace: ${WORKSPACE_NAMESPACE}
    spec:
      appRef:
        name: fluent-bit-0.19.21
        kind: ClusterApp
      configOverrides:
        name: fluent-bit-overrides
    CODE
  13. Log in into the Grafana logging UI of your workspace and verify that logs with a label log_source=kubernetes_host are now present in Loki.

Related information

For information on related topics or procedures, refer to the following: