Logging

D2iQ Kubernetes Platform (DKP) comes with a pre-configured logging stack that allows you to collect and visualize pod and admin log data at the Workspace level. The logging stack is also multi-tenant capable. Multi-tenancy is enabled at the Project level through role-based access control (RBAC).

By default, logging is disabled on managed and attached clusters. You need to enable the logging stack applications explicitly on the workspace to make use of these capabilities.

The primary components of the logging stack include these platform services:

• BanzaiCloud Logging-operator

• Grafana and Grafana Loki

• Fluent Bit and Fluentd

In addition to these platform services, logging relies on other software and system facilities, including the container runtime, the journald facility, and systemd configuration are used to collect logs and messages from all the machines in the cluster.

The following diagram illustrates how different components of the logging stack collect log data and provide information about clusters:

The DKP logging stack aggregates logs from applications and nodes running inside your cluster.

DKP uses the BanzaiCloud Logging-operator to manage the Fluent Bit and Fluentd deployments that collect pod logs, using Kubernetes API extensions called custom resources. The custom resources allow users to declare logging configurations using kubectl commands. The Fluent Bit instance deployed by Logging-operator gathers pod logs data and sends it to Fluentd, which forwards it to the appropriate Grafana Loki servers based on the configuration defined in custom resources.

Loki then indexes the log data by label and stores it for querying. Loki maintains log order integrity but does not index the log messages themselves, which improves its efficiency and lowers its footprint.

Logging Operator

This section contains information about setting up the logging operator to manage the Fluent Bit and Fluentd resources.

Fluent Bit Buffer Information

Fluent Bit collects container logs from the host filesystem and performs the following:

• Maintains a small buffer of logs (5 MB in memory)

• Maintains a checkpoint on each host for each file it's consumed, so if it’s restarted, it can resumed where it left off.

• Flushes the buffer every one second.

• Has a five second grace period to flush the buffer on exit (along with a 30 second termination grace period on the pod).

Every pod restart should result in the buffer being flushed to fluentd(allows up to five seconds for that flush to happen).

If the buffer is not fully flushed during that five seconds, a small amount of log data may be dropped, but if fluentdis functional, it is unlikely that fluent-bit would be unable to flush its logs on pod termination.

Fluent Bit can be configured to use a hostPath volume to store the buffer information, so it can be picked up again when Fluent Bit restarts.

For more information about fluent-bit, see https://kube-logging.dev/docs/logging-infrastructure/fluentbit/#hostpath-volumes-for-buffers-and-positions.

For more information about Logging in relation to how it is used in DKP, refer to these pages in our Help Center: