Centralized Monitoring

Monitor clusters, created with Kommander, on any attached cluster

Kommander provides centralized monitoring, in a multi-cluster environment, using the monitoring stack running on any attached clusters. Centralized monitoring is provided by default in every managed or attached cluster.

Managed or attached clusters are distinguished by a monitoring ID. The monitoring ID corresponds to the kube-system namespace UID of the cluster. To find a cluster’s monitoring ID, you can go to the Clusters tab on the DKP UI (in the relevant workspace), or go to the Clusters page in the Global workspace:

https://<CLUSTER_URL>/dkp/kommander/dashboard/clusters

Select the View Details link on the attached cluster card, and then select the Configuration tab, and find the monitoring ID under Monitoring ID (clusterId).

You may also search or filter by monitoring IDs on the Clusters page, linked above.

You can also run this kubectl command, using the correct cluster’s context or kubeconfig, to look up the cluster’s kube-system namespace UID to determine which cluster the metrics and alerts correspond to:

kubectl get namespace kube-system -o jsonpath='{.metadata.uid}'

Centralized Metrics

Managed and attached clusters collects and presents metrics from all attached clusters remotely using Thanos. You can visualize these metrics in Grafana using a set of provided dashboards.

The Thanos Query component is installed on attached and managed clusters. Thanos Query queries the Prometheus instances on the attached clusters, using a Thanos sidecar running alongside each Prometheus container. Grafana is configured with Thanos Query as its datasource, and comes with pre-installed dashboards for a global view of all attached clusters. The Thanos Query dashboard is also installed, by default, to monitor the Thanos Query component.

NOTE: Metrics from clusters are read remotely from Kommander; they are not backed up. If a attached cluster goes down, Kommander no longer collects or presents its metrics, including past data.

You can access the centralized Grafana UI at:

https://<CLUSTER_URL>/dkp/kommander/monitoring/grafana

NOTE: This is a separate Grafana instance than the one installed on all attached clusters. It is dedicated specifically to components related to centralized monitoring.

Optionally, if you want to access the Thanos Query UI (essentially the Prometheus UI), the UI is accessible at:

https://<CLUSTER_URL>/dkp/kommander/monitoring/query

You can also check that the attached cluster’s Thanos sidecars are successfully added to Thanos Query by going to:

https://<CLUSTER_URL>/dkp/kommander/monitoring/query/stores

The preferred method to view the metrics for a specific cluster is to go directly to that cluster’s Grafana UI.

Adding Custom Dashboards

You can also define custom dashboards for centralized monitoring on Kommander. There are a few methods to import dashboards to Grafana. For simplicity, assume the desired dashboard definition is in json format:

{
    "annotations":
    ...
    # Complete json file here
    ...
    "title": "Some Dashboard",
    "uid": "abcd1234",
    "version": 1
}

After creating your custom dashboard json, insert it into a ConfigMap and save it as some-dashboard.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: some-dashboard
  labels:
    grafana_dashboard_kommander: "1"
data:
  some_dashboard.json: |
    {
      "annotations":
      ...
      # Complete json file here
      ...
      "title": "Some Dashboard",
      "uid": "abcd1234",
      "version": 1
    }

Apply the ConfigMap, which will automatically get imported to Grafana via the Grafana dashboard sidecar:

kubectl apply -f some-dashboard.yaml

Centralized Alerts

A centralized view of alerts, from attached clusters, is provided using an alert dashboard called Karma. Karma aggregates all alerts from the Alertmanagers running in the attached clusters, allowing you to visualize these alerts on one page. Using the Karma dashboard, you can get an overview of each alert and filter by alert type, cluster, and more.

Silencing alerts using the Karma UI is currently not supported.

You can access the Karma dashboard UI at:

https://<CLUSTER_URL>/dkp/kommander/monitoring/karma

When there are no attached clusters, the Karma UI displays an error message Get https://placeholder.invalid/api/v2/status: dial tcp: lookup placeholder.invalid on 10.0.0.10:53: no such host. This is expected, and the error disappears when clusters are connected.

Federating Prometheus Alerting Rules

You can define additional Prometheus alerting rules on attached and managed clusters and federate them to all of the attached clusters by following these instructions. To use these instructions you must install the kubefedctl CLI.

1. Enable the PrometheusRule type for federation.

   CODE

   kubefedctl enable PrometheusRules --kubefed-namespace kommander

2. Modify the existing alertmanager configuration.

   CODE

   kubectl edit PrometheusRules/kube-prometheus-stack-alertmanager.rules -n kommander

3. Append a sample rule.

   CODE

   - alert: MyFederatedAlert
     annotations:
       message: A custom alert that will always fire.
     expr: vector(1)
     labels:
       severity: warning

4. Federate the rules you just modified.

   CODE

   kubefedctl federate PrometheusRules kube-prometheus-stack-alertmanager.rules --kubefed-namespace kommander -n kommander

5. Ensure that the clusters selection (status.clusters) is appropriately set for your desired federation strategy and check the propagation status.

   CODE

   kubectl get federatedprometheusrules kube-prometheus-stack-alertmanager.rules -n kommander -oyaml