Centralized Monitoring
Monitor clusters, created with Kommander, on any attached cluster
Kommander provides centralized monitoring, in a multi-cluster environment, using the monitoring stack running on any attached clusters. Centralized monitoring is provided by default in every managed or attached cluster.
Managed or attached clusters are distinguished by a monitoring ID. The monitoring ID corresponds to the kube-system namespace UID of the cluster. To find a cluster’s monitoring ID, you can go to the Clusters tab on the DKP UI (in the relevant workspace), or go to the Clusters page in the Global workspace:
https://<CLUSTER_URL>/dkp/kommander/dashboard/clusters
Select the View Details
link on the attached cluster card, and then select the Configuration tab, and find the monitoring ID under Monitoring ID (clusterId).
You may also search or filter by monitoring IDs on the Clusters page, linked above.
You can also run this kubectl command, using the correct cluster’s context or kubeconfig, to look up the cluster’s kube-system namespace UID to determine which cluster the metrics and alerts correspond to:
kubectl get namespace kube-system -o jsonpath='{.metadata.uid}'
Centralized Metrics
Managed and attached clusters collects and presents metrics from all attached clusters remotely using Thanos. You can visualize these metrics in Grafana using a set of provided dashboards.
The Thanos Query component is installed on attached and managed clusters. Thanos Query queries the Prometheus instances on the attached clusters, using a Thanos sidecar running alongside each Prometheus container. Grafana is configured with Thanos Query as its datasource, and comes with pre-installed dashboards for a global view of all attached clusters. The Thanos Query
dashboard is also installed, by default, to monitor the Thanos Query component.
NOTE: Metrics from clusters are read remotely from Kommander; they are not backed up. If a attached cluster goes down, Kommander no longer collects or presents its metrics, including past data.
You can access the centralized Grafana UI at:
https://<CLUSTER_URL>/dkp/kommander/monitoring/grafana
NOTE: This is a separate Grafana instance than the one installed on all attached clusters. It is dedicated specifically to components related to centralized monitoring.
Optionally, if you want to access the Thanos Query UI (essentially the Prometheus UI), the UI is accessible at:
https://<CLUSTER_URL>/dkp/kommander/monitoring/query/
You can also check that the attached cluster’s Thanos sidecars are successfully added to Thanos Query by going to:
https://<CLUSTER_URL>/dkp/kommander/monitoring/query/stores
The preferred method to view the metrics for a specific cluster is to go directly to that cluster’s Grafana UI.
Adding Custom Dashboards
You can also define custom dashboards for centralized monitoring on Kommander. There are a few methods to import dashboards to Grafana. For simplicity, assume the desired dashboard definition is in json
format:
{
"annotations":
...
# Complete json file here
...
"title": "Some Dashboard",
"uid": "abcd1234",
"version": 1
}
After creating your custom dashboard json, insert it into a ConfigMap and save it as some-dashboard.yaml
:
apiVersion: v1
kind: ConfigMap
metadata:
name: some-dashboard
labels:
grafana_dashboard_kommander: "1"
data:
some_dashboard.json: |
{
"annotations":
...
# Complete json file here
...
"title": "Some Dashboard",
"uid": "abcd1234",
"version": 1
}
Apply the ConfigMap, which will automatically get imported to Grafana via the Grafana dashboard sidecar:
kubectl apply -f some-dashboard.yaml
Centralized Alerts
A centralized view of alerts, from attached clusters, is provided using an alert dashboard called Karma. Karma aggregates all alerts from the Alertmanagers running in the attached clusters, allowing you to visualize these alerts on one page. Using the Karma dashboard, you can get an overview of each alert and filter by alert type, cluster, and more.
Silencing alerts using the Karma UI is currently not supported.
You can access the Karma dashboard UI at:
https://<CLUSTER_URL>/dkp/kommander/monitoring/karma
When there are no attached clusters, the Karma UI displays an error message Get https://placeholder.invalid/api/v2/status: dial tcp: lookup placeholder.invalid on 10.0.0.10:53: no such host
. This is expected, and the error disappears when clusters are connected.
Federating Prometheus Alerting Rules
You can define additional Prometheus alerting rules on attached and managed clusters and federate them to all of the attached clusters by following these instructions. To use these instructions you must install the kubefedctl CLI.
Enable the PrometheusRule type for federation.
CODEkubefedctl enable PrometheusRules --kubefed-namespace kommander
Modify the existing alertmanager configuration.
CODEkubectl edit PrometheusRules/kube-prometheus-stack-alertmanager.rules -n kommander
Append a sample rule.
CODE- alert: MyFederatedAlert annotations: message: A custom alert that will always fire. expr: vector(1) labels: severity: warning
Federate the rules you just modified.
CODEkubefedctl federate PrometheusRules kube-prometheus-stack-alertmanager.rules --kubefed-namespace kommander -n kommander
Ensure that the clusters selection (
status.clusters
) is appropriately set for your desired federation strategy and check the propagation status.CODEkubectl get federatedprometheusrules kube-prometheus-stack-alertmanager.rules -n kommander -oyaml