Verify That Alertmanager Sends Notifications

Verify the configuration to ensure that alerts are properly routed to your communication platform, and the notifications are reaching the intended recipients.

Prerequisite

You have enabled an Alertmanager configuration for Slack or Microsoft Teams:

Send a Test Alert

Trigger a mock DKP Insights alert to confirm the configuration is successful.

Open a local port for the Alertmanager mock alert:

CODE

kubectl -n kommander port-forward svc/kube-prometheus-stack-alertmanager 8083:9093

In another terminal session, send a mock alert to the open port:

CODE

curl -L 'http://localhost:8083/api/v2/alerts' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-d \
'[{
    "labels": 
    {
        "alertname": "Test Insight Alert",
        "namespace": "kommander",
        "status": "Open",
        "source": "Insights",
        "severity": "Critical",
        "cluster": "Kommander Host (Test)"
    },
    "annotations":
    {
        "description": "This is a mock Insight for testing",
        "generatorURL": "https://test-endpoint.com",
        "categories": "Best-Practices, Configuration"
    }
}]'

This sends a Critical mock Insights alert to Alertmanager, which triggers sending a notification to the configured communication platform.

Troubleshooting

Verify the Alertmanager Dashboard

Verify if DKP Insights alerts are displayed in the Alertmanager Dashboard.

Access the DKP UI.
For Enterprise only: Select Management Cluster Workspace.
Select Application Dashboards, and look for the Prometheus Alert Manager application card.
Select Dashboard to open the Alertmanager console.

If you can see DKP Insights alerts present, the DKP Insights-Alertmanager route is configured successfully.

You can recognize DKP Insights alerts from other default DKP alerts, because the alert severity tags are capitalized. For example, a DKP Insight alert is Critical, whereas other non-Insights alerts are critical.

Verify the Alertmanager Log Files

If Slack does not display any Alert messages, but you can see DKP Insights alerts in the Alertmanager console, verify the deployment logs:

CODE

kubectl -n kommander logs alertmanager-kube-prometheus-stack-alertmanager-0

If the output is blank, the configuration has been successful. The output displays errors if the deployment has failed.

Next Topic:

Configuration Anomalies