Monitor Applications using Prometheus
Before attempting to monitor your own applications, you should be familiar with the Prometheus conventions for exposing metrics. In general, there are two key recommendations:
You should expose metrics using an HTTP endpoint named
/metrics
.The metrics you expose must be in a format that Prometheus can consume.
By following these conventions, you ensure that your application metrics can be consumed by Prometheus itself or by any Prometheus-compatible tool that can retrieve metrics, using the Prometheus client endpoint.
The kube-prometheus-stack
for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances. It provides a Kubernetes resource called ServiceMonitor
.
By default, the kube-prometheus-stack
provides the following service monitors to collect internal Kubernetes components:
kube-apiserver
kube-scheduler
kube-controller-manager
etcd
kube-dns/coredns
kube-proxy
The operator is in charge of iterating over all of these ServiceMonitor
objects and collecting the metrics from these defined components.
The following example illustrates how to retrieve application metrics. In this example, there are:
Three instances of a simple app named
my-app
The sample app listens and exposes metrics on port 8080
The app is assumed to already be running
To prepare for monitoring of the sample app, create a service that selects the pods that have my-app
as the value defined for their app label setting.
The service object also specifies the port on which the metrics are exposed. The ServiceMonitor
has a label selector to select services and their underlying endpoint objects. For example:
kind: Service
apiVersion: v1
metadata:
name: my-app
namespace: my-namespace
labels:
app: my-app
spec:
selector:
app: my-app
ports:
- name: metrics
port: 8080
This service object is discovered by a ServiceMonitor
, which defines the selector to match the labels with those defined in the service. The app label must have the value my-app
.
In this example, in order for kube-prometheus-stack
to discover this ServiceMonitor
, add a specific label prometheus.kommander.d2iq.io/select: "true"
in the yaml
:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app-service-monitor
namespace: my-namespace
labels:
prometheus.kommander.d2iq.io/select: "true"
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
In this example, you would modify the Prometheus settings to have the operator collect metrics from the service monitor by appending the following configuration to the overrides ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-prometheus-stack-overrides
namespace: ${WORKSPACE_NAMESPACE}
data:
values.yaml: |
---
prometheus:
additionalServiceMonitors:
- name: my-app-service-monitor
selector:
matchLabels:
app: my-app
namespaceSelector:
matchNames:
- my-namespace
endpoints:
- port: metrics
interval: 30s
Official documentation about using a ServiceMonitor
to monitor an app with the Prometheus-operator on Kubernetes can be found on this GitHub repository.