Prometheus Operator

Overview

How Prometheus Operator interacts with Prometheus:

Basic Usage

1. Deploy Prometheus Operator and Prometheus

Create namespace.
```
kubectl create ns monitoring
```
Install Prometheus operator in default namespace.
```
kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
```
This command creates the following resources:
1. 8 CRDs:
  1. AlertmanagerConfig
  2. Alertmanager
  3. PodMonitor
  4. Probe
  5. Prometheus
  6. PrometheusRule
  7. ServiceMonitor
  8. ThanosRuler
2. ClusterRoleBinding & ClusterRole: prometheus-operator
3. Deployment: prometheus-operator
4. ServiceAccount: prometheus-operator
5. Service: prometheus-operator
Deploy Prometheus in monitoring namespace.

The resources to deploy: 1. Prometheus 1. rbac: ClusterRole, ClusterRoleBinding, and ServiceAccount 1. Service for Prometheus Pods. 1. ServiceMonitor for Prometheus itself.
```
kubectl apply -k .
```

2. Monitor an application with `ServiceMonitor`

Deploy example application with ServiceMonitor.

kubectl apply -f example-app-with-service-monitor

kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090

Open http://localhost:9090/targets:

We can see serviceMonitor/default/example-app-with-service-monitor/0 in scrape_configs

ServiceMonitor

spec:
  selector:
    matchLabels:
      app: example-app-with-service-monitor
  endpoints:
  - port: web

Prometheus scrape_config

The above ServiceMonitor is converted into a job serviceMonitor/default/example-app-with-service-monitor/0 in scrape_config

relabel_configs: relabel based on the available Kubernetes metadata for endpoints

relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (example-app-with-service-monitor);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web
    replacement: $1
    action: keep
...

kubernetes_sd_configs: Kubernetes service discovery config. ServiceMonitor uses endpoints role.

kubernetes_sd_configs:
- role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    namespaces:
    names:
    - default

You can check the whole config by the following command:

kubectl get secret prometheus-prometheus -n monitoring -o yaml | yq '.data["prometheus.yaml.gz"]' | base64 -d | gunzip | yq '.scrape_configs[] | select(.job_name == "serviceMonitor/default/example-app-with-service-monitor/0")'

scrape_configs

job_name: serviceMonitor/default/example-app-with-service-monitor/0
honor_labels: false
kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
        - default
relabel_configs:
  - source_labels:
      - job
    target_label: __tmp_prometheus_job_name
  - action: keep
    source_labels:
      - __meta_kubernetes_service_label_app
      - __meta_kubernetes_service_labelpresent_app
    regex: (example-app-with-service-monitor);true
  - action: keep
    source_labels:
      - __meta_kubernetes_endpoint_port_name
    regex: web
  - source_labels:
      - __meta_kubernetes_endpoint_address_target_kind
      - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
      - __meta_kubernetes_endpoint_address_target_kind
      - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
      - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
      - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
      - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
      - __meta_kubernetes_pod_container_name
    target_label: container
  - source_labels:
      - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: web
  - source_labels:
      - __address__
    target_label: __tmp_hash
    modulus: 1
    action: hashmod
  - source_labels:
      - __tmp_hash
    regex: $(SHARD)
    action: keep
metric_relabel_configs: []

3. Monitor an application with `PodMonitor`

Deploy example application with PodMonitor.

kubectl apply -f example-app-with-pod-monitor

We can see podMonitor/default/example-app-with-pod-monitor/0 in scrape_configs

relabel_configs: relabel based on the available Kubernetes metadata for pod

kubernetes_sd_configs: Kubernetes service discovery config. ServiceMonitor uses endpoints role.

kubernetes_sd_configs:
- role: pod
    kubeconfig_file: ""
    follow_redirects: true
    namespaces:
    names:
    - default

You can check the whole config by the following command:

kubectl get secret prometheus-prometheus -n monitoring -o yaml | yq '.data["prometheus.yaml.gz"]' | base64 -d | gunzip | yq '.scrape_configs[] | select(.job_name == "podMonitor/default/example-app-with-pod-monitor/0")'

scrape_configs

job_name: podMonitor/default/example-app-with-pod-monitor/0
honor_labels: false
kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
        - default
relabel_configs:
  - source_labels:
      - job
    target_label: __tmp_prometheus_job_name
  - action: keep
    source_labels:
      - __meta_kubernetes_pod_label_app
      - __meta_kubernetes_pod_labelpresent_app
    regex: (example-app-with-pod-monitor);true
  - action: keep
    source_labels:
      - __meta_kubernetes_pod_container_port_name
    regex: web
  - source_labels:
      - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
      - __meta_kubernetes_pod_container_name
    target_label: container
  - source_labels:
      - __meta_kubernetes_pod_name
    target_label: pod
  - target_label: job
    replacement: default/example-app-with-pod-monitor
  - target_label: endpoint
    replacement: web
  - source_labels:
      - __address__
    target_label: __tmp_hash
    modulus: 1
    action: hashmod
  - source_labels:
      - __tmp_hash
    regex: $(SHARD)
    action: keep
metric_relabel_configs: []

4. Clean up

kubectl delete -f example-app-with-pod-monitor
kubectl delete -f example-app-with-service-monitor
kubectl delete -k .
kubectl delete -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
kubectl delete ns monitoring

Important Configurations

PrometheusSpec

serviceMonitorNamespaceSelector: Namespace's labels to match for ServiceMonitor discovery. If nil, only check own namespace. e.g. serviceMonitorNamespaceSelector: {} if you want to monitor all namespaces.
podMonitorNamespaceSelector: Namespace's labels to match for PodMonitor discovery. If nil, only check own namespace. e.g. podMonitorNamespaceSelector: {} if you want to monitor all namespaces.

Operator Implementation

How to create Prometheus server:
- StatefulSet for Prometheus is created in prometheus/statefulset.go with prometheus and prometheus-config-reloader containers.
How to reflect ServiceMonitor and PodMonitor to scrape_config in Prometheus configuration:
1. Operator.createOrUpdateConfigurationSecret calls in cg.Generate in operator.go
```
conf, err := cg.Generate(
    p,
    smons,
    pmons,
    bmons,
    store,
    additionalScrapeConfigs,
    additionalAlertRelabelConfigs,
    additionalAlertManagerConfigs,
    ruleConfigMapNames,
)
```
2. Generate calls in promcfg.go
  - generatePodMonitorConfig
  - generateServiceMonitorConfig
  - generateProbeConfig
  - generateAlertmanagerConfig
3. All generateXXXXConfig call generateK8SSDConfig with different role in promcfg.go
4. Generate kubernetes_sd_config: generateK8SSDConfig
5. conf is returned in createOrUpdateConfigurationSecret in operator.go.
6. conf is compressed.
7. Initialize Secret and set the compressed configData as s.Data[configFilename] = buf.Bytes().
8. Create or update the Secret.

History

2016-11-09 v0.0.1: Prometheus, AlertManager and ServiceMonitor
2017-05-09 v0.1.0: Added Alertmanager in PR#28
2018-06-05 v0.20.0: Added PrometheusRule in PR#1333
2019-06-20 v0.31.0: Added PodMonitor in PR#2566 related to Issue#38
2020-02-10 v0.36.0: Added ThanosRuler in PR#2943
2020-10-27 v0.43.0 Added AlertmanagerConfig in PR#3451

References

prometheus-operator
getting-started: The examples above are from here.
prometheus/discovery/kubernetes: Implementation of Kubernetes Discovery in Prometheus.
Kube-Prometheus-Stack and ArgoCD 2.5 – Server-Side Apply to the Rescue!