Skip to content

HPA with custom metrics

Overview

1. Deploy Prometheus with prometheus-operator

References: - https://github.com/prometheus-operator/prometheus-operator#quickstart - https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md

Steps:

  1. Create prometheus operator

    kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

  2. Prometheus

    kubectl create ns monitoring

    kubectl apply -k ../../../prometheus-operator -n monitoring

  3. Check UI at http://localhost:30900

    You can check targets

Monitoring RabbitMQ: https://www.rabbitmq.com/kubernetes/operator/operator-monitoring.html

We cannot use ServiceMonitor for RabbitMQ as RabbitMQ service doesn't have prometheus port (15692). We need to use PodMonitor as is recommended in the documentation.

2. Deploy RabbitMQ with operator

https://www.rabbitmq.com/kubernetes/operator/quickstart-operator.html

  1. RabbitMQ Operator

    kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

  2. Create a RabbitMQ cluster

    kubectl apply -f rabbitmq/rabbitmq-cluster.yaml

  3. Create PodMonitor for RabbitMQ

    kubectl apply -f rabbitmq/pod-monitor-rabbitmq.yaml

  4. Get username and password

    username="$(kubectl get secret rabbitmq-default-user -o jsonpath='{.data.username}' | base64 --decode)"
echo "username: $username"
password="$(kubectl get secret rabbitmq-default-user -o jsonpath='{.data.password}' | base64 --decode)"
echo "password: $password"

  5. port-forward

    kubectl port-forward "service/rabbitmq" 15672

    Open: http://localhost:15672/ and use the username and password got in the previous step.

Metrics:

As of 3.8.0, RabbitMQ ships with built-in Prometheus & Grafana support. Support for Prometheus metric collector ships in the rabbitmq_prometheus plugin. The plugin exposes all RabbitMQ metrics on a dedicated TCP port, in Prometheus text format.

Check if rabbitmq_prometheus plugin is enabled.

kubectl exec -it rabbitmq-server-0 -- rabbitmq-plugins list | grep prometheus
[E*] rabbitmq_prometheus               3.8.12

3. Deploy RabbitMQ producer (simple JAVA app)

Create rabbitmq-producer CronJob (run every five minutes)

kubectl apply -f rabbitmq-producer-cronjob.yaml

If you want to run a job manually, you can run the following command after creating CronJob

kubectl create job --from=cronjob/rabbitmq-producer rabbitmq-producer-$(date '+%s')

4. Deploy RabbitMQ consumer (simple JAVA app)

Create rabbitmq-consumer Deployment

kubectl apply -f rabbitmq-consumer-deployment.yaml

5. Deploy Grafana

https://devopscube.com/setup-grafana-kubernetes/

kubectl apply -k ../../../grafana

log in to http://localhost:32111 with admin for both username and password

Dashboard 10991 is already imported

6. Create HPA with custom metrics

  1. Collect metrics from your applications. (Already done by Prometheus)

  2. Extend the Kubernetes custom metrics API with the metrics. (https://github.com/kubernetes-sigs/prometheus-adapter)

    Generate secrets

    cd k8s-prom-hpa
touch metrics-ca.key metrics-ca.crt metrics-ca-config.json
make certs
cd -

  3. Deploy prometheus-adapter

    kubectl create -f ./k8s-prom-hpa/custom-metrics-api

    Check the custom metrics API

    kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready"| jq .

    Custom Metrics API Result

    kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready"| jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/rabbitmq_queue_messages_ready"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "rabbitmq-server-0",
        "apiVersion": "/v1"
      },
      "metricName": "rabbitmq_queue_messages_ready",
      "timestamp": "2021-03-27T12:01:15Z",
      "value": "1274"
    }
  ]
}

  4. With custom API

    kubectl apply -f rabbitmq-consumer-hpa.yaml

    kubectl describe hpa rabbitmq-consumer
Name:                                                                               rabbitmq-consumer
Namespace:                                                                          default
Labels:                                                                             <none>
Annotations:                                                                        <none>
CreationTimestamp:                                                                  Sat, 27 Mar 2021 21:36:14 +0900
Reference:                                                                          Deployment/rabbitmq-consumer
Metrics:                                                                            ( current / target )
  "rabbitmq_queue_messages_ready" on Pod/rabbitmq-server-0 (target average value):  442 / 1
Min replicas:                                                                       1
Max replicas:                                                                       10
Deployment pods:                                                                    4 current / 8 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    SucceededRescale  the HPA controller was able to update the target scale to 8
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from external metric rabbitmq_queue_messages_ready(nil)
  ScalingLimited  True    ScaleUpLimit      the desired replica count is increasing faster than the maximum scale rate
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  20s   horizontal-pod-autoscaler  New size: 8; reason: external metric rabbitmq_queue_messages_ready(nil) above target

7. Observe the behavior

  • Without HPA

  • With HPA

8. Clean up

kubectl delete -f rabbitmq-consumer-hpa.yaml
kubectl delete -k ../../../grafana
for component in rabbitmq rabbitmq-consumer-deployment.yaml rabbitmq-producer-cronjob.yaml; do
    kubectl delete -f $component
done
kubectl delete -k ../../../prometheus-operator -n monitoring
kubectl delete -f ./k8s-prom-hpa/custom-metrics-api
kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
kubectl delete -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
kubectl delete ns monitoring

References