Adapt RabbitMQ connection alert threshold

priteau · priteau · commit ce76182f0e89 · 2023-10-13T17:18:37.000+02:00
The threshold of 1000 connections to a RabbitMQ server can be easily
reached even on moderately-sized deployments.

On several deployments we saw around 25 connections per hypervisor and
between 250 and 750 connections per controller. The threshold is based
on twice these numbers to be conservative.
diff --git a/etc/kayobe/kolla/config/prometheus/rabbitmq.rules b/etc/kayobe/kolla/config/prometheus/rabbitmq.rules
@@ -56,7 +56,7 @@ groups:
     annotations:
       description: RabbitMQ too much unack on {{ $labels.instance }}
   - alert: RabbitMQTooMuchConnections
-    expr: rabbitmq_connections > 1000
+    expr: rabbitmq_connections > {% endraw %}{{ (1500 * groups['controllers'] | length + 50 * groups['compute'] | length) }}{% raw %}
     for: 2m
     labels:
       severity: warning
diff --git a/releasenotes/notes/rabbitmq-connection-alert-85cc7b29ddf8e3c3.yaml b/releasenotes/notes/rabbitmq-connection-alert-85cc7b29ddf8e3c3.yaml
@@ -0,0 +1,5 @@
+---
+features:
+  - |
+    Adapt threshold of RabbitMQ connection alert based on the size of the
+    deployment to avoid spurious alerts.
