vault: Fix HAProxy backend healthchecks for standbys

In the previous HAProxy config for Vault, 200, 501 and 503 were treated
as healthy. This allowed for bootstrapping Vault via HAProxy, but made
standby backends appear as unhealthy, leading to a Prometheus alert. We
no longer bootstrap Vault via HAProxy, so we can treat 200 (active) and
429 (standby) as healthy.

Co-Authored-By: Dawud Mehmood <dawud@stackhpc.com>

Author: markgoddard

doc/source/configuration/vault.rst

@@ -204,9 +204,8 @@ HAProxy integration is no longer required for generating OpenStack control plane
          option httpchk GET /v1/sys/health
          # https://www.vaultproject.io/api-docs/system/health
          # 200: initialized, unsealed, and active
-         # 501: not initialised (required for bootstrapping)
-         # 503: sealed (required for bootstrapping)
-         http-check expect rstatus (200|501|503)
+         # 429: standby
+         http-check expect rstatus (200|429)
 
       {% for host in groups['control'] %}
       {% set host_name = hostvars[host].ansible_facts.hostname %}

etc/kayobe/environments/ci-multinode/kolla/config/haproxy/services.d/vault.cfg

@@ -10,9 +10,8 @@ backend vault_back
    option httpchk GET /v1/sys/health
    # https://www.vaultproject.io/api-docs/system/health
    # 200: initialized, unsealed, and active
-   # 501: not initialised (required for bootstrapping)
-   # 503: sealed (required for bootstrapping)
-   http-check expect rstatus (200|501|503)
+   # 429: standby
+   http-check expect rstatus (200|429)
 
 {% for host in groups['control'] %}
 {% set host_name = hostvars[host].ansible_facts.hostname %}

releasenotes/notes/fix-vault-standby-alert-501b743a40971926.yaml

@@ -0,0 +1,8 @@
+---
+fixes:
+  - |
+    Fixes an issue where HashiCorp Vault standby nodes would trigger a
+    Prometheus alert. To apply this fix to an existing system, the HAProxy
+    configuration for Vault (``kolla/config/haproxy/services.d/vault.cfg``)
+    must be manually updated following the `Vault documentation
+    <https://stackhpc-kayobe-config.readthedocs.io/en/stackhpc-2023.1/configuration/vault.html>`.