fix: remove max without(prometheus_replica) from histogram_quantile (AROSLSRE-1746)

[Inbharaj Mani (inbharajmani)]

Summary

• Remove max without(prometheus_replica) from histogram_quantile expressions in the FrontendPathLatency alert and the control plane latency dashboard.
• Per Prometheus docs, sum by (le, ...) is the correct aggregation for histogram buckets — histogram_quantile works on ratios between buckets, so doubled counts from HA replicas cancel out.
• Remove FrontendPathLatency from known issues (added in fix: add FrontendPathLatency to known issues (AROSLSRE-1746) #6449) since the root cause is fixed in this PR.
• The existing FrontendLatency alert (1h window, info severity) already uses sum without max and is unaffected.
• Non-histogram panels (request rate, SLO fraction) correctly retain max without for dedup.

Root Cause

Using max without(prometheus_replica) per-bucket picks from different HA replicas independently, breaking cumulative histogram monotonicity. With sparse CI traffic (1–2 requests per 5m window), histogram_quantile interpolated into empty high-latency buckets — reporting P99 = 12.083s when the actual max request was 26ms (a 466x overestimate).

This is the same class of problem documented in prometheus/prometheus#2610 and prometheus/prometheus#13671: non-atomic data fed into histogram_quantile breaks the monotonicity invariant that the algorithm depends on.

The Prometheus docs explicitly warn about this:

"if there are non-monotonic bucket counts even after this adjustment, they are increased to the value of the previous buckets to enforce monotonicity. The latter is evidence for an actual issue with the input data... If you encounter this annotation, you should find and remove the source of the invalid data."

The source of the invalid data was max without(prometheus_replica). This PR removes it.

Why sum is correct

The Prometheus docs say: "To aggregate, use the sum() aggregator around the rate() function." — no mention of max for histogram buckets anywhere.

sum by (le, route, method, cluster) sums across all other labels including prometheus_replica. The doubled counts from 2 replicas don't affect the quantile because histogram_quantile divides buckets by each other — the 2x factor cancels:

1 replica:  bucket{le="1.0"} = 990,  bucket{le="+Inf"} = 1000 → 99% under 1s
2 replicas: bucket{le="1.0"} = 1980, bucket{le="+Inf"} = 2000 → still 99% under 1s

Evidence

Failing E2E run: pull-ci-Azure-ARO-HCP-main-e2e-parallel/2085205020218757120

Kusto proof — alert fired with expressionValue=12.083:

-- Database: MonitoringEvents, Cluster: https://hcp-dev-us-2.eastus2.kusto.windows.net
alertEvents
| where firedDateTime == datetime(2026-08-06T04:21:20.0868379Z)
| where alertRule has "FrontendPathLatency"
| where targetResourceId has "j8757120"
| where monitorCondition == "Fired"
| project firedDateTime, expressionValue = toreal(alertContext.expressionValue),
    cluster = tostring(alertContext.labels.cluster),
    method = tostring(alertContext.labels.method),
    route = tostring(alertContext.labels.route)

Kusto proof — all 9,475 requests across the entire E2E run, none exceeded 1s:

-- Database: ServiceLogs, Cluster: https://hcp-dev-us-2.eastus2.kusto.windows.net
frontendLogs
| where timestamp between (datetime(2026-08-06T03:50:00Z) .. datetime(2026-08-06T05:20:00Z))
| where cluster == "ci01-j8757120-svc"
| where msg == "response complete"
| extend duration_ms = round(toreal(log.duration) * 1000, 2)
| summarize count(), max(duration_ms), countif(duration_ms > 1000) by request_method

Result: 9,475 requests, max 791ms, 0 exceeded 1 second.

Tracked in AROSLSRE-1746.

Test plan

• Verify FrontendPathLatency alert rule test passes (promtool via CI)
• Verify dashboard JSON is valid (no broken expressions)
• Confirm the FrontendLatency alert (1h window) is unchanged and already uses the same sum pattern
• Confirm non-histogram panels (request rate, SLO fraction) still retain max without for correct dedup
• Verify FrontendPathLatency is removed from known issues (no longer needed with root cause fixed)

[Copilot]

Pull request overview

This PR fixes incorrect PromQL aggregation for histogram quantiles by removing max without(prometheus_replica) before histogram_quantile, preventing broken histogram monotonicity and false-positive latency signals in HA Prometheus setups.

Changes:

• Update the Frontend control plane latency dashboard P50/P95/P99 panels to use sum by (le, ...) (rate(..._bucket)) directly under histogram_quantile.
• Update the FrontendPathLatency PrometheusRule alert to use the same correct histogram aggregation pattern and document the rationale (AROSLSRE-1746).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
observability/grafana-dashboards/frontend/frontend-control-plane-latency.json	Removes HA replica max without(...) from histogram_quantile queries for dashboard latency quantiles.
frontend/alerts/frontend-path-latency-prometheusRule.yaml	Removes max without(prometheus_replica) from the alert’s histogram quantile expression and adds explanatory comments about the HA/monotonicity issue.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

frontend/alerts/frontend-path-latency-prometheusRule.yaml:23

• PR standards require before/after screenshots for changes that affect dashboards, alerting rules, or other metrics/visualizations. This PR updates both a Grafana dashboard and Prometheus alerting expressions, so please add annotated before/after screenshots to the PR description.

    # HA Prometheus replicas are handled by sum (not max): histogram_quantile works on
    # ratios between buckets, so doubled counts from two replicas cancel out. Using max
    # per-bucket broke cumulative monotonicity and caused false positives (AROSLSRE-1746).

observability/grafana-dashboards/frontend/frontend-control-plane-latency.json:90

• This dashboard still has a histogram-bucket ratio panel ("Share of requests ≤ 1s by method + route") that dedups each bucket with max without(prometheus_replica) (see expression around line ~291). That has the same non-atomic-per-bucket problem described in the PR (it can select different replicas for le="1" vs le="+Inf", producing incorrect ratios, potentially >1). Consider switching that panel to the same sum by (...) (rate(...)) pattern so HA replica duplication cancels out in the ratio, rather than per-bucket max dedup.

          "datasource": { "type": "prometheus", "uid": "${datasource}" },
          "editorMode": "code",
          "expr": "histogram_quantile(\n  0.99,\n  sum by (le, route, method) (\n    rate(frontend_http_requests_duration_seconds_bucket{cluster=\"$cluster\",route!=\"/subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationresults/{operationid}\"}[$__rate_interval])\n  )\n)",
          "legendFormat": "{{method}} {{route}}",
          "refId": "A"

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: inbharajmani
Once this PR has been reviewed and has the lgtm label, please assign geoberle for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• dev-infrastructure/OWNERS
• frontend/alerts/OWNERS
• observability/OWNERS
• test/cmd/aro-hcp-tests/gather-observability/known-issues/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

observability/grafana-dashboards/frontend/frontend-control-plane-latency.json:88

• This PR changes a Grafana dashboard and an alerting rule expression. CONTRIBUTING.md requires before/after screenshots for any dashboard/graph/metrics/alerting-rule change (CONTRIBUTING.md:95-98). Please add the required screenshots to the PR description so reviewers can validate the visual/behavioral impact.

          "expr": "histogram_quantile(\n  0.99,\n  sum by (le, route, method) (\n    rate(frontend_http_requests_duration_seconds_bucket{cluster=\"$cluster\",route!=\"/subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationresults/{operationid}\"}[$__rate_interval])\n  )\n)",

frontend/alerts/frontend-path-latency-prometheusRule.yaml:28

• gather-observability/queries.yaml still defines “Frontend Request Latency” as histogram_quantile(... sum by (...) (max without(prometheus_replica) (rate(..._bucket...)))) and its description says it “matches FrontendPathLatency”. With this PR changing the alert (and dashboard) to remove the per-bucket max without, that gather query is now out of sync and can still reproduce the same non-monotonic histogram issue during CI triage. Please update the gather query to use the same sum by (le, ...) (rate(...)) shape as the alert.

    # HA Prometheus replicas are handled by sum (not max): histogram_quantile works on
    # ratios between buckets, so doubled counts from two replicas cancel out. Using max
    # per-bucket broke cumulative monotonicity and caused false positives (AROSLSRE-1746).
    - alert: FrontendPathLatency
      expr: |
        histogram_quantile(0.99,
          sum by (le, route, method, cluster) (
            rate(frontend_http_requests_duration_seconds_bucket{route!="/subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationresults/{operationid}"}[5m])