Opt-in REST client metrics: allow overriding histogram buckets (sub-5ms resolution for fast controllers)

[aditya-shantanu]

Context

We've been using the opt-in REST client metrics from #3510 (thanks for that work — also for the cherry-pick to release-0.24 in #3525) during a controller latency investigation in kubernetes-sigs/agent-sandbox, where they were instrumental in finding client-side bottlenecks (e.g. HTTP/2 stream-limit queuing that is invisible to server-side metrics).

Problem

The classic buckets for rest_client_request_duration_seconds and rest_client_rate_limiter_duration_seconds start at 5ms:

Buckets: []float64{0.005, 0.025, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 15.0, 30.0, 60.0},

For controllers whose hot-path requests are fast — in our case warm-path PATCH/PUT calls with real p50s in the 1–15ms range — everything below 5ms collapses into the first bucket and the 5–25ms span into the second, so p50/p90 estimates over the exact range being optimized are mostly quantization artifact.

We're aware NativeHistogramBucketFactor is set, and native histograms fully solve this where the scrape pipeline supports them — but many pipelines still consume only classic buckets.

Ask

Since the file documents that the default buckets intentionally mirror core Kubernetes, changing the defaults seems off the table. Instead: would you accept making buckets overridable at registration, e.g. an options variant of RegisterRESTClientMetrics (or a WithBuckets-style option) that applies caller-supplied buckets before the collectors are registered? Callers who need low-end resolution could then prepend e.g. 0.001, 0.0025 without any divergence in the defaults.

Happy to send a PR if maintainers are open to it.

[sbueringer]

Since the file documents that the default buckets intentionally mirror core Kubernetes, changing the defaults seems off the table.

I mostly see that as a comment documenting where we got the buckets from. I would be okay with changing these buckets if we realize that these values are not ideal.

We have a similar issue in Cluster API, we have many fast reconciles (let's say in the general 1-50ms range) and they are not properly reflected in these buckets. I think reconciles without external calls that only use cached calls are usually quite fast.

We have a bunch of buckets above 100ms, I think it's okay to add some more at the lower end.

@alvaroaleman WDYT?

[alvaroaleman]

I am not a fan of the idea of changing the bucket size, people should find a way to make native histograms work for them.

I am okay with making the buckets configurable.

[sbueringer]

I am okay with making the buckets configurable.

Fine for me, then let's go with that

[GaurangKhanderay]

I'd like to work on this.

Based on the discussion, I'll implement configurable REST client metric histogram buckets while keeping the existing default buckets unchanged.

The goal would be to allow callers to override the buckets at registration time, while preserving the current behavior for existing users that don't provide custom configuration.

I'll take a look at the current RegisterRESTClientMetrics implementation and propose an API that keeps the default path simple while allowing custom buckets when needed.

[alvaroaleman]

@aditya-shantanu is this something you wanted to work on?

[sbueringer]

@GaurangKhanderay Please stop posting AI generated comments here. This violates the Kubernetes AI guidance (https://www.kubernetes.dev/docs/guide/pull-requests/#ai-guidance)