fix(destination): Remove duplicate Job informer - #15523
Open
jojinkb wants to merge 1 commit into
Open
Conversation
Problem
At startup, the destination controller logs:
failed to register Prometheus gauge Desc{fqName: "job_cache_size",
help: "Number of items in the client-go job cache", constLabels:
{cluster="local"}, variableLabels: {}}: duplicate metrics collector
registration attempted
The controller watches Jobs twice: once through a typed client-go
informer (`k8s.Job` in the `InitializeAPI` resource list) and once
through the metadata API (`k8s.Job` in the `InitializeMetadataAPI`
resource list, added in linkerd#11541). Each informer registers a
`job_cache_size` gauge with identical name and labels, so the second
registration fails and produces the warning.
The typed Job informer is unused: all Job lookups in the destination
controller (owner-reference resolution in the workload watcher and the
endpoints watcher's port publisher) go through the metadata API. The
typed informer only maintains a cluster-wide cache of full Job objects
that nothing reads.
Solution
Remove `k8s.Job` from the typed API resource lists in the destination
controller. Jobs remain watched by the metadata API, whose informer now
successfully registers the `job_cache_size` gauge. The exported metric
value is unchanged (the number of Jobs in the cluster); as a side
benefit, the controller no longer caches full Job objects, reducing its
memory footprint and API server watch load.
Validation
Built the destination controller and ran the unit tests for the
affected packages:
go build ./controller/...
go test ./controller/k8s/... ./controller/cmd/destination/... ./controller/api/destination/...
To verify on a cluster: install linkerd from this branch and check that
`kubectl -n linkerd logs deploy/linkerd-destination -c destination`
no longer contains the "duplicate metrics collector registration
attempted" warning, while `job_cache_size` is still exported on the
admin server (`:9996/metrics`).
Fixes linkerd#12710
Signed-off-by: Jojin <jojin.kb@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
At startup, the destination controller logs:
The controller watches Jobs twice: once through a typed client-go informer (
k8s.Jobin theInitializeAPIresource list) and once through the metadata API (k8s.Jobin theInitializeMetadataAPIresource list, added in #11541). Each informer registers ajob_cache_sizegauge with identical name and labels, so the second registration fails and produces the warning.The typed Job informer is unused: all Job lookups in the destination controller (owner-reference resolution in the workload watcher and the endpoints watcher's port publisher) go through
MetadataAPI.GetOwnerKindAndName. The typed informer only maintains a cluster-wide cache of full Job objects that nothing reads.Solution
Remove
k8s.Jobfrom the typed API resource lists in the destination controller. Jobs remain watched by the metadata API, whose informer now successfully registers thejob_cache_sizegauge. The exported metric value is unchanged (the number of Jobs in the cluster); as a side benefit, the controller no longer caches full Job objects, reducing its memory footprint and API server watch load — in line with the goal of #10326, which introduced the metadata API to the destination controller for exactly this reason.Validation
Built the destination controller and ran the unit tests for the affected packages:
To verify on a cluster: install linkerd from this branch and check that
no longer contains the
duplicate metrics collector registration attemptedwarning, whilejob_cache_sizeis still exported on the admin server (:9996/metrics).Fixes #12710