Skip to content

fix(destination): Remove duplicate Job informer - #15523

Open
jojinkb wants to merge 1 commit into
linkerd:mainfrom
jojinkb:fix-destination-duplicate-metrics
Open

fix(destination): Remove duplicate Job informer#15523
jojinkb wants to merge 1 commit into
linkerd:mainfrom
jojinkb:fix-destination-duplicate-metrics

Conversation

@jojinkb

@jojinkb jojinkb commented Jul 25, 2026

Copy link
Copy Markdown

Problem

At startup, the destination controller logs:

failed to register Prometheus gauge Desc{fqName: "job_cache_size", help: "Number of items in the client-go job cache", constLabels: {cluster="local"}, variableLabels: {}}: duplicate metrics collector registration attempted

The controller watches Jobs twice: once through a typed client-go informer (k8s.Job in the InitializeAPI resource list) and once through the metadata API (k8s.Job in the InitializeMetadataAPI resource list, added in #11541). Each informer registers a job_cache_size gauge with identical name and labels, so the second registration fails and produces the warning.

The typed Job informer is unused: all Job lookups in the destination controller (owner-reference resolution in the workload watcher and the endpoints watcher's port publisher) go through MetadataAPI.GetOwnerKindAndName. The typed informer only maintains a cluster-wide cache of full Job objects that nothing reads.

Solution

Remove k8s.Job from the typed API resource lists in the destination controller. Jobs remain watched by the metadata API, whose informer now successfully registers the job_cache_size gauge. The exported metric value is unchanged (the number of Jobs in the cluster); as a side benefit, the controller no longer caches full Job objects, reducing its memory footprint and API server watch load — in line with the goal of #10326, which introduced the metadata API to the destination controller for exactly this reason.

Validation

Built the destination controller and ran the unit tests for the affected packages:

go build ./controller/...
go test ./controller/k8s/... ./controller/cmd/destination/... ./controller/api/destination/...

To verify on a cluster: install linkerd from this branch and check that

kubectl -n linkerd logs deploy/linkerd-destination -c destination

no longer contains the duplicate metrics collector registration attempted warning, while job_cache_size is still exported on the admin server (:9996/metrics).

Fixes #12710

Problem

At startup, the destination controller logs:

    failed to register Prometheus gauge Desc{fqName: "job_cache_size",
    help: "Number of items in the client-go job cache", constLabels:
    {cluster="local"}, variableLabels: {}}: duplicate metrics collector
    registration attempted

The controller watches Jobs twice: once through a typed client-go
informer (`k8s.Job` in the `InitializeAPI` resource list) and once
through the metadata API (`k8s.Job` in the `InitializeMetadataAPI`
resource list, added in linkerd#11541). Each informer registers a
`job_cache_size` gauge with identical name and labels, so the second
registration fails and produces the warning.

The typed Job informer is unused: all Job lookups in the destination
controller (owner-reference resolution in the workload watcher and the
endpoints watcher's port publisher) go through the metadata API. The
typed informer only maintains a cluster-wide cache of full Job objects
that nothing reads.

Solution

Remove `k8s.Job` from the typed API resource lists in the destination
controller. Jobs remain watched by the metadata API, whose informer now
successfully registers the `job_cache_size` gauge. The exported metric
value is unchanged (the number of Jobs in the cluster); as a side
benefit, the controller no longer caches full Job objects, reducing its
memory footprint and API server watch load.

Validation

Built the destination controller and ran the unit tests for the
affected packages:

    go build ./controller/...
    go test ./controller/k8s/... ./controller/cmd/destination/... ./controller/api/destination/...

To verify on a cluster: install linkerd from this branch and check that
`kubectl -n linkerd logs deploy/linkerd-destination -c destination`
no longer contains the "duplicate metrics collector registration
attempted" warning, while `job_cache_size` is still exported on the
admin server (`:9996/metrics`).

Fixes linkerd#12710

Signed-off-by: Jojin <jojin.kb@gmail.com>
@jojinkb
jojinkb requested a review from a team as a code owner July 25, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

duplicate metrics in destination controller

1 participant