Sharded coordinator does not prevent multiple replicas from reconciling the same project

[scotwells]

Problem

NSO shards projects across replicas using multicluster-runtime's sharded.Coordinator (--cluster-sharding-enabled), so each project should have exactly one replica actively reconciling it. In practice this guarantee doesn't hold for every controller.

Controllers that both (a) watch a project's resources directly (correctly gated by the coordinator) and (b) also watch our single shared downstream cluster via a fixed Source.ForCluster raw source (gateway_controller.go, gateway_resource_replicator_controller.go, httpproxy_controller.go, iroh_dns_controller.go) can still act on a project they don't own. A downstream-side change fires the raw source's watch on every replica unconditionally, and each replica's Reconcile then calls mgr.GetCluster(ctx, req.ClusterName) to reach the project — which succeeds regardless of coordinator ownership, since nothing in multicluster-runtime checks it. We reproduced this directly: with a project's fence Lease held by exactly one replica, all three replicas still reconciled the same object.

This is the root cause of a production incident where a customer's DNS record was left orphaned for 11 days: two replicas raced to reconcile the same project's Gateway, and a stale write landed after a fresh one.

Fix

The gap is in sigs.k8s.io/multicluster-runtime itself — Coordinator has no way for application code to ask whether this process owns a given cluster, and Manager.GetCluster doesn't check ownership before returning a working client. We've implemented and validated a fix upstream (adds Coordinator.Owns(), gates GetCluster, and skips non-owned requests at reconcile dispatch) against a real repro environment. Once that's proposed/merged upstream, NSO needs to bump its sigs.k8s.io/multicluster-runtime dependency to pick it up — no code changes needed on our side beyond the version bump.

Scope note

A separate, already-fixed gap (production's deployed RBAC missing the update verb on coordination.k8s.io/leases, which independently breaks fence-lease re-acquisition after a pod is replaced) is not covered by this issue.