Skip to content

Use Server-Side Apply for co-owned Gateway writes to eliminate 409 contention #305

Description

@ecv

Problem

The Gateway backing an HTTPProxy has two legitimate writers: the HTTPProxy controller owns its spec (via CreateOrUpdate), and the gateway controller writes one field on top — the default listener hostname (prepareUpstreamGateway), which it has to do post-create because the hostname is derived from the Gateway UID.

They never write the same field, but read-modify-write CreateOrUpdate collides on the shared resourceVersion whenever the two writes interleave — a burst of 409s at tunnel creation. We handle that today with retry-on-conflict (#303), which is the correct pattern for the contention but is still a workaround: it papers over the collision instead of removing it, and it forces the HTTPProxy controller to read the controller-owned hostname back off the live object each reconcile so it doesn't clobber it.

Proposal

Move the contended writes to Server-Side Apply with a stable field manager per controller. Each controller applies only the fields it owns; the apiserver merges by per-field ownership, so there's no read-modify-write and no 409 — for the whole class, not just this instance.

Benefits:

  • Eliminates the conflict class rather than retrying through it — no more creation-time 409 bursts, no retry round-trips on the hot path.
  • Removes the hostname read-back coordination in the HTTPProxy controller: it simply stops owning that field, and the gateway controller owns it. Ownership becomes explicit instead of hand-coded.

Scope / considerations

  • Start with the Gateway (the hot path), then the sibling CreateOrUpdate writes on the same pattern (HTTPRoute, HTTPRouteFilter, EndpointSlice) as follow-ups.
  • Gateway.spec.listeners is a listType=map keyed by name, so two managers can own different fields of the same listener entry — confirm the applied objects are structured so the hostname stays owned by the gateway controller and everything else by the HTTPProxy controller.
  • Decide conflict handling (force-apply vs surfacing conflicts) and pick stable field-manager names.
  • This introduces SSA where the controllers are currently all CreateOrUpdate, so it's a pattern change worth doing deliberately; keep the retry-on-conflict from fix: requeue HTTPProxy promptly on write conflict #303 as a safety net during the migration and drop it once SSA covers the writes.

Context

Root-caused while reviewing #303 (retry-on-conflict fix for #166). SSA raised by @scotwells there as the class-level alternative. Related: #294 (created the Gateway a reconcile earlier, which widened the write-overlap window and surfaced the 409 burst).

Metadata

Metadata

Assignees

No one assigned

    Labels

    tech-debtWorkarounds/pins/shortcuts to revisit

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions