Gateway policy status writes fail the reconcile on conflict instead of retrying

[ecv]

Summary

Now that #324 has unblocked upstream status writes, the gateway resource replicator surfaces a second, milder failure on the same path: when a tenant's policy object is modified between the replicator's read and its status write, the update returns a conflict and the entire reconcile fails. Status does eventually converge — the reconcile requeues with backoff — but every occurrence logs an ERROR, and a conflict is a routine, expected outcome of optimistic concurrency, not a fault.

Two costs:

• Log noise on a healthy path. Operators grepping the replicator for real problems wade through conflicts that resolved themselves. This is precisely the signal that made Tenants see no status on Envoy Gateway policies: manager lacks /status RBAC #322 hard to spot for as long as it was.
• Inflated reconcile error ratio. controller_runtime_reconcile_total{result="error"} counts these, and ControllerReconcileErrorRatioHigh / ControllerReconcileErrorRatioCritical (config/telemetry/alerts/controller-health.yaml) are ratio alerts over that series. Routine conflicts eat into the 20% / 50% budgets, so a genuine regression has less headroom before it trips — or the noise floor makes the alert untrustworthy.

Observed on staging, v0.0.0-main-20260730-163828, several times an hour during e2e churn:

2026-07-30T17:19:48Z	ERROR	Reconciler error	{"controller": "gateway_resource_replicator", "error": "failed to update upstream status for gateway.envoyproxy.io/v1alpha1, Kind=BackendTrafficPolicy default/e2e-rate-limiting-1785431984-btp: Operation cannot be fulfilled on backendtrafficpolicies.gateway.envoyproxy.io \"e2e-rate-limiting-1785431984-btp\": the object has been modified; please apply your changes to the latest version and try again"}

Cause

Three status-write sites take the object read at the top of Reconcile and call Status().Update() on a deep copy, with no re-fetch and no conflict retry:

Path	Location
Downstream→upstream status sync	internal/controller/gateway_resource_replicator_controller.go:487
Upstream status clear	internal/controller/gateway_resource_replicator_controller.go:526
Held-policy condition	internal/controller/gateway_resource_replicator_securitypolicy_guard.go:176

A conflict is returned to the controller-runtime as a plain error, so it counts as a failed reconcile rather than a retry.

Expected

Treat conflict as a retry, not a failure: wrap each status write in retry.RetryOnConflict with a fresh Get inside the closure, or return a short explicit requeue on apierrors.IsConflict without erroring the reconcile.

This is the same class as #166, where the HTTPProxy controller backed off for minutes after a 409 burst instead of retrying promptly.

Acceptance criteria

• Conflicts on the three status-write paths retry against a freshly-read object instead of failing the reconcile
• Reconciler error entries for the object has been modified no longer appear on the replicator during normal e2e churn
• Conflicts no longer increment controller_runtime_reconcile_total{result="error"}, so the controller-health ratio alerts reflect real faults only
• Genuine (non-conflict) status-write failures still error and still surface

Follow-up from #324 / #322.

[ecv]

Worth pairing with https://github.com/datum-cloud/infra/issues/3745 — NSO exports no metrics scrape target in production, so the error-ratio budgets this noise eats into aren't observable there yet. Fixing the conflict retry makes the ratio meaningful; fixing the infra side makes it visible.