fix: retry transient ARM deployment polling 404s

[roi (roivaz)]

Related: AROSLSRE-1787

Historical context: ARO-23706

What

Ports the E2E test client's ARM deployment LRO retry handling into templatize and attaches it to the deployment client.

The policy retries only GET requests to Microsoft.Resources/deployments/.../operationStatuses/... that fail with HTTP 404 and error code DeploymentNotFound. It uses the codebase-standard k8s.io/apimachinery/pkg/util/wait.Backoff helper for two short retries with 1-second initial delay, exponential growth, and jitter.

Why

ARM can briefly return DeploymentNotFound when templatize polls a deployment immediately after creating it. CIHealth recorded five provision failures from August 5-11 where polling failed within 142 ms of deployment creation; the deployments subsequently appeared as Succeeded.

ARO-23706 tracked the same ARM eventual-consistency failure and was closed after the E2E test binary gained retry handling. Applying the same narrowly scoped handling to templatize prevents valid provision pipelines from failing without delaying ordinary deployment-resource GETs used for cache lookup.

Testing

• Unit tests cover matching and non-matching requests, retry success, retry exhaustion, non-retryable errors, and context cancellation
• cd tooling/templatize && go test ./pkg/pipeline -count=1

Special notes for your reviewer

This intentionally does not use the broader implementation from origin/arm-deployment-retry-policy, which also retries ordinary deployment GETs and could delay expected cache misses.

PR Checklist

• PR is scoped to a single task (no mixed concerns)
• Title follows Conventional Commits format
• Summary explains the "Why" behind the change
• Linked to relevant ticket/issue
• Screenshots included (if graph/UI/metrics changes) — not applicable
• Self-reviewed the diff
• CI/CD checks are passing (ignore Tide) — pending draft PR CI
• Draft PR used for WIP (if applicable)
• Commit history is clean (rebased/squashed)
• Tricky code blocks are commented
• Specific reviewers tagged — intentionally omitted for this draft
• All comment threads resolved before merge

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[Copilot]

Pull request overview

This PR updates templatize’s ARM deployments client to tolerate a known ARM eventual-consistency edge case during LRO polling (operationStatuses returning transient 404 DeploymentNotFound immediately after deployment creation), preventing otherwise-valid pipeline runs from failing.

Changes:

• Attach a custom azcore per-call policy to the armresources.DeploymentsClient used by templatize.
• Add a narrowly scoped retry policy that retries only GET .../deployments/.../operationStatuses/... when ARM returns 404 DeploymentNotFound, using wait.ExponentialBackoffWithContext.
• Add unit tests covering match/non-match behavior, retry success/exhaustion, non-retryable errors, and context cancellation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
tooling/templatize/pkg/pipeline/arm.go	Wires the new retry policy into the ARM deployments client via azcorearm.ClientOptions per-call policies.
tooling/templatize/pkg/pipeline/arm_retry_policy.go	Implements the targeted retry policy for transient DeploymentNotFound 404s on deployment operationStatuses polling.
tooling/templatize/pkg/pipeline/arm_retry_policy_test.go	Adds focused unit tests to validate the policy’s retry eligibility and stop conditions.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Steve Kuznetsov (stevekuznetsov)]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: roivaz, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• tooling/templatize/OWNERS [roivaz,stevekuznetsov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment