OCPBUGS-75913: Fix VolumeSnapshot race condition in backup processing

[jparrill]

What this PR does / why we need it

This PR fixes a race condition in the CheckVolumeSnapshot() function where VolumeSnapshots deleted between the List and Get calls would cause the entire backup to fail. The fix adds graceful handling of NotFound errors, allowing the plugin to skip deleted snapshots and continue processing remaining ones.

Problem: When iterating over VolumeSnapshots from a List call, if a snapshot is deleted between the List (line 286) and subsequent Get call (line 292), the 404 NotFound error was treated as fatal, aborting processing of subsequent snapshots and marking backups as PartiallyFailed.

Solution: Add apierrors.IsNotFound(err) handling in CheckVolumeSnapshot() to match the existing pattern in CheckVolumeSnapshotContent(), allowing deleted snapshots to be skipped gracefully.

Impact: Reduces backup failure rate from 25/100 (observed with ARO-HCP PremiumV2 disks) to near 0% by eliminating this specific race condition.

Which issue(s) this PR fixes

• Fixes https://issues.redhat.com/browse/OCPBUGS-75913
• Depending on NO_JIRA: chore(dependencies): Update dependencies to be aligned with Hypershift repo #184 to fix the tests

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

🤖 Generated with Claude Code via /jira:solve OCPBUGS-75913 origin

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-75913, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it

This PR fixes a race condition in the CheckVolumeSnapshot() function where VolumeSnapshots deleted between the List and Get calls would cause the entire backup to fail. The fix adds graceful handling of NotFound errors, allowing the plugin to skip deleted snapshots and continue processing remaining ones.

Problem: When iterating over VolumeSnapshots from a List call, if a snapshot is deleted between the List (line 286) and subsequent Get call (line 292), the 404 NotFound error was treated as fatal, aborting processing of subsequent snapshots and marking backups as PartiallyFailed.

Solution: Add apierrors.IsNotFound(err) handling in CheckVolumeSnapshot() to match the existing pattern in CheckVolumeSnapshotContent(), allowing deleted snapshots to be skipped gracefully.

Impact: Reduces backup failure rate from 25/100 (observed with ARO-HCP PremiumV2 disks) to near 0% by eliminating this specific race condition.

Which issue(s) this PR fixes

Fixes https://issues.redhat.com/browse/OCPBUGS-75913

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

🤖 Generated with Claude Code via /jira:solve OCPBUGS-75913 origin

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

---

Walkthrough

Updates Go toolchain from 1.24.4 to 1.25.3 and upgrades multiple Kubernetes-related dependencies. Modifies .gitignore to include AI work directory and adjusts cache ignore pattern. Alters error handling in CheckVolumeSnapshot to skip missing VolumeSnapshot objects instead of failing.

Changes

Cohort / File(s)	Summary
Configuration & Dependencies .gitignore, go.mod	.gitignore updated to replace cache ignore pattern and add AI work directory exclusion. go.mod bumps Go toolchain to 1.25.3 and upgrades dependencies including kubernetes, controller-runtime, prometheus, and various golang.org/x packages.
Source Code Logic pkg/common/utils.go	Error handling in CheckVolumeSnapshot modified to continue to next item when VolumeSnapshot is not found, rather than returning an error.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jparrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jparrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jparrill]

/jira refresh

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-75913, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (yli2@redhat.com), skipping review request.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jparrill]

/auto-cc

[enxebre]

seems like you might want to add a test case and a log for this case.
I'm also curious about the success ratio, why do those volumes are deleted?
/lgtm

[michalskrivanek]

LGTM, I observed the problem with a similar probability on BM as well, cherry-picked this fix to latest oadp-1.5 and it fixes the problem indeed

@enxebre IIUC in my case at least the VolumeSnapshot is processed and deleted by the time plugin gets to it

[openshift-ci]

New changes are detected. LGTM label has been removed.

[jparrill]

Separating the deps bump in other commit:

• NO_JIRA: chore(dependencies): Update dependencies to be aligned with Hypershift repo #184

[openshift-ci]

@jparrill: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/verify	a3d51c8	link	true	/test verify
ci/prow/unit	a3d51c8	link	true	/test unit
ci/prow/build	a3d51c8	link	true	/test build

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-75913, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request.

Details

In response to this:

What this PR does / why we need it

This PR fixes a race condition in the CheckVolumeSnapshot() function where VolumeSnapshots deleted between the List and Get calls would cause the entire backup to fail. The fix adds graceful handling of NotFound errors, allowing the plugin to skip deleted snapshots and continue processing remaining ones.

Problem: When iterating over VolumeSnapshots from a List call, if a snapshot is deleted between the List (line 286) and subsequent Get call (line 292), the 404 NotFound error was treated as fatal, aborting processing of subsequent snapshots and marking backups as PartiallyFailed.

Solution: Add apierrors.IsNotFound(err) handling in CheckVolumeSnapshot() to match the existing pattern in CheckVolumeSnapshotContent(), allowing deleted snapshots to be skipped gracefully.

Impact: Reduces backup failure rate from 25/100 (observed with ARO-HCP PremiumV2 disks) to near 0% by eliminating this specific race condition.

Which issue(s) this PR fixes

• Fixes https://issues.redhat.com/browse/OCPBUGS-75913
• Depending on NO_JIRA: chore(dependencies): Update dependencies to be aligned with Hypershift repo #184 to fix the tests

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

🤖 Generated with Claude Code via /jira:solve OCPBUGS-75913 origin

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.