Skip to content

[scanner] fix: add pre-deploy PV cleanup steps to deploy-vllm-d#20566

Merged
kubestellar-hive[bot] merged 2 commits into
mainfrom
scanner/fix-20548
Jul 8, 2026
Merged

[scanner] fix: add pre-deploy PV cleanup steps to deploy-vllm-d#20566
kubestellar-hive[bot] merged 2 commits into
mainfrom
scanner/fix-20548

Conversation

@kubestellar-hive

Copy link
Copy Markdown
Contributor

Summary

The deploy-vllm-d job was missing five pre-deploy steps that deploy-pok-prod already uses to clear stuck PVCs/PVs before Helm release recovery. Without them, a stuck or orphaned PersistentVolume causes helm rollback to fail (no clean revision to roll back to), blocking all subsequent deploy steps on every main merge.

Added to deploy-vllm-d (adapted to the kc namespace):

  • Clean up stuck persistent volumes — removes orphaned PVCs referencing missing PVs, and unsticks PVs in deletion >24 h
  • Validate persistent volumes before deploy — blocks the deploy if any PVC/PV is in a broken state Helm cannot recover from
  • Pre-deploy cluster health check — logs pod status and warning events for diagnostics
  • Clean up stuck pods — force-deletes pods stuck in Terminating or ContainerCreating for >10 min
  • Wait for volume detachment — waits for VolumeAttachments to clear before Helm touches the PVCs

Test plan

  • Verify deploy-vllm-d proceeds past PV/PVC cleanup when the cluster is healthy (no-op path)
  • Verify the new steps clear a stuck RWO PVC/PV scenario without blocking the deploy
  • Confirm the workflow passes end-to-end on the next main merge

Fixes #20548

…ploy-vllm-d

The deploy-vllm-d job was missing five steps that deploy-pok-prod uses to
clear stuck PVCs/PVs before attempting Helm release recovery. Without them,
a stuck or orphaned PersistentVolume causes 'helm rollback' to fail (the
release has no clean revision to roll back to), which blocks all subsequent
deploy steps.

Add the following steps, adapted to the kc namespace used by vllm-d:
- Clean up stuck persistent volumes
- Validate persistent volumes before deploy
- Pre-deploy cluster health check
- Clean up stuck pods
- Wait for volume detachment

Fixes #20548

Signed-off-by: scanner <scanner@kubestellar.io>
Signed-off-by: scanner-bot <scanner-bot@kubestellar.io>
Copilot AI review requested due to automatic review settings July 8, 2026 06:30
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Jul 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot can't review bot-authored pull requests automatically. A user with Copilot access can request a review manually.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign eeshaansa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@netlify

netlify Bot commented Jul 8, 2026

Copy link
Copy Markdown

Deploy Preview for kubestellarconsole canceled.

Name Link
🔨 Latest commit 85c83b2
🔍 Latest deploy log https://app.netlify.com/projects/kubestellarconsole/deploys/6a4df9758f2c0a0009133cdd

@github-actions github-actions Bot added the ai-generated Pull request generated by AI label Jul 8, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

👋 Hey @kubestellar-hive[bot] — thanks for opening this PR!

🤖 This project is developed exclusively using AI coding assistants.

Please do not attempt to code anything for this project manually.
All contributions should be authored using an AI coding tool such as:

This ensures consistency in code style, architecture patterns, test coverage,
and commit quality across the entire codebase.


This is an automated message.

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 8, 2026
…over stuck Helm release

When the kc Helm release is in 'failed' state with no good prior
revision, 'helm rollback' itself fails and the Recover stuck Helm
release step exits 1, blocking all subsequent deploy steps.

Add a helm uninstall fallback so the next 'helm upgrade --install'
step can perform a clean fresh install rather than stopping the
pipeline. Applies the same fix to both deploy-vllm-d and
deploy-pok-prod which share the identical recovery pattern.

Fixes #20548

Signed-off-by: kubestellar-hive[bot] <kubestellar-hive[bot]@users.noreply.github.com>
@kubestellar-hive kubestellar-hive Bot merged commit 423d30e into main Jul 8, 2026
21 of 22 checks passed
@kubestellar-prow kubestellar-prow Bot deleted the scanner/fix-20548 branch July 8, 2026 09:24
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

Check out what's new:

Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Post-merge build verification passed

Both Go and frontend builds compiled successfully against merge commit 423d30e64d3a8e4684e91e639c88cae8fe622a5a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated Pull request generated by AI dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tier/3-restricted

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflow failure: Build and Deploy KC

1 participant