Improve snapshot analyze context for stuck deletion scenario - #5989
Conversation
There was a problem hiding this comment.
Pull request overview
This PR enhances hcpctl snapshot analyze troubleshooting guidance and data collection for cluster deletions that stall during the Clusters Service (CS) sequential destruct chain—specifically by surfacing mgmt-agent PodWatcher pod lifecycle/eviction signals and documenting how addon pre-delete pod eviction can block ManagedCluster detaching.
Changes:
- Added two new mgmt-agent snapshot KQL queries (
podEvents,podEvictions) for PodWatcher-derived pod lifecycle + eviction visibility inklusterlet-<cluster-id>. - Extended snapshot query README guidance and agent reference docs to describe the CS destruct chain and how it can block at the ManagedCluster/addon layer.
- Added a worked RCA exemplar for a “deletion stuck due to addon pre-delete pod eviction (MemoryPressure)” scenario and linked it from the AI query cookbook.
PR standards notes (per CONTRIBUTING.md#pull-request-standards):
- The PR title does not appear to follow Conventional Commits format.
- The PR description excerpt does not include a tracking ticket/issue reference.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tooling/hcpctl/pkg/snapshot/queries/mgmtAgent/podEvictions/README.md | Documents how to interpret eviction-focused PodWatcher results for klusterlet namespace pods. |
| tooling/hcpctl/pkg/snapshot/queries/mgmtAgent/podEvictions/query.kql | Adds KQL to filter PodWatcher events down to evicted pods for the cluster’s klusterlet namespace. |
| tooling/hcpctl/pkg/snapshot/queries/mgmtAgent/podEvents/README.md | Documents what pod lifecycle signals to look for (evictions, failures, deletes). |
| tooling/hcpctl/pkg/snapshot/queries/mgmtAgent/podEvents/query.kql | Adds KQL to list all PodWatcher pod events for the cluster’s klusterlet namespace. |
| tooling/hcpctl/pkg/snapshot/queries/clustersService/phases/README.md | Adds “uninstalling stuck” guidance pointing to destructor logs + pod eviction checks. |
| tooling/hcpctl/pkg/snapshot/queries/clustersService/logs/README.md | Adds cross-links from CS logs to mgmt-agent pod event/eviction queries when destruct chain stalls. |
| tooling/hcpctl/pkg/snapshot/queries.go | Registers the two new mgmt-agent queries in the snapshot query catalog. |
| tooling/hcpctl/pkg/agent/prompts/references/service-components.md | Adds ACM/ManagedCluster layer responsibilities, failure modes, and relevant logs for deletion analysis. |
| tooling/hcpctl/pkg/agent/prompts/references/architecture.md | Documents the sequential CS destruct chain and where it can block during deletion. |
| tooling/hcpctl/pkg/agent/prompts/exemplars/cluster-deletion-addon-pod-eviction.md | Adds an end-to-end RCA exemplar for addon pre-delete pod eviction blocking deletion. |
| docs/ai/query-cookbook.md | Links the new exemplar from the cookbook’s worked-examples section. |
|
/hold |
92091cd to
2589687
Compare
| @@ -21,6 +21,31 @@ Customer → ARM → Frontend → Backend (async) → Clusters Service → Maest | |||
| 5. **HyperShift** reconciles HostedCluster and NodePool custom resources on the management cluster, | |||
| creating the actual control plane pods in a hosted-control-plane namespace. | |||
|
|
|||
| ## Deletion Flow | |||
2589687 to
bf79d6c
Compare
| | Running destructor 'break-glass-credential-secrets-deleter' for cluster | 7/2/2026, 10:13:38.805 AM | 7/2/2026, 10:19:17.932 AM | 64 | | ||
| | Running destructor 'swift-podnetworkinstance-deleter' for cluster | 7/2/2026, 10:13:38.822 AM | 7/2/2026, 10:19:17.938 AM | 64 | | ||
|
|
||
| ### Why was the `ManagedCluster` destructor stuck? |
There was a problem hiding this comment.
We'll be better off first check the ManagedCluster CR conditions;
As written here it feels like a step was skipped;
The condition will tell that the ManagedCluster CR is awaiting for something to be deleted;
Then check why that something is not finalised;
which lands to the uninstall pods not running
There was a problem hiding this comment.
you also likely need to cite the cs codebase itself to tell us what Running destructor 'hypershift-managed-cluster-destructor' for cluster does - cite the code to make a normative claim of "what ought to happen"
There was a problem hiding this comment.
yes done .
| containerLogs | ||
| | where timestamp between (datetime(2026-07-01) .. datetime(2026-07-03)) | ||
| | where namespace_name == "mgmt-agent" and log.msg == "pod event" | ||
| | where log.namespace == "klusterlet-2r9nhugpbdko2vai55lv2ikki9h9958r" |
There was a problem hiding this comment.
give a hint on how to retrieve the ocm cluster id as indicated above
There was a problem hiding this comment.
we might need a new set of pre-canned queries for events & logs from components we care about in the klusterlet- namespace
| **Suggestions:** | ||
|
|
||
| - The addon pre-delete hook mechanism should be resilient to transient pod eviction. Consider using a `Job` with | ||
| `restartPolicy: OnFailure` and a configurable `backoffLimit` so that evicted pods are automatically retried. | ||
| - Addon pre-delete pods could use `PriorityClass` or `PodDisruptionBudget` settings to reduce the likelihood of | ||
| eviction during critical cleanup operations. | ||
| - The destruct chain in Clusters Service could implement a timeout after which it force-removes addon finalizers | ||
| and proceeds with the rest of the chain, preventing indefinite blocking. |
There was a problem hiding this comment.
This can be dropped
There was a problem hiding this comment.
done
|
|
||
| 1. **ARM** delivers the DELETE request to the **Frontend**, which creates an async operation. | ||
| 2. The **Backend** translates it into a Clusters Service API call. | ||
| 3. **Clusters Service** sets the cluster state to `'uninstalling'` and runs the destruct chain: |
There was a problem hiding this comment.
There are more deletion chains; and node pool has different set.
I wonder if we can make it generic?
There was a problem hiding this comment.
I suggest to remove the mgmtAgent folder in favor of an acm folder in https://github.com/Azure/ARO-HCP/tree/5a8758f3b797aaf54f3caee3d565a37bf29f2eb6/tooling/hcpctl/pkg/snapshot/queries
that contains managedClusterConditions folder and enlist all the managedCluster CR conditions: similar to how its done for the hostedCluster or node pool but leveraging the logs of the CR from the mgmt-agent;
The conditions will indicate why deletion is stuck.
This way we can drop https://github.com/Azure/ARO-HCP/tree/5a8758f3b797aaf54f3caee3d565a37bf29f2eb6/tooling/hcpctl/pkg/snapshot/queries/mgmtAgent which are kind of specific and are already captured in the examplar .
Once you've the managedCluster conditions;
Add a reference to them in: https://github.com/Azure/ARO-HCP/blob/5a8758f3b797aaf54f3caee3d565a37bf29f2eb6/tooling/hcpctl/pkg/snapshot/queries/clustersService/phases/README.md in place of what you've now.
cluster reaches uninstalling but never completes, review logs/clustersService/logs.md for repeated Not continuing to the next destructor messages, and check conditions/acm/managedClusterConditions.md for .....
PS: Notice how I used cluster, you might want to review the node pool destructor code as well to capture corresponding destructor logs and add a similar line
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
tooling/hcpctl/pkg/snapshot/queries/acm/managedClusterConditions/query.kql:13
- The query is intended to return the latest ManagedCluster condition snapshot, but
take_any()combined with grouping byeventcan decoupleobservedTimefrom the correspondinglog.object, andtop 1can pick an arbitrary event row. This can yield conditions from an older object while labeling it with a newer timestamp (or vice versa). Consider selecting the single most-recent row witharg_max(timestamp, ...)and then expanding conditions from that object.
| summarize content=take_any(log.object), observedTime=take_any(timestamp) by event=tostring(log.event)
| top 1 by observedTime desc
| mv-expand condition = content.status.conditions
| project observedTime, type=tostring(condition.type), status=tostring(condition.status), reason=tostring(condition.reason), message=tostring(condition.message), lastTransitionTime=todatetime(condition.lastTransitionTime)
docs/ai/query-cookbook.md:182
- PR description mentions new snapshot queries named
podEvents/podEvictionsand an exemplarcluster-deletion-addon-pod-eviction.md, but this PR addsacm/klusterletLogs,acm/klusterletEvents, andcluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md. Please align the PR description with the actual query/exemplar names to avoid confusion when searching docs/history.
- `cluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md` — cluster deletion stuck at ManagedCluster destructor due to addon pod eviction (MemoryPressure)
tooling/hcpctl/pkg/agent/prompts/references/architecture.md:52
- This section lists specific cluster destructors, but the exemplar in this PR shows additional destructors (e.g. namespaces / Azure cleanup) appearing in Clusters Service logs. As written, this reads like a complete list and could mislead analysis. Suggest marking the list as non-exhaustive and pointing readers to CS logs for the authoritative step list.
3. **Clusters Service** sets the cluster state to `'uninstalling'` and runs the cluster destruct
chain (see `aro-hcp-clusters-service` repo, `pkg/clusterprovisioner/acm/destruct/`):
- `hypershift-managed-cluster-destructor`: waits for the **ManagedCluster** (ACM/MCE) CR
to be fully deleted. Deletion requires **ManagedClusterAddon** pre-delete hook pods to
complete and remove their finalizers first. If the ManagedCluster CR still exists, the
Manyanda Chitimbo (machi1990)
left a comment
Manyanda Chitimbo (machi1990)
left a comment
There was a problem hiding this comment.
/lgtm
| reason, | ||
| message, | ||
| type | ||
| | order by timestamp asc |
There was a problem hiding this comment.
Can you summarize these like we do for all other events?
There was a problem hiding this comment.
yes done .
| @@ -0,0 +1,16 @@ | |||
| cluster('{{ .ClusterURI }}').database('{{ .ServiceDatabase }}').table('containerLogs') | |||
There was a problem hiding this comment.
Shouldn't this be kubernetesResourceSnapshots?
There was a problem hiding this comment.
The query worked only when I set it to containerLogs where as for kubernetesResourceSnapshots it returns empty results for the cluster id and timeline .
There was a problem hiding this comment.
shared the query results over DM .
There was a problem hiding this comment.
So this was added recently so have updated in latest commit .
| @@ -0,0 +1,13 @@ | |||
| cluster('{{ .ClusterURI }}').database('{{ .ServiceDatabase }}').table('containerLogs') | |||
There was a problem hiding this comment.
Shouldn't this be kubernetesResourceSnapshots?
There was a problem hiding this comment.
same as responded above .
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (2)
tooling/hcpctl/pkg/snapshot/queries.go:652
- PR description says it adds “two new mgmt-agent KQL snapshot queries (podEvents, podEvictions)” and an RCA exemplar named
cluster-deletion-addon-pod-eviction.md, but the diff adds three queries (managedClusterConditions,klusterletEvents,klusterletLogs) and the exemplar file iscluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md. Please align the PR description (or rename the new queries/exemplar) so reviewers and release notes don’t drift.
// --- ACM: ManagedCluster conditions and klusterlet events ---
{
component: "acm",
queryName: "managedClusterConditions",
templatePath: "queries/acm/managedClusterConditions/query.kql",
docs/ai/query-cookbook.md:182
- The “Worked-example analyses” list includes three exemplar filenames that don’t exist in
tooling/hcpctl/pkg/agent/prompts/exemplars/(cluster-cleanup-unknown-failure.md,kubernetes-events.md,mgmt-agent-event-logs.md). This makes the cookbook’s cross-references misleading/broken; consider updating the list to match the current exemplars directory (or adding the missing files).
- `cluster-installation-azure-disk-failure.md`
- `cluster-cleanup-unknown-failure.md`
- `kubernetes-events.md` — `ServiceLogs.kubernetesEvents` and ad-hoc filtering beyond snapshot `controlPlaneEvents`
- `mgmt-agent-event-logs.md` — ad-hoc KQL for mgmt-agent `resource event` / `pod event` timelines
- `cluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md` — cluster deletion stuck at ManagedCluster destructor due to addon pod eviction (MemoryPressure)
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (1)
tooling/hcpctl/pkg/agent/prompts/references/architecture.md:86
- The reference to the ManagedCluster conditions snapshot is missing the
.mdsuffix, so it won’t match the generated snapshot output file name (managedClusterConditions.md) and the link won’t work.
For cluster deletion stuck at the ManagedCluster step, check
`conditions/acm/managedClusterConditions` for the ManagedCluster state. For node pool deletion
stuck at the CR destructor step, check HyperShift NodePool conditions and ManifestWork status.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: machi1990, stevekuznetsov, venkateshsredhat The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/unhold |
Uh oh!
There was an error while loading. Please reload this page.