feat(hcpctl): add taxonomy classification to snapshot analyze

[Giulio Frasca (gmfrasca)]

ARO-28427

What

Adds taxonomy classification to hcpctl snapshot analyze output. Each analysis now includes an L1 category, optional L2 subcategory (for Product Failures), and a confidence score assigned by the LLM.

Why

Automated classification enables downstream aggregation and triage of test failures by category (Azure Problems, Deployment Failures, Product Failures, Test Reliability) without manual review of each analysis.

Testing

Unit tests (classification_test.go): schema parsing (with/without classification), validation (14 subtests covering all L1/L2/confidence rules and boundary values), and markdown rendering (with L2, without L2, without classification).

Manual verification with hcpctl snapshot analyze:

1. Build hcpctl:

cd tooling/hcpctl && make build

2. Gather a snapshot from a failed Prow job:

hcpctl snapshot from-prow-job --url <prow-job-url> --test <test-name> --output-dir <snapshot-path>

3. Run the analysis:

hcpctl snapshot analyze <snapshot-path>/path/to/<test-name> \
--aro-hcp <path-to-aro-hcp-repo> \
--hypershift <path-to-hypershift-repo> \
--maestro <path-to-maestro-repo> \
--clusters-service <path-to-CS-repo>

4. Verify analysis.json contains a classification object with l1_category, confidence, and l2_subcategory (when applicable)
5. Verify analysis.md contains a "Classification" section with Category, Component (if Product Failures), and Confidence percentage

PR Checklist

• PR is scoped to a single task (no mixed concerns)
• Title follows Conventional Commits format
• Summary explains the "Why" behind the change
• Linked to relevant ticket/issue
• Screenshots included (if graph/UI/metrics changes)
• Self-reviewed the diff
• CI/CD checks are passing (ignore Tide)
• Draft PR used for WIP (if applicable)
• Commit history is clean (rebased/squashed)
• Tricky code blocks are commented
• Specific reviewers tagged
• All comment threads resolved before merge

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: gmfrasca
Once this PR has been reviewed and has the lgtm label, please assign janboll for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• tooling/hcpctl/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[Copilot]

Pull request overview

Adds an LLM-provided failure taxonomy classification to hcpctl snapshot analyze analysis artifacts so downstream systems can aggregate and triage failures by category/component with an associated confidence score.

Changes:

• Introduces a classification object (L1 category, optional L2 subcategory, confidence) to the analysis JSON schema and hydrates it through to markdown rendering.
• Adds validation rules enforcing allowed L1/L2 values and confidence bounds, and updates the agent system prompt to require the classification output.
• Adds unit tests for parsing/validation/rendering behavior around classification.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tooling/hcpctl/pkg/agent/validation.go	Validates presence and correctness of classification (L1/L2/confidence).
tooling/hcpctl/pkg/agent/schema.go	Adds taxonomy constants and Classification struct; wires into Draft/Hydrated chain schemas.
tooling/hcpctl/pkg/agent/render.go	Renders a “Classification” markdown section including confidence percentage.
tooling/hcpctl/pkg/agent/prompts/system.md	Updates the agent contract to require classification and defines taxonomy rules.
tooling/hcpctl/pkg/agent/prompts/exemplars/clusters-service-worker-race-condition.md	Adds a Classification section to an exemplar.
tooling/hcpctl/pkg/agent/prompts/exemplars/cluster-installation-ignition-failure.md	Adds a Classification section to an exemplar.
tooling/hcpctl/pkg/agent/prompts/exemplars/cluster-installation-azure-disk-failure.md	Adds a Classification section to an exemplar.
tooling/hcpctl/pkg/agent/prompts/exemplars/cluster-creation-timeout-z-stream-upgrade.md	Adds a Classification section to an exemplar.
tooling/hcpctl/pkg/agent/hydration.go	Carries classification from draft into hydrated output.
tooling/hcpctl/pkg/agent/classification_test.go	Adds unit tests for classification parsing, validation, and rendering.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Giulio Frasca (gmfrasca)]

Intentional - confidence is omitted from exemplars so the LLM infers it from the evidence rather than mimicking a hardcoded value. The system prompt and JSON schema example already demonstrate the confidence field; the exemplars teach format and category assignment, not specific confidence values.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

tooling/hcpctl/pkg/agent/validation.go:149

• The confidence validation can’t distinguish between an explicitly provided 0.0 and a missing confidence field (JSON unmarshalling defaults float64 to 0). If the agent omits confidence, this check will pass and downstream consumers will treat it as 0% confidence, contradicting the prompt requirement that confidence is always present.

		if draft.Classification.Confidence < 0 || draft.Classification.Confidence > 1 {

tooling/hcpctl/pkg/agent/classification_test.go:212

• In the "valid" test cases, this only asserts there are no classification problems; other validation problems would still allow these cases to pass and could mask regressions in non-classification draft validation.

			if tt.wantCategory == "" {
				if len(problems) > 0 {
					t.Errorf("expected no classification problems, got %d: %v", len(problems), problems)
				}
				return
			}

[Giulio Frasca (gmfrasca)]

/retest