feat(triage-security): add format validation to Step 2.1 matrix loading

[ruromero]

Summary

• Add Step 2.1.1 to validate each security matrix file against the canonical template (docs/templates/security-matrix.template.md) before aggregation in Step 2.1
• Auto-repair safe fixes (missing Forward Pointer section, whitespace in column headers); warn on critical issues (missing Supportability Matrix/Ecosystem Mappings sections, column mismatches)
• Add 4 eval assertions (IDs 29–32) and 3 new malformed matrix fixtures covering valid matrix, missing section, wrong columns, and auto-repair scenarios

Implements TC-5136

Test plan

• Run eval 29: valid mock matrix passes validation silently
• Run eval 30: missing Ecosystem Mappings section triggers warning and halt
• Run eval 31: wrong Ecosystem Mappings columns show expected vs actual diff
• Run eval 32: missing Forward Pointer auto-repairs without user prompt
• Verify existing evals (1–28) are unaffected by the new sub-step

🤖 Generated with Claude Code

Summary by Sourcery

Introduce a format validation sub-step for security matrix files in the triage security version impact analysis workflow.

New Features:

• Add a Step 2.1.1 matrix format validation phase that checks security matrix files against the canonical template before aggregation.

Enhancements:

• Document auto-repair behavior and warning handling for common security matrix format issues, including missing sections and column mismatches.

Tests:

• Add new malformed security matrix fixtures and eval entries to cover valid, missing-section, wrong-column, and auto-repair validation scenarios.

[sourcery-ai]

Reviewer's Guide

Adds a new matrix format validation sub-step to triage-security version impact analysis, and introduces eval fixtures to test valid, missing section, wrong columns, and auto-repair scenarios for security matrix files.

File-Level Changes

Change	Details	Files
Introduce Step 2.1.1 to validate security matrix files against a canonical template before aggregation, including auto-repair behavior and user-facing warnings.	Define a new validation sub-step that reads the canonical security matrix template and extracts required section headings and Ecosystem Mappings column names. Specify validation rules for presence of required sections, exact Ecosystem Mappings column structure, and basic Markdown table parsability for Supportability Matrix and Ecosystem Mappings. Describe auto-repair behaviors for missing Forward Pointer section and whitespace normalization in table header column names, with corresponding log messages. Define warning conditions for missing critical sections, mismatched Ecosystem Mappings columns, and malformed tables, including halt behavior per stream and expected diff output for column mismatches. Add user interaction flow for presenting validation results, differentiating pass, repaired, and warnings outcomes, and controlling whether triage proceeds or aborts based on user choice.	plugins/sdlc-workflow/skills/triage-security/version-impact-analysis.md
Add malformed security matrix fixtures to support new evals for format validation, covering wrong columns, missing Forward Pointer, and missing section cases.	Create a fixture with incorrect Ecosystem Mappings column names and order to exercise column mismatch validation behavior. Create a fixture missing the Forward Pointer section while otherwise structurally valid to test auto-repair of the Forward Pointer section. Create a fixture missing the Ecosystem Mappings section to validate critical warning handling and stream halt behavior.	evals/triage-security/files/security-matrix-wrong-columns-mock.md evals/triage-security/files/security-matrix-no-forward-pointer-mock.md evals/triage-security/files/security-matrix-missing-section-mock.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[github-actions]

Eval Results

Eval Results: triage-security

Eval	Passed	Failed	Pass Rate
eval-1	11/11	0	100%
eval-10	5/5	0	100%
eval-11	5/5	0	100%
eval-12	5/5	0	100%
eval-13	5/5	0	100%
eval-14	4/5	1	80%
eval-15	5/5	0	100%
eval-16	7/7	0	100%
eval-17	4/5	1	80%
eval-18	5/5	0	100%
eval-19	5/5	0	100%
eval-2	5/5	0	100%
eval-20	4/4	0	100%
eval-21	4/4	0	100%
eval-22	4/4	0	100%
eval-23	4/4	0	100%
eval-24	4/4	0	100%
eval-25	4/4	0	100%
eval-26	5/5	0	100%
eval-27	5/5	0	100%
eval-28	5/5	0	100%
eval-29	5/5	0	100%
eval-3	5/5	0	100%
eval-30	4/4	0	100%
eval-31	4/4	0	100%
eval-32	4/4	0	100%
eval-4	5/5	0	100%
eval-5	6/6	0	100%
eval-6	6/6	0	100%
eval-7	5/5	0	100%
eval-8	8/8	0	100%
eval-9	5/5	0	100%

Failed Assertions

eval-14: 1 failing assertion

• Assertion: "The SBOM verification result is presented alongside the rpms.lock.yaml result in the dependency chain output, not as a separate section (Step 2.3.5 sub-step 6)"
  Evidence: "The SBOM result IS presented alongside rpms.lock.yaml within the dependency chain code block in section 2.3.5 (lines 38-54 show both rpms.lock.yaml and SBOM results in a single block). However, the output ALSO contains a separate '## SBOM Verification Summary' section (lines 86-94) that restates the SBOM verification results outside the dependency chain output. The spec at lines 425-427 explicitly states: 'Present all SBOM verification results inline within the dependency chain output — do not create a separate output file or section for SBOM results.' The separate '## SBOM Verification Summary' heading constitutes a separate section for SBOM results, directly violating this requirement. Contradictory evidence (inline presentation correct, but separate section also exists) means FAIL per grading rules."

eval-17: 1 failing assertion

• Assertion: "Step 0 extracts the Embargo policy URL from the Security Configuration as an optional field without raising an error (§1.71 — backward compatible extraction)"
  Evidence: "No Step 0 output file exists. The four output files are: data-extraction.md (Step 1), embargo-check.md (Step 1.7), version-impact.md (Step 2), remediation.md (Step 8). The data extraction step (Step 1, data-extraction.md) does not mention the embargo policy URL at all. The embargo policy URL extraction is instead documented in Step 1.7 (embargo-check.md) under a 'Configuration' section: 'Embargo policy URL is configured in Security Configuration: https://example.com/security/embargo-policy'. While the URL is extracted and treated as optional ('Since the Embargo policy URL is present, this step is active (not skipped)'), it is not extracted in Step 0 as the assertion requires — Step 0 does not exist in the outputs, and the logical data extraction step (Step 1) omits this field entirely."

Pass rate: 99% · Tokens: 48,306 · Duration: 107s

Baseline (4d9733a7): 100% · 0 tokens · 0s

---

Generated by sdlc-workflow/run-evals v0.13.0

[ruromero]

Verification Report for TC-5136 (commit fa88508)

Check	Result	Details
Review Feedback	N/A	No review comments on the PR
Root-Cause Investigation	N/A	No sub-tasks created
Scope Containment	WARN	3 eval fixture files not in explicit task spec but implied by Implementation Notes
Diff Size	PASS	393 lines across 5 files — proportionate for validation logic, fixtures, and evals
Commit Traceability	PASS	Single commit references TC-5136 in body
Sensitive Patterns	PASS	No secrets, credentials, or sensitive data detected
CI Status	PASS	All 5 CI checks pass (Eval PR Run, Plugin Validation, Skill Lint, Sourcery, Trigger Eval Dispatch)
Acceptance Criteria	PASS	7 of 7 criteria met
Test Quality	PASS	Eval Quality: PASS (99% pass rate, 160/162; 2 pre-existing failures in eval-14, eval-17 unrelated to this PR; all 4 new evals 29-32 pass 100%)
Test Change Classification	ADDITIVE	3 new fixture files and 4 new eval entries added; no existing evals modified or removed
Verification Commands	N/A	No verification commands specified; no eval infrastructure changes

Overall: WARN

Scope Containment is WARN because 3 new fixture files (security-matrix-missing-section-mock.md, security-matrix-no-forward-pointer-mock.md, security-matrix-wrong-columns-mock.md) are not explicitly listed in the task's Files to Create section. However, these files are directly implied by the task's Implementation Notes ("Eval fixtures: existing mock matrices... can serve as starting points for validation test scenarios") and Test Requirements ("Add eval assertions covering matrix format validation scenarios"). All task-required files are present — no unimplemented files.

---

This comment was AI-generated by sdlc-workflow/verify-pr v0.13.0.

[mrizzi]

@ruromero thanks for adding this template usage