Skip to content

[Infra, PoC] Add static eval via skill-validator#1186

Open
JanKrivanek wants to merge 5 commits intogithub:mainfrom
JanKrivanek:dev/jankrivanek/static-eval
Open

[Infra, PoC] Add static eval via skill-validator#1186
JanKrivanek wants to merge 5 commits intogithub:mainfrom
JanKrivanek:dev/jankrivanek/static-eval

Conversation

@JanKrivanek
Copy link
Contributor

@JanKrivanek JanKrivanek commented Mar 26, 2026

Motivation

Add static non-LLM evaluation of skills by skill-validator from dotnet/skills
As a:

  • nonblocking PR gate
  • nightly run creating discussion item with report

Prereq

  1. go to https://github.com/github/awesome-copilot/discussions/categories

  2. Click New category

  3. Set:

    • Name: Skill Quality Reports
    • Description: Automated nightly skills eval reports
    • Format: Announcement (so only the workflow bot can post, others can comment)
    • (+ optional - emoji)
  4. Click Create

Copilot AI review requested due to automatic review settings March 26, 2026 19:54
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds GitHub Actions workflows to run the dotnet/skills skill-validator for static (non-LLM) evaluation of skills/agents, publishing a nightly quality report and posting PR feedback on changes.

Changes:

  • Introduces a nightly scheduled workflow that scans all skills/ and agents/ and publishes results to a Discussions category (with Issue fallback).
  • Adds a pull-request-triggered workflow that runs skill-validator on changed skills/agents and posts/updates a PR comment with findings.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File Description
.github/workflows/skill-quality-report.yml Nightly scan + report generation and publishing to Discussions/Issues.
.github/workflows/skill-check.yml PR-time validation of changed skills/agents with a comment summarizing results.
Comments suppressed due to low confidence (1)

.github/workflows/skill-quality-report.yml:71

  • Same issue as the skills scan: the agents scan pipes into tee, so $? reflects tee rather than skill-validator, causing exit_code to be unreliable. Capture the validator exit status via PIPESTATUS and/or enable pipefail.
          set +e
          AGENT_FILES=$(find agents -name '*.agent.md' -type f 2>/dev/null | tr '\n' ' ')
          if [ -n "$AGENT_FILES" ]; then
            .skill-validator/skill-validator check \
              --agents $AGENT_FILES \
              --verbose \
              2>&1 | tee sv-agents-output.txt
            echo "exit_code=$?" >> "$GITHUB_OUTPUT"
          else

- Use date-based cache key so skill-validator refreshes daily
- Fix exit code capture: use pipefail + PIPESTATUS[0] instead of \True
- Add plugin agent file detection (plugins/**/agents/*.agent.md)
- Fix CODEOWNERS matching: last-match-wins + wildcard (*) support
- Create 'skill-quality' label on-the-fly in issue fallback
- Remove broken link to integration plan doc from nightly report
Workflow can only run once it exists on the base branch (staged).
Will be testable after merge via a follow-up PR or workflow_dispatch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants