[Infra, PoC] Add static eval via skill-validator#1186
Open
JanKrivanek wants to merge 5 commits intogithub:mainfrom
Open
[Infra, PoC] Add static eval via skill-validator#1186JanKrivanek wants to merge 5 commits intogithub:mainfrom
JanKrivanek wants to merge 5 commits intogithub:mainfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds GitHub Actions workflows to run the dotnet/skills skill-validator for static (non-LLM) evaluation of skills/agents, publishing a nightly quality report and posting PR feedback on changes.
Changes:
- Introduces a nightly scheduled workflow that scans all
skills/andagents/and publishes results to a Discussions category (with Issue fallback). - Adds a pull-request-triggered workflow that runs
skill-validatoron changed skills/agents and posts/updates a PR comment with findings.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| .github/workflows/skill-quality-report.yml | Nightly scan + report generation and publishing to Discussions/Issues. |
| .github/workflows/skill-check.yml | PR-time validation of changed skills/agents with a comment summarizing results. |
Comments suppressed due to low confidence (1)
.github/workflows/skill-quality-report.yml:71
- Same issue as the skills scan: the agents scan pipes into
tee, so$?reflectsteerather thanskill-validator, causingexit_codeto be unreliable. Capture the validator exit status viaPIPESTATUSand/or enablepipefail.
set +e
AGENT_FILES=$(find agents -name '*.agent.md' -type f 2>/dev/null | tr '\n' ' ')
if [ -n "$AGENT_FILES" ]; then
.skill-validator/skill-validator check \
--agents $AGENT_FILES \
--verbose \
2>&1 | tee sv-agents-output.txt
echo "exit_code=$?" >> "$GITHUB_OUTPUT"
else
- Use date-based cache key so skill-validator refreshes daily - Fix exit code capture: use pipefail + PIPESTATUS[0] instead of \True - Add plugin agent file detection (plugins/**/agents/*.agent.md) - Fix CODEOWNERS matching: last-match-wins + wildcard (*) support - Create 'skill-quality' label on-the-fly in issue fallback - Remove broken link to integration plan doc from nightly report
Workflow can only run once it exists on the base branch (staged). Will be testable after merge via a follow-up PR or workflow_dispatch.
4c98da8 to
c58116f
Compare
a9fcf7e to
c789530
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Add static non-LLM evaluation of skills by skill-validator from dotnet/skills
As a:
Prereq
go to https://github.com/github/awesome-copilot/discussions/categories
Click New category
Set:
Click Create