Bind challenge draft validation and publish to the reviewed Git commit

[ifsheldon]

Problem

Challenge drafts record GitHub PR and commit metadata, but the current MVP flow can still assemble validation/publish bundles from creator-provided local inputs. A mutable checkout or mismatched local directory can cause the validated or published bundle to differ from the reviewed PR commit.

For MVP, Agentics can trust a verified GitHub creator identity and surface PR/commit/digest metadata to creators and reviewers. However, before broader hosted usage, the platform should make the reviewed Git commit the authoritative source for public challenge files.

Current MVP position

For the first hosted MVP, the accepted trust boundary is:

• creator identity is verified through GitHub OAuth,
• draft metadata records PR URL, repository, PR number, claimed head SHA, manifest hash, and assembled bundle digest,
• creator and admin UI show those values,
• validation and approval record bundle digests,
• approval freezes the accepted digest,
• publish requires the current assembled digest to match the approved digest.

This helps honest creators and reviewers catch mistakes, but it does not fully prove that the public bundle content came from the reviewed Git commit.

Desired hardening

Make the server materialize challenge bundles from the stored Git identity rather than trusting caller-provided local paths.

A hardened publish flow should:

1. Store repo_url, PR number, and reviewed commit_sha on the draft.
2. Fetch or clone the configured challenge repository server-side.
3. Check out exactly the stored commit SHA.
4. Refuse to continue if the checkout is dirty, missing, or does not contain the expected challenge manifest path.
5. Assemble the public challenge bundle from that checkout.
6. Overlay private assets from managed Agentics storage only.
7. Compute a canonical bundle digest over public files, private asset metadata/content digests, and normalized manifest/spec content.
8. Persist the digest at validation and approval time.
9. Require publish to reassemble the same commit plus frozen private overlay and match the approved digest.
10. Store published challenge artifacts under immutable managed storage keys.

Current solution for MVP

The MVP solution is intentionally lighter:

• verify the creator with GitHub OAuth,
• fetch PR metadata when practical,
• record and display PR/commit/digest metadata,
• freeze approved bundle/private-asset digests,
• reject publish if the current assembled digest differs from the approved digest.

This keeps the challenge creation workflow simple while still creating an audit trail.

Preferred long-term solution

Use server-owned Git materialization.

Implementation sketch:

• Add a managed checkout/cache directory under server storage.
• Fetch the configured challenge repository using GitHub URL metadata from the draft.
• Checkout the exact commit SHA with detached HEAD.
• Use git status --porcelain or equivalent library checks to require a clean tree.
• Build the public bundle from the checked-out challenge directory.
• Apply private asset ZIP overlays from Agentics storage.
• Canonicalize and hash the final assembled bundle.
• Make validation, approval, and publish compare the same canonical digest.

Alternatives and tradeoffs

Trust verified creators plus digest audit trail

Pros:

• simplest MVP path
• low implementation complexity
• keeps creator workflow fast
• gives admins useful PR/commit/digest visibility

Cons:

• does not cryptographically bind published content to the reviewed Git commit
• accidental local checkout drift can still happen before publish
• malicious verified creators could publish content that differs from the public PR if reviewer checks are weak

Server-owned Git checkout at validation/publish

Pros:

• strong binding between reviewed commit and public challenge content
• removes local-path ambiguity
• easier to audit and reproduce published versions
• better foundation for public challenge repositories and CI-like workflows

Cons:

• more backend complexity
• requires Git/GitHub error handling, cache cleanup, and network assumptions
• needs careful handling of submodules/LFS if challenge repo ever uses them

GitHub App or GitHub Actions integration

Pros:

• best fit for a future GitHub-native challenge workflow
• GitHub can provide signed webhook events and PR/commit context
• CI can validate challenge bundles before Agentics publish

Cons:

• higher setup complexity
• requires GitHub App permissions and webhook operations
• more moving parts than needed for MVP

Require creators to upload a signed bundle artifact generated by CI

Pros:

• can bind content to CI output
• reduces server-side Git complexity

Cons:

• shifts complexity to challenge repo CI
• harder for casual creators
• still needs signature/provenance verification

Recommendation

Keep the MVP trust boundary simple: GitHub OAuth creator verification plus visible PR/commit/digest metadata and approval freeze.

Schedule server-owned Git materialization before broader public challenge creation, external creator onboarding, or any workflow where reviewers should not have to trust local creator inputs.

Acceptance criteria

• Validation and publish no longer accept arbitrary local checkout paths as authoritative public challenge content.
• The server assembles public challenge files from the stored repository and exact commit SHA.
• Private assets are read only from managed Agentics storage.
• Approval freezes a canonical bundle digest.
• Publish reassembles and verifies the digest before creating an immutable challenge version.
• Regression tests cover a draft whose local checkout differs from the stored commit and assert publish rejects it.

[ifsheldon]

2026-06 update

Agentics now has a GitHub App configured for human GitHub sign-in. This solves creator/admin identity for the web session and creator API-token flow, but it does not yet solve this issue.

The current GitHub App integration uses only OAuth client credentials:

• AGENTICS_GITHUB_APP_CLIENT_ID
• AGENTICS_GITHUB_APP_CLIENT_SECRET
• AGENTICS_GITHUB_APP_REDIRECT_URL

It does not currently configure a GitHub App private key, webhook secret, installation id, installation tokens, or repository permissions. Therefore Agentics still cannot use the GitHub App to materialize reviewed repository content server-side.

Current review-record validation and publishing still accept an admin-provided repository_path. Publish does reassemble and compare the approved bundle digest, but the authoritative public files still come from a local checkout supplied at validation/publish time, not from the stored Git commit.

Updated direction

For MVP hardening, prefer server-owned Git materialization from the recorded public repository and exact commit SHA. Since agentics-challenges is public, the first implementation does not need GitHub App installation permissions.

Concrete direction:

1. Add a managed challenge-review checkout/cache root.
2. For validation and publish, fetch the stored repo_url.
3. Check out exactly the stored commit_sha in detached HEAD.
4. Reject missing commits, dirty checkouts, unsupported submodules/LFS if not supported, and missing challenge_path.
5. Validate that the manifest hash matches the review-record creation hash.
6. Assemble public files only from that server-owned checkout.
7. Overlay private assets only from Agentics-managed storage.
8. Freeze and compare the canonical bundle digest across validation, approval, and publish.
9. Remove repository_path from the normal admin validation/publish API once this lands.

Future GitHub App installation scope

If we later support private challenge repositories, webhook-driven review-record sync, or GitHub-native PR validation, open a separate issue for GitHub App installation/private-key/webhook support.