Proposal: non-blocking CI report for pages/api/** doc-search responses

[geminimir]

Feature request

Is your feature request related to a problem? Please describe.

When working on PRs that touch the pages/api code in this repo, there’s currently no automated way to confirm that the OpenAI Doc Search API still returns responses in the expected shape (and without regressions) without manually testing it.

This makes it easy for silent regressions to creep in — for example, the API could start omitting a field like sources or returning malformed answers, and this wouldn’t be caught until after deployment.

Describe the solution you'd like

Add a small, report-only CI check (via GitHub Actions) that:

• Runs only on changes to pages/api/** and the new files below
• Replays a recorded doc-search request (no live API calls, no secrets required)
• Validates the response against an explicit schema (e.g., { answer: string, sources: string[] })
• Flags basic safety issues (e.g., PII leakage)
• Uploads a single HTML report to the PR for easy reviewer inspection
• Uses fixed seed + runs=3 to avoid flakiness
• Does not block merges — informational only

Describe alternatives you've considered

• Manual QA after each change — time-consuming, inconsistent, and often skipped.
• Unit tests for specific functions — useful but don’t cover the full end-to-end output shape and safety checks.
• Live API calls in CI — adds cost, requires secrets, and introduces instability from network/model changes.

Additional context

The PR would add three small files:

---

1. .github/workflows/promptproof.yml

name: PromptProof

on:
  pull_request:
    paths:
      - "pages/api/**"
      - ".github/workflows/promptproof.yml"
      - "promptproof.yaml"
      - "fixtures/promptproof/**"

jobs:
  promptproof:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run PromptProof
        uses: promptproof/action@v1
        with:
          config: promptproof.yaml
      - name: Upload HTML report
        uses: actions/upload-artifact@v4
        with:
          name: promptproof-report
          path: promptproof-report.html

---

2. promptproof.yaml

schema:
  type: object
  required: [answer, sources]
  properties:
    answer:
      type: string
    sources:
      type: array
      items:
        type: string
        pattern: "^https?://"

checks:
  - type: pii
  - type: schema
  - type: cost

stability:
  seed: 42
  runs: 3

report:
  format: html
  output: promptproof-report.html

fixtures:
  - fixtures/promptproof/doc-search.json

---

3. fixtures/promptproof/doc-search.json

{
  "request": {
    "query": "How do I set up row-level security in Supabase?"
  },
  "response": {
    "answer": "You can enable Row Level Security (RLS) in Supabase by ...",
    "sources": [
      "https://supabase.com/docs/guides/auth/row-level-security"
    ]
  }
}

---

Benefits for this repo

• Prevents silent regressions in API shape for doc search
• Gives maintainers & contributors a clear, visual HTML report on each PR
• Zero API keys or runtime calls — purely replayed fixtures
• Scoped to API routes only — no impact on unrelated parts of the repo
• Deterministic by default — avoids flakiness in CI

---

If you’re good with this, I can open the PR as-is.