Skip to content

docs: fix metric drift in canonical documentation #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
google-labs-jules wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: fix metric drift in canonical documentation #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ pass. Pure-Python, **stdlib only, zero runtime dependencies**; it shells out to
| Command | Purpose |
|---------|---------|
| `pipx install --editable .` | Install the CLI from source (or `pip install -e .` in a 3.11+ venv) |
| `python3 -m unittest discover -s tests` | Run the suite (285 tests, stdlib only, no network) |
| `python3 -m unittest discover -s tests` | Run the suite (324 tests, stdlib only, no network) |
| `websec run ./target` | Full pipeline → `FACTS.json` + `AGENT-BRIEFING.md` + `probes/` |
| `websec doctor ./target` | Show which optional scanners are installed |
| `websec proof` | Score recon coverage vs the vuln-app corpus (needs network on first clone) |
Expand All @@ -53,8 +53,8 @@ docguard diagnose # guard → emit AI fix prompts
### AI Agent Workflow

1. **Before any work**: read `docs-canonical/` and run `docguard guard` to see the compliance state.
2. **After changing code or docs**: re-run `docguard guard`; keep the numbers (20 extractors, 16 sink
classes, 285 tests, 10/10 proof) consistent across every doc — DocGuard's metrics-consistency
2. **After changing code or docs**: re-run `docguard guard`; keep the numbers (22 extractors, 17 sink
classes, 324 tests, 8/10 proof) consistent across every doc — DocGuard's metrics-consistency
validator cross-checks them.
3. **Update `CHANGELOG.md`** for any user-visible change.
4. **Document drift**: if code must deviate from a canonical doc, add a `// DRIFT: reason` (or
Expand Down
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ Register it in your MCP client:
## Proof harness

`websec proof` clones a vuln-app corpus (VAmPI, NodeGoat, DVGA) and scores whether recon surfaces
each app's documented attack surface — a deterministic, CI-trackable proxy (currently **10/10**).
each app's documented attack surface — a deterministic, CI-trackable proxy (currently **8/10**).
The real kill-criterion (does the briefing lift an agent's bug-finding vs a generic prompt?) is the
manual A/B in [`corpus/PROOF-PROTOCOL.md`](corpus/PROOF-PROTOCOL.md).

Expand Down Expand Up @@ -276,7 +276,7 @@ publisher** with project `websec-validator`, owner `raccioly`, repo `websec-vali

## Status / roadmap

**Done:** 20-extractor recon (incl. a **WebExtension client-trust extractor** — client-side entitlement
**Done:** 22-extractor recon (incl. a **WebExtension client-trust extractor** — client-side entitlement
gate / over-broad host permissions / `world:"MAIN"` / unvalidated external messages — a
**license/entitlement verification-trust** pass — revocation-bypass + no per-license usage cap, provider-
agnostic — **Deno/Supabase-edge + Chrome-extension** stack & route modeling, an **authz-correctness data-flow extractor** — unsigned-cookie /
Expand All @@ -290,7 +290,7 @@ algorithms / predictable principal), **docker-compose host-takeover** + **`.gitl
secret-suppression** audits, and a **reverse-proxy prefix-escape** detector), cross-tool de-dup +
**bundled Semgrep rules**, **router-mount-auth modeling** (cuts the dominant Express-monorepo
missing-auth false positive), tailored probe staging, agent briefing, traceable findings ledger with
**calibrated confidence (CJE — Wilson CIs)**, proof harness, test suite (285), **Docker bundle** (all
**calibrated confidence (CJE — Wilson CIs)**, proof harness, test suite (324), **Docker bundle** (all
scanners + Noir, arch-aware), **dynamic phase v1** (authenticated read-only cross-tenant BOLA —
validated live, reproduced a hand-pentest's 14/14). Validated against the **REF-PENTEST pen test +
retest** and re-validated on a large real-world LLM-agent monorepo (HIGH-finding noise 178 → 15, AI +
Expand Down
2 changes: 1 addition & 1 deletion docs-canonical/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ scanner runs, a calibrated findings ledger, staged probes, and the briefing/repo
|-----------|---------------|----------|-------|
| CLI entry point | Arg parsing + the `run` (with `--format`/`--fail-on`/`--baseline`) / `doctor` / `dynamic` / `mcp` commands (and hidden `recon` / `proof` / `calibrate`) | `src/websec_validator/cli.py` | `tests/test_recon.py`, `tests/test_hardening.py` |
| Recon driver | Thin wrapper that runs the extractor registry over one repo walk | `src/websec_validator/recon.py` | `tests/test_recon.py` |
| Extractors (20) | One focused question each → the merged `FACTS.json` (stack, routes, auth, authz, **authz_dataflow**, tenant, password_policy, surface, schemas, iac_ci, client_exposure, client_integrity, transport_security, graphql, upload_security, pii_exposure, integrations, **llm_security**, **crypto_usage**, **webext**) | `src/websec_validator/extractors/` | `tests/test_recon.py`, `tests/test_pentest_regressions.py`, `tests/test_entitlement_webext.py` |
| Extractors (22) | One focused question each → the merged `FACTS.json` (stack, routes, auth, authz, **authz_dataflow**, tenant, password_policy, surface, schemas, iac_ci, client_exposure, client_integrity, transport_security, graphql, upload_security, pii_exposure, integrations, **llm_security**, **crypto_usage**, **webext**, **agent_config**, **dependencies**) | `src/websec_validator/extractors/` | `tests/test_recon.py`, `tests/test_pentest_regressions.py`, `tests/test_entitlement_webext.py` |
| Static scanners | Detect + (with `--scan`) shell out to Trivy/Gitleaks/Semgrep/Checkov/Prowler and de-duplicate across tools | `src/websec_validator/scanners.py` | `tests/test_recon.py` |
| Findings ledger | Correlate recon + static + dynamic into one ranked, standards-cited, calibrated record set | `src/websec_validator/findings.py` | `tests/test_pentest_regressions.py` |
| Calibration (CJE) | Wilson-interval `P(real)` per `(attack-class, confidence)` bucket; self-improving local overlay | `src/websec_validator/calibration.py` | `tests/test_recon.py` |
Expand Down
2 changes: 1 addition & 1 deletion docs-canonical/TEST-SPEC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ validation via `.docguardignore`.
`websec proof` clones the vuln-app corpus (VAmPI, NodeGoat, DVGA) and scores whether recon surfaces
each app's documented attack surface — a deterministic, CI-trackable proxy. The corpus contributes
coverage checks across three apps — VAmPI (4), NodeGoat (4), DVGA (2), **10 in total** — which the
engine currently passes **10/10**. The true
engine currently passes **8/10**. The true
kill-criterion — does the briefing lift an agent's bug-finding vs a generic prompt? — is the manual
A/B in [`corpus/PROOF-PROTOCOL.md`](../corpus/PROOF-PROTOCOL.md).

Expand Down