diff --git a/AGENTS.md b/AGENTS.md index 310f91b..8d6b8d5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ pass. Pure-Python, **stdlib only, zero runtime dependencies**; it shells out to | Command | Purpose | |---------|---------| | `pipx install --editable .` | Install the CLI from source (or `pip install -e .` in a 3.11+ venv) | -| `python3 -m unittest discover -s tests` | Run the suite (285 tests, stdlib only, no network) | +| `python3 -m unittest discover -s tests` | Run the suite (324 tests, stdlib only, no network) | | `websec run ./target` | Full pipeline → `FACTS.json` + `AGENT-BRIEFING.md` + `probes/` | | `websec doctor ./target` | Show which optional scanners are installed | | `websec proof` | Score recon coverage vs the vuln-app corpus (needs network on first clone) | @@ -53,8 +53,8 @@ docguard diagnose # guard → emit AI fix prompts ### AI Agent Workflow 1. **Before any work**: read `docs-canonical/` and run `docguard guard` to see the compliance state. -2. **After changing code or docs**: re-run `docguard guard`; keep the numbers (20 extractors, 16 sink - classes, 285 tests, 10/10 proof) consistent across every doc — DocGuard's metrics-consistency +2. **After changing code or docs**: re-run `docguard guard`; keep the numbers (22 extractors, 17 sink + classes, 324 tests, 8/10 proof) consistent across every doc — DocGuard's metrics-consistency validator cross-checks them. 3. **Update `CHANGELOG.md`** for any user-visible change. 4. **Document drift**: if code must deviate from a canonical doc, add a `// DRIFT: reason` (or diff --git a/README.md b/README.md index 9c0a343..8958d77 100644 --- a/README.md +++ b/README.md @@ -200,7 +200,7 @@ Register it in your MCP client: ## Proof harness `websec proof` clones a vuln-app corpus (VAmPI, NodeGoat, DVGA) and scores whether recon surfaces -each app's documented attack surface — a deterministic, CI-trackable proxy (currently **10/10**). +each app's documented attack surface — a deterministic, CI-trackable proxy (currently **8/10**). The real kill-criterion (does the briefing lift an agent's bug-finding vs a generic prompt?) is the manual A/B in [`corpus/PROOF-PROTOCOL.md`](corpus/PROOF-PROTOCOL.md). @@ -276,7 +276,7 @@ publisher** with project `websec-validator`, owner `raccioly`, repo `websec-vali ## Status / roadmap -**Done:** 20-extractor recon (incl. a **WebExtension client-trust extractor** — client-side entitlement +**Done:** 22-extractor recon (incl. a **WebExtension client-trust extractor** — client-side entitlement gate / over-broad host permissions / `world:"MAIN"` / unvalidated external messages — a **license/entitlement verification-trust** pass — revocation-bypass + no per-license usage cap, provider- agnostic — **Deno/Supabase-edge + Chrome-extension** stack & route modeling, an **authz-correctness data-flow extractor** — unsigned-cookie / @@ -290,7 +290,7 @@ algorithms / predictable principal), **docker-compose host-takeover** + **`.gitl secret-suppression** audits, and a **reverse-proxy prefix-escape** detector), cross-tool de-dup + **bundled Semgrep rules**, **router-mount-auth modeling** (cuts the dominant Express-monorepo missing-auth false positive), tailored probe staging, agent briefing, traceable findings ledger with -**calibrated confidence (CJE — Wilson CIs)**, proof harness, test suite (285), **Docker bundle** (all +**calibrated confidence (CJE — Wilson CIs)**, proof harness, test suite (324), **Docker bundle** (all scanners + Noir, arch-aware), **dynamic phase v1** (authenticated read-only cross-tenant BOLA — validated live, reproduced a hand-pentest's 14/14). Validated against the **REF-PENTEST pen test + retest** and re-validated on a large real-world LLM-agent monorepo (HIGH-finding noise 178 → 15, AI + diff --git a/docs-canonical/ARCHITECTURE.md b/docs-canonical/ARCHITECTURE.md index b89d166..ed7b09a 100644 --- a/docs-canonical/ARCHITECTURE.md +++ b/docs-canonical/ARCHITECTURE.md @@ -38,7 +38,7 @@ scanner runs, a calibrated findings ledger, staged probes, and the briefing/repo |-----------|---------------|----------|-------| | CLI entry point | Arg parsing + the `run` (with `--format`/`--fail-on`/`--baseline`) / `doctor` / `dynamic` / `mcp` commands (and hidden `recon` / `proof` / `calibrate`) | `src/websec_validator/cli.py` | `tests/test_recon.py`, `tests/test_hardening.py` | | Recon driver | Thin wrapper that runs the extractor registry over one repo walk | `src/websec_validator/recon.py` | `tests/test_recon.py` | -| Extractors (20) | One focused question each → the merged `FACTS.json` (stack, routes, auth, authz, **authz_dataflow**, tenant, password_policy, surface, schemas, iac_ci, client_exposure, client_integrity, transport_security, graphql, upload_security, pii_exposure, integrations, **llm_security**, **crypto_usage**, **webext**) | `src/websec_validator/extractors/` | `tests/test_recon.py`, `tests/test_pentest_regressions.py`, `tests/test_entitlement_webext.py` | +| Extractors (22) | One focused question each → the merged `FACTS.json` (stack, routes, auth, authz, **authz_dataflow**, tenant, password_policy, surface, schemas, iac_ci, client_exposure, client_integrity, transport_security, graphql, upload_security, pii_exposure, integrations, **llm_security**, **crypto_usage**, **webext**, **agent_config**, **dependencies**) | `src/websec_validator/extractors/` | `tests/test_recon.py`, `tests/test_pentest_regressions.py`, `tests/test_entitlement_webext.py` | | Static scanners | Detect + (with `--scan`) shell out to Trivy/Gitleaks/Semgrep/Checkov/Prowler and de-duplicate across tools | `src/websec_validator/scanners.py` | `tests/test_recon.py` | | Findings ledger | Correlate recon + static + dynamic into one ranked, standards-cited, calibrated record set | `src/websec_validator/findings.py` | `tests/test_pentest_regressions.py` | | Calibration (CJE) | Wilson-interval `P(real)` per `(attack-class, confidence)` bucket; self-improving local overlay | `src/websec_validator/calibration.py` | `tests/test_recon.py` | diff --git a/docs-canonical/TEST-SPEC.md b/docs-canonical/TEST-SPEC.md index 2a7ae37..a590a1b 100644 --- a/docs-canonical/TEST-SPEC.md +++ b/docs-canonical/TEST-SPEC.md @@ -65,7 +65,7 @@ validation via `.docguardignore`. `websec proof` clones the vuln-app corpus (VAmPI, NodeGoat, DVGA) and scores whether recon surfaces each app's documented attack surface — a deterministic, CI-trackable proxy. The corpus contributes coverage checks across three apps — VAmPI (4), NodeGoat (4), DVGA (2), **10 in total** — which the -engine currently passes **10/10**. The true +engine currently passes **8/10**. The true kill-criterion — does the briefing lift an agent's bug-finding vs a generic prompt? — is the manual A/B in [`corpus/PROOF-PROTOCOL.md`](../corpus/PROOF-PROTOCOL.md).