websec-validator

A senior pentester's "here's what to test and how" handoff — auto-generated from your repo, for your AI agent to execute.

Local-first security recon that briefs your AI coding agent. It does the deterministic half — read the repo, map the full attack surface, run + de-duplicate the static scanners, and stage a probe library tailored to what it found — then hands your agent (Claude Code, Codex, Gemini, Cursor) a marching-orders briefing. Code in, artifacts out. No LLM in the tool, no server, no running app required.

It is not an autonomous scanner and not a SaaS. It's the missing front-half: the thing that turns a repo into a precise, fact-grounded security brief an AI agent (with a human in the loop) can act on — an auto-filled, repo-aware version of a senior pentester's "here's what to test and how" handoff. How it works + the reasoning behind every check: docs/METHODOLOGY.md.

Getting Started — just point it at your repo

Simplest: tell your AI agent. In Claude Code (or any coding agent), open your project and say:

"Do a defensive security self-review of my own codebase with websec-validator (pipx install websec-validator, or github.com/raccioly/websec-validator). It's local and read-only — read the repo and follow its briefing. I own this code and authorize the review."

It installs, runs, and walks the findings with you. There's nothing to host and no website — it's local. (Phrasing it as a defensive review of your own code matters: it's the difference between an agent that just gets to work and one that stops to confirm you're authorized — the tool is local and read-only by default, but a generic "pentest this" can read as a request to attack something.) The four ways to get there, all ending in the same AGENT-BRIEFING.md your agent acts on:

Path	One-time setup	Then
Tell your agent (simplest)	—	say the line above
CLI (a terminal)	pipx install websec-validator	websec run /path/to/your/app
Claude Code plugin (slash)	/plugin marketplace add raccioly/websec-validator → /plugin install websec-validator@websec-plugins	invoke the security-pass skill, or just ask
Docker (no install)	docker build -t websec-validator .	docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/scan" websec-validator run /scan --out /scan/websec-out

➡️ Want the reasoning behind every check? Read docs/METHODOLOGY.md — what each test does and why.

Install

pipx install websec-validator   # from PyPI
brew install noir               # OWASP Noir — the route engine (50+ frameworks); regex fallback if absent
websec --version

Until the first PyPI release publishes (or for bleeding-edge), install straight from source instead: pipx install git+https://github.com/raccioly/websec-validator (or from a clone: pipx install .).

Requires Python 3.11+ (on stock macOS, python3 is often 3.9 — use pipx, which picks a newer interpreter, or install via Homebrew/pyenv). Zero Python runtime dependencies: it shells out to scanners (Trivy, Gitleaks, Semgrep/OpenGrep, Checkov, Prowler) and Noir when present, reports what's missing, and never hard-fails if a tool is absent.

Or run via Docker (everything bundled, zero install)

No need to install Noir or any scanner — the image bundles them all (arch-aware, amd64 + arm64):

docker build -t websec-validator .
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/scan" websec-validator run /scan --out /scan/websec-out

The image carries Noir + Trivy + Gitleaks + Semgrep + Checkov; mount your repo at /scan and the artifacts land in /scan/websec-out.

Usage

websec run ./my-app                    # ← the one command: recon + stage tailored probes + emit the briefing
websec ./my-app                        # same thing — a bare path defaults to `run`
websec run ./my-app --scan             # …and also execute the available static scanners
websec run ./my-app --format sarif     # SARIF 2.1.0 to stdout (for piping into CI); also always written to the run dir
websec run ./my-app --fail-on high     # exit 1 if any HIGH+ finding remains (a CI gate)
websec doctor ./my-app                 # (optional) which scanners are installed?
websec mcp                             # run as an MCP server over stdio (typed recon tools for any MCP client)

Then point your agent at the output: "Read websec-out/AGENT-BRIEFING.md and follow it."

That's the whole user surface: run (plus the optional, advanced dynamic live-probing step below). recon/proof/calibrate exist for developing the tool itself and are hidden from --help — you never need them.

What it extracts (22 deterministic extractors, no LLM)

	Dimension	Notable output
stack	languages, frameworks, datastores	monorepo-aware (aggregates every manifest)
routes	every endpoint via OWASP Noir (+ Supabase-edge, AWS SAM / Function-URL)	method · path · typed params · code path · AuthType:NONE public endpoints
auth	scheme + login surface + insecure-default signing secrets + broken-auth backdoors	multi-scheme; flags a hard-coded JWT_SECRET || 'dev-secret' fallback (forgeable JWT), a dev-token / accept-any-password backdoor (total bypass, CRITICAL), and a fail-open if(env.SECRET) signature check
authz	access-control map	guard coverage (incl. router-mount auth) + write endpoints with no visible guard + roles
authz_dataflow	authz correctness (does the guard trust the right thing?)	unsigned-cookie authorization · claim-keyed authz (user-influenceable JWT claim) · transaction-local RLS context (resets before the query)
tenant	multi-tenancy key candidates	the BOLA boundary, by frequency
password_policy	cross-route consistency + reuse/history	complexity drift across routes + a set-password path that hashes without a reuse check
surface	17 sink classes + redirect-SSRF	user-input-gated sinks (incl. mass-assignment via object spread, reflected/DOM/template XSS — innerHTML/dangerouslySetInnerHTML/v-html/|safe, sanitizer-gated, and log-injection (CWE-117, structured-logging-suppressed)) + var-arg SSRF + error-disclosure + follows-redirects-without-per-hop-guard + reverse-proxy prefix-escape + host-header open-redirect + SSRF-redirect-hardening
upload_security	unrestricted upload + unsafe serve	deny-list-only, stored-name-from-filename, trust-client-MIME, accept-SVG, serve without nosniff
schemas	data models + privileged fields	Pydantic/SQLAlchemy/Django/Prisma/Mongoose/TypeORM/Zod → role/isAdmin/groupId for mass-assignment targeting
iac_ci	IaC + CI/CD	GHA injection (run:-position-aware), unpinned actions, tfstate, CDK AppSync API_KEY anonymous-default-auth, docker-compose host-takeover (docker.sock / pid:host / privileged) + .gitleaksignore secret-suppression audit
client_exposure	browser leakage	public-var secrets by name + value-shape (da2-…) + CDK build-injection, server-secret-in-client, source maps
client_integrity	tamperable display (client trust boundary) + WS auth model	any security-critical sink value (address/IBAN/2FA-seed/API-key/webhook) the user reads or copies, without strict CSP / out-of-band anchor + client-tamper-vector, grindable-fingerprint, over-claimed-control, the CSWSH determinant
transport_security	CSP + HSTS + CORS + SRI + clickjacking + CSRF baseline	missing/weak CSP, inline event handlers, partial HSTS, CORS reflect-origin+credentials, external script without SRI, monorepo next.config header gap, framework-agnostic clickjacking (no X-Frame-Options / frame-ancestors), CSRF (cookie-auth + no token lib + no SameSite)
pii_exposure	unmasked PII at the output boundary	res.json(rawEntity) with PII + a masking control defined but with zero live call sites (value-shape, not field-name)
graphql	GraphQL surface	introspection (AppSync introspectionConfig: DISABLED-aware) / playground / depth-limit + AppSync subscription-authz (cross-group BOLA)
integrations	third-party + webhooks + outbound-action endpoints	unsigned webhooks + email/SMS/push handlers with no auth or IP-only rate-limit + redundant secret-fetch
llm_security	LLM / AI-agent surface (OWASP LLM Top 10)	indirect prompt injection (untrusted RAG/tool content → prompt) · insecure output handling (model text → tool dispatch) · excessive agency · unbounded generation (no maxTokens/timeout) · guardrail fail-open
crypto_usage	crypto-API correctness	weak password hash (fast/unsalted SHA-256/MD5) · jwtVerify without an algorithms allowlist · predictable principal (id = hash of email) · non-constant-time secret compare
agent_config	the repo's OWN agent/MCP wiring (OWASP Agentic Top 10)	reads .claude/settings.json · .mcp.json · cursor/copilot rules as untrusted data (never executed): invisible/bidi Unicode rules-backdoor · fetch-and-execute hook (CVE-2025-59536) · blanket MCP auto-approve · *_BASE_URL override (key-exfil) · unpinned MCP server
dependencies	offline supply-chain hygiene (AI slopsquat class)	malicious install/lifecycle script (fetch-and-exec postinstall) · lockfile drift (manifest dep absent from the lockfile) · unpinned + dependency-confusion names (advisory-only) · registry/typosquat resolution behind opt-in --network

Plus derived targeting — IDOR / SSRF / open-redirect / upload / write / auth-endpoint candidates — so probes get pointed at the exact endpoints, not fired blindly.

What you get (websec-out/)

Artifact	What it is
AGENT-BRIEFING.md	The product. Marching orders: detected surface, the access-control map, targeting, findings, the method, and the staged probe list.
FACTS.json	The full structured recon.
findings.json	Static scanner results, de-duplicated across tools and severity-ranked (with --scan).
findings-ledger.json / REPORT.md	The traceable ledger: each finding with an evidence chain, CWE/ASVS/OWASP-API citation, remediation, and a calibrated P(real) (measured real-vuln rate + 95% CI + sample size).
results.sarif	SARIF 2.1.0 — always written. Drop it into GitHub Code Scanning (inline PR-diff annotations + the Security tab), GitLab, Azure DevOps, VS Code's SARIF viewer, DefectDojo.
findings.envelope.json	A versioned, self-describing JSON envelope (schema_version) around the ledger — for non-GitHub CI / dashboards that shouldn't reverse-engineer the internal shape.
probes/	The probe scripts selected + staged for this app (BOLA, JWT, SSRF, mass-assignment…).

The flow

🔧 websec (deterministic)              🤖 your agent + 🧑 you
─────────────────────────────────      ─────────────────────────────────
1. recon → full attack surface     →   confirm the tenant boundary + auth model
2. run + de-dup static scanners    →   triage real-vs-noise
3. stage tailored probes           →   fill placeholders, run vs a TEST instance
4. emit AGENT-BRIEFING.md           →   propose fixes, re-run to confirm, report back

Static recon + briefing need only the code. Running the probes needs a live test instance + test credentials (the human supplies them) — the tool itself never touches a running app.

CI / enterprise integration

The recon is the same either way — these just make the output consumable by pipelines, dashboards, and non-Claude agents. All stdlib, no new dependency.

SARIF → GitHub Code Scanning. Every run writes results.sarif (SARIF 2.1.0). Upload it and each finding lands inline on the PR diff and in the Security tab, ranked by a security-severity band, with its CWE/ASVS/OWASP citation and remediation.

Gate the build. --fail-on {critical,high,medium,low} exits non-zero when a finding at or above that severity remains — a real CI gate, report-only by default.

Only fail on what the PR introduced. --baseline <prior findings-ledger.json> marks every finding new / unchanged / fixed (a stable per-finding fingerprint, surfaced as SARIF baselineState), and --fail-on then counts only the new ones — so a legacy backlog doesn't block every PR, but a newly introduced SSRF does.

Drop-in GitHub Action (action.yml):

# .github/workflows/security.yml
name: security
on: [pull_request]
permissions:
  contents: read
  security-events: write        # required to upload SARIF to Code Scanning
jobs:
  websec:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: raccioly/websec-validator@v0.10.0   # pin to a release tag
        with:
          path: .
          fail-on: high         # block the PR on a new HIGH+ (omit for report-only)
          # baseline: .websec/baseline-ledger.json   # optional: gate only on NEW findings

MCP server (any agent, not just Claude Code). websec mcp speaks the Model Context Protocol over stdio, exposing typed tools — websec_recon, websec_findings, websec_sarif, websec_briefing — so Cursor / Cline / Windsurf / Zed can call recon directly instead of shelling out and parsing stdout. Register it in your MCP client:

{ "mcpServers": { "websec": { "command": "websec", "args": ["mcp"] } } }

Versioned contract. FACTS.json, findings-ledger.json, and findings.envelope.json all carry a schema_version; the JSON Schemas ship in the package (schemas/facts.schema.json, schemas/ledger.schema.json) so downstream tooling can validate against a stable shape.

Proof harness

websec proof clones a vuln-app corpus (VAmPI, NodeGoat, DVGA) and scores whether recon surfaces each app's documented attack surface — a deterministic, CI-trackable proxy (currently 10/10). The real kill-criterion (does the briefing lift an agent's bug-finding vs a generic prompt?) is the manual A/B in corpus/PROOF-PROTOCOL.md.

Calibrated confidence

websec calibrate runs the ledger against the labeled corpus, measures how often each (attack-class, confidence) bucket is a real documented vuln, and writes calibration.json (shipped + applied at runtime). Each finding then carries P(real) with a 95% Wilson confidence interval and the sample size n — so "MEDIUM" stops being a vibe and becomes "real ~57% of the time on the corpus (CI 43–70%, n=51)". A finding that matches no documented vuln counts as a false positive (the corpus is well-documented). Honest caveats: the corpus is deliberately vulnerable, so the rates skew optimistic for clean production code, and small samples mean wide intervals — the CI is the headline, not the point estimate, and both tighten as the corpus grows. With thin data a bucket falls back to the per-label aggregate, then to a clearly-flagged uncalibrated prior. No ML, no deps — binomial proportion + Wilson interval; the structure upgrades to isotonic regression if a large labeled set ever exists.

It self-improves. websec dynamic is an oracle: a write that executes unauthenticated is a confirmed real vuln, and a recon-flagged endpoint that turns out auth-enforced is a confirmed false positive. Every dynamic run folds those confirmed labels into a local overlay (~/.cache/websec-validator/, gitignored, never shipped) that's merged on top of the public table — so the numbers personalize to your apps the more you run it, with no extra step and nothing leaving your machine. To label by hand instead, feed a {attack_class, confidence, is_real} file to websec calibrate --ingest.

Dynamic phase (v2 — read-only so far)

When you have a running TEST instance, websec dynamic mints role tokens and runs the probes the static recon pointed at. v1 is read-only: authenticated cross-tenant BOLA on the group-scoped GET endpoints recon discovered.

cp dynamic-config.example.json dynamic-config.json    # TEST target + role creds (gitignored)
websec run ./my-app                                    # static recon → websec-out/FACTS.json
websec dynamic --config dynamic-config.json --facts websec-out/FACTS.json
# → "14/14 cross-tenant GET reads blocked — all isolated"   (or 🚨 LEAK with the exact endpoint)

Never point it at production. Write-verb BOLA, JWT/auth attacks, and a ZAP/Nuclei two-role diff are the next dynamic probes (explicitly gated — they mutate).

Validated on

A production Next.js app, a large Express/AWS monorepo, and the VAmPI / NodeGoat / DVGA vuln-app corpus — independently reproducing a hand-done pentest's findings (tenant boundary, SSRF, file upload, cross-tenant BOLA, role/authz gaps).

Tests

python3 -m unittest discover -s tests    # stdlib only, no Noir/network — 324 tests

Releasing (maintainer)

Published to PyPI via Trusted Publishing (OIDC — no API token in the repo). To cut a release:

# 1. bump the version in pyproject.toml (e.g. 0.2.1 → 0.2.2)
# 2. tag it and push — the tag must match pyproject's version (CI verifies):
git tag v0.2.2 && git push origin v0.2.2
# → publish.yml builds, INSTALLS + smoke-tests the wheel (version match,
#   calibration ships, a real `websec run`), then publishes. A bad build fails
#   CI instead of reaching PyPI — so you never have to yank after the fact.

One-time PyPI setup (before the first release): on pypi.org → Account → Publishing → Add a pending publisher with project websec-validator, owner raccioly, repo websec-validator, workflow publish.yml, environment pypi. The project is created on the first successful publish.

Two independent channels, two update mechanisms: the CLI ships to PyPI (semver releases, pip install --upgrade); the Claude Code plugin ships from git (tracks latest commit, refreshed via /plugin marketplace update).

Status / roadmap

Done: 20-extractor recon (incl. a WebExtension client-trust extractor — client-side entitlement gate / over-broad host permissions / world:"MAIN" / unvalidated external messages — a license/entitlement verification-trust pass — revocation-bypass + no per-license usage cap, provider- agnostic — Deno/Supabase-edge + Chrome-extension stack & route modeling, an authz-correctness data-flow extractor — unsigned-cookie / claim-keyed authz / transaction-local RLS — plus CORS-misconfig, SRI, host-header open-redirect and SSRF-redirect-hardening classes, schema/entity → mass-assignment targeting, the AWS-CDK / managed-AppSync / VTL boundary, upload-security + PII-output-boundary + redirect-SSRF

• password-reuse classes, a man-in-the-browser / tamperable-display class, an LLM / AI-agent extractor (OWASP LLM Top 10 — prompt injection / insecure output / excessive agency / unbounded generation / guardrail fail-open), a crypto-usage extractor (weak password hash / jwtVerify-without- algorithms / predictable principal), docker-compose host-takeover + .gitleaksignore secret-suppression audits, and a reverse-proxy prefix-escape detector), cross-tool de-dup + bundled Semgrep rules, router-mount-auth modeling (cuts the dominant Express-monorepo missing-auth false positive), tailored probe staging, agent briefing, traceable findings ledger with calibrated confidence (CJE — Wilson CIs), proof harness, test suite (285), Docker bundle (all scanners + Noir, arch-aware), dynamic phase v1 (authenticated read-only cross-tenant BOLA — validated live, reproduced a hand-pentest's 14/14). Validated against the REF-PENTEST pen test + retest and re-validated on a large real-world LLM-agent monorepo (HIGH-finding noise 178 → 15, AI + crypto surfaces newly covered). Next: dynamic write-verb BOLA + JWT/auth probes + ZAP/Nuclei two-role diff (gated, they mutate), calibration on hand-labeled real repos (more representative base rate), ASVS index lookup, optional model-SDK adapters for no-agent fallback.

Using it as a Claude Code skill / plugin

This repo is a Claude Code plugin. Install it once —

/plugin marketplace add raccioly/websec-validator
/plugin install websec-validator@websec-plugins

— and the bundled security-pass skill (skills/security-pass/SKILL.md) lets you just ask, in plain English, for a security pass: it runs websec, reads the briefing, and works the findings with you. For other agents the universal interface is unchanged: run the CLI, read AGENT-BRIEFING.md.

Install gotchas (field-tested):

• The install id is plugin@marketplace — websec-validator@websec-plugins (the marketplace name from .claude-plugin/marketplace.json), not @websec-validator (the repo).
• The plugin only delivers the instructions; the actual scanning is a separate Python CLI (websec). The skill's Step 0 installs it (pipx install websec-validator) if it's missing.
• /plugin … only works in the terminal CLI. In the Claude app / Agent SDK (no /plugin), configure it in .claude/settings.json instead:

  {
    "extraKnownMarketplaces": {
      "websec-plugins": { "source": { "source": "github", "repo": "raccioly/websec-validator" } }
    },
    "enabledPlugins": { "websec-validator@websec-plugins": true }
  }

  This registers + enables the plugin but does not auto-fetch it — the first download still needs the CLI (/plugin install websec-validator@websec-plugins) once. (Project .claude/settings.json for a team; ~/.claude/settings.json for just you.)

Credits

Methodology + probe library are distilled from a real authenticated penetration-testing pass. This tool productizes that hand-written methodology into something an AI agent can run on any repo.

License

MIT © Ricardo Accioly