diff --git a/.github/carl/current-pr-contract.md b/.github/carl/current-pr-contract.md index 74babd8..670b33d 100644 --- a/.github/carl/current-pr-contract.md +++ b/.github/carl/current-pr-contract.md @@ -11,71 +11,80 @@ PR constraints are historical evidence unless they are explicitly promoted to durable invariants. ## Goal -Add a safe lightweight test command and CI workflow for SpecQL tests that do not -require CodeQL, `azure-rest-api-specs`, or a built database. +Harden generated artefact boundaries so generated databases, cloned spec corpora, +SARIF/results, exported inventory, and local-only cache/env files cannot be +accidentally committed. ## Contract status active ## Non-goals - Do not refactor product code or change query logic. -- Do not run or add CodeQL database build steps. -- Do not touch `queries/`, generated inventory output, `database/`, `results/`, or `azure-rest-api-specs/`. -- Do not add Python packages to `requirements.txt` unless explicitly required. +- Do not regenerate inventory, database, SARIF, or shard outputs. +- Do not modify workflows or CodeQL queries. +- Do not add Python packages to `requirements.txt`. ## Carry-forward rules The following constraints from this PR are promoted to durable invariants and must persist into all future PRs: - Always pin CodeQL CLI to 2.20.1 or 2.20.2 (JSON-only database compatibility). -- Do not commit `database/`, `results/`, `azure-rest-api-specs/`, or generated inventory JSON files. +- Do not commit `database/`, `results/`, `azure-rest-api-specs/`, or generated inventory exports. - `requirements.txt` lists only runtime-required Python packages; pytest is documented separately. - `carl doctor` must remain healthy after any changes to `.github/carl/` artefacts. ## Approved scope -- Add/update lightweight test command documentation. -- Add a dedicated GitHub Actions workflow for lightweight pytest checks only. -- Update cARL durable docs/memory only if canonical validation or workflow truth changes. +- Update `.gitignore` to ignore generated/heavy/local artefacts and add ownership comments. +- Remove already tracked generated inventory ZIP exports if needed to make the ignore boundary effective. +- Update `README.md` or `CONTRIBUTING.md` only if artefact ownership needs clarification. +- Update `.github/carl/memory.md` only if generated artefact boundaries or canonical operating assumptions change. +- Update this contract to match the approved scope and validation for this PR. ## Intentional amendments -- Supersedes the previous cARL-install-only scope in this file. -- Allows `.github/workflows/` updates for the lightweight-test workflow only. +- Supersedes the previous lightweight-test workflow scope in this file. +- Allows tracked generated inventory ZIP removals as part of repository-boundary hardening. ## Forbidden scope - Modifying product behaviour in `SpeQL.py`, `analyze.py`, `refresh_database.py`, `run-queries.sh`, or export logic. - Modifying CodeQL queries under `queries/`. +- Modifying workflows under `.github/workflows/`. - Modifying source configs under `config/sources/`. -- Adding/removing/changing Python packages in `requirements.txt` without explicit approval. -- Touching generated artefact directories (`database/`, `results/`, `inventory/*.json`, `azure-rest-api-specs/`). +- Adding/removing/changing Python packages in `requirements.txt`. +- Regenerating or editing generated database, SARIF, JSON inventory, or shard contents. ## Architectural constraints - SpeQL architecture and product behaviour must remain identical before and after this PR. -- Lightweight tests must remain independent from CodeQL and database/spec clone prerequisites. -- `requirements.txt` remains runtime-focused; pytest is installed in CI as a test tool. +- `.gitignore` should document which script or pipeline owns each generated artefact area. +- Generated inventory boundaries must cover flat, grouped, sharded, and packaged export outputs. +- `requirements.txt` remains runtime-focused; pytest is installed separately as a test tool. ## Security constraints - No secrets, tokens, or credentials may be committed. -- No new external network dependencies are introduced. -- Workflow permissions must stay least-privilege. +- No new external network dependencies or dependencies in-repo may be introduced. +- Ignore rules must not accidentally hide committed source files outside the intended generated/local artefact boundaries. ## Files expected to change - `.github/carl/current-pr-contract.md` — amended scope for this PR -- `README.md` and/or `CONTRIBUTING.md` — lightweight test entry-point documentation -- `.github/workflows/lightweight-tests.yml` — new CI workflow for lightweight pytest suite -- `.github/carl/memory.md` — workflow inventory/canonical command reconciliation (if needed) +- `.gitignore` — hardened generated/local artefact boundaries +- `inventory/api-index-23389057738.zip` — remove tracked generated export if confirmed in scope +- `inventory/api-index-sharded-23390552485.zip` — remove tracked generated export if confirmed in scope +- `README.md` and/or `CONTRIBUTING.md` — ownership clarification only if needed +- `.github/carl/memory.md` — durable generated-artefact boundary reconciliation (if needed) ## Tests / validation - Install runtime deps: `pip3 install -r requirements.txt` - Install test tool explicitly: `pip3 install pytest` - Run lightweight tests: `python3 -m pytest tests/test_api_inventory_export.py tests/test_api_inventory_normalization.py -v` -- Confirm no generated artefact churn in `git status` +- Confirm `git status` contains only intended `.gitignore`, docs, cARL, and generated ZIP removal changes +- Confirm no generated artefacts are newly added ## Stop conditions - Any change that alters product script or CodeQL query behaviour. -- Any generated database, inventory, or results file appearing in git status. +- Any generated database, SARIF, JSON inventory, or shard file appearing in git status as a new addition. - Any secret detected in changed files. ## Escalation triggers -- If requested workflow scope expands beyond lightweight pytest tests. -- If changes require touching queries, generated artefacts, or dependency policy. +- If hardening requires modifying workflows, product code, or query logic. +- If additional tracked generated artefacts outside the approved scope need deletion. +- If ignore patterns would overlap committed source files or docs unexpectedly. ## Context reset notes -This contract covers adding a lightweight test entry point and CI workflow only. Close/reset after merge. +This contract covers generated-artefact boundary hardening only. Close/reset after merge. diff --git a/.github/carl/memory.md b/.github/carl/memory.md index 2349670..fa0a69d 100644 --- a/.github/carl/memory.md +++ b/.github/carl/memory.md @@ -51,7 +51,7 @@ SpeQL is a Python + CodeQL analysis pipeline targeting Azure REST API JSON specs |---|---|---| | `database/` | Generated; not committed | Built by refresh scripts from azure-rest-api-specs | | `results/` | Generated; gitignored | SARIF output from CodeQL queries | -| `inventory/` | Generated; `.json`/`.min.json` gitignored | Exported API index artefacts | +| `inventory/` | Generated; only `.gitkeep` tracked | Exported API index artefacts (`.json`, `.min.json`, `shards/`, and packaged `.zip` outputs are gitignored) | | `azure-rest-api-specs/` | Cloned externally; gitignored | Source spec corpus managed by refresh scripts | | `queries/azure-security/` | Committed | CodeQL query pack with `qlpack.yml` | | `config/sources/` | Committed | One JSON per spec source (currently `azure.json`) | @@ -71,6 +71,7 @@ Adding a new source requires only a source config JSON and optionally a platform - `api-index-grouped.json` — nested by provider/host/route (with `--grouped`) - `shards/.json` — per-provider shards (with `--sharded`) - Minified variants (`*.min.json`) with `--minified` +- Packaged sharded export ZIPs for release publication Published to UndREST-APISpy via GitHub Releases. @@ -133,7 +134,7 @@ requires a package, document it explicitly rather than silently adding. ## Core invariants -- Do not commit database/, results/, azure-rest-api-specs/, or generated inventory JSON files. +- Do not commit database/, results/, azure-rest-api-specs/, or generated inventory exports (JSON, shards, or packaged ZIPs). - Always pin CodeQL CLI to 2.20.1 or 2.20.2 in setup, docs, and CI. - requirements.txt lists only runtime-required Python packages; test tools (pytest) are documented separately. - Export pipeline output is deterministic given the same spec corpus input. diff --git a/.gitignore b/.gitignore index 44fa13c..c5c51e6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,22 +1,26 @@ -# Results directory +# Generated SARIF and query results +# Owned by: `run-queries.sh`, local CodeQL CLI runs, and `.github/workflows/speql-security-scan.yml` results/ *.sarif -# CodeQL temporary files +# CodeQL temporary files and downloaded local tool copies +# Owned by: local CodeQL CLI setup and query execution .codeql/ *.qls - -# CodeQL libraries (downloaded/extracted) codeql-javascript/ codeql-shared/ codeql/ -# Python cache +# Local Python cache and build artefacts __pycache__/ *.py[cod] *$py.class +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ -# Virtual environments +# Local virtual environments +.venv/ venv/ env/ ENV/ @@ -34,16 +38,22 @@ Thumbs.db # Logs *.log -# Azure REST API specs clone (managed by refresh scripts) +# Cloned upstream API spec corpus +# Owned by: `refresh_database.py`, `refresh-database.sh`, and export/scan workflows azure-rest-api-specs/ -# Database directory (created by refresh scripts, not tracked in git) -# Empty directories can't be committed to Git +# Generated CodeQL databases and build logs +# Owned by: `refresh_database.py`, `refresh-database.sh`, and `.github/workflows/speql-security-scan.yml` database/ -# API inventory export output (generated by scripts/export/) +# Generated API inventory exports, shards, and packaged release inputs +# Owned by: `scripts/export/export_api_inventory.py`, +# `.github/workflows/daily-api-index-export.yml`, and +# `.github/workflows/daily-api-index-export-sharded.yml` inventory/*.json inventory/*.min.json +inventory/shards/ +inventory/*.zip # APISpy portal sweep output files apispy/scripts/apispy-*.csv diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 59e179a..cbc2dac 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,7 +23,7 @@ This repository uses [cARL](https://github.com/goldjg/cARL) for AI coding agent Key rules for agents and contributors: - **CodeQL CLI must be pinned to 2.20.1 or 2.20.2.** 2.23.x+ breaks JSON-only database builds. -- **Do not commit** `database/`, `results/`, `azure-rest-api-specs/`, or generated inventory JSON files. +- **Do not commit** `database/`, `results/`, `azure-rest-api-specs/`, or generated inventory exports (`inventory/*.json`, `inventory/shards/`, packaged inventory ZIPs). - **Lightweight tests** (no CodeQL/DB required): `python3 -m pytest tests/test_api_inventory_export.py tests/test_api_inventory_normalization.py -v` - **requirements.txt** lists only `pyfiglet`. Install `pytest` separately for testing. diff --git a/inventory/api-index-23389057738.zip b/inventory/api-index-23389057738.zip deleted file mode 100644 index 1a58eb1..0000000 Binary files a/inventory/api-index-23389057738.zip and /dev/null differ diff --git a/inventory/api-index-sharded-23390552485.zip b/inventory/api-index-sharded-23390552485.zip deleted file mode 100644 index 7c19071..0000000 Binary files a/inventory/api-index-sharded-23390552485.zip and /dev/null differ