Harden generated artefact boundaries and stop tracking exported inventory packages

.github/carl/current-pr-contract.md

@@ -11,71 +11,80 @@ PR constraints are historical evidence unless they are explicitly
 promoted to durable invariants.
 
 ## Goal
-Add a safe lightweight test command and CI workflow for SpecQL tests that do not
-require CodeQL, `azure-rest-api-specs`, or a built database.
+Harden generated artefact boundaries so generated databases, cloned spec corpora,
+SARIF/results, exported inventory, and local-only cache/env files cannot be
+accidentally committed.
 
 ## Contract status
 active
 
 ## Non-goals
 - Do not refactor product code or change query logic.
-- Do not run or add CodeQL database build steps.
-- Do not touch `queries/`, generated inventory output, `database/`, `results/`, or `azure-rest-api-specs/`.
-- Do not add Python packages to `requirements.txt` unless explicitly required.
+- Do not regenerate inventory, database, SARIF, or shard outputs.
+- Do not modify workflows or CodeQL queries.
+- Do not add Python packages to `requirements.txt`.
 
 ## Carry-forward rules
 The following constraints from this PR are promoted to durable invariants and must persist into all future PRs:
 - Always pin CodeQL CLI to 2.20.1 or 2.20.2 (JSON-only database compatibility).
-- Do not commit `database/`, `results/`, `azure-rest-api-specs/`, or generated inventory JSON files.
+- Do not commit `database/`, `results/`, `azure-rest-api-specs/`, or generated inventory exports.
 - `requirements.txt` lists only runtime-required Python packages; pytest is documented separately.
 - `carl doctor` must remain healthy after any changes to `.github/carl/` artefacts.
 
 ## Approved scope
-- Add/update lightweight test command documentation.
-- Add a dedicated GitHub Actions workflow for lightweight pytest checks only.
-- Update cARL durable docs/memory only if canonical validation or workflow truth changes.
+- Update `.gitignore` to ignore generated/heavy/local artefacts and add ownership comments.
+- Remove already tracked generated inventory ZIP exports if needed to make the ignore boundary effective.
+- Update `README.md` or `CONTRIBUTING.md` only if artefact ownership needs clarification.
+- Update `.github/carl/memory.md` only if generated artefact boundaries or canonical operating assumptions change.
+- Update this contract to match the approved scope and validation for this PR.
 
 ## Intentional amendments
-- Supersedes the previous cARL-install-only scope in this file.
-- Allows `.github/workflows/` updates for the lightweight-test workflow only.
+- Supersedes the previous lightweight-test workflow scope in this file.
+- Allows tracked generated inventory ZIP removals as part of repository-boundary hardening.
 
 ## Forbidden scope
 - Modifying product behaviour in `SpeQL.py`, `analyze.py`, `refresh_database.py`, `run-queries.sh`, or export logic.
 - Modifying CodeQL queries under `queries/`.
+- Modifying workflows under `.github/workflows/`.
 - Modifying source configs under `config/sources/`.
-- Adding/removing/changing Python packages in `requirements.txt` without explicit approval.
-- Touching generated artefact directories (`database/`, `results/`, `inventory/*.json`, `azure-rest-api-specs/`).
+- Adding/removing/changing Python packages in `requirements.txt`.
+- Regenerating or editing generated database, SARIF, JSON inventory, or shard contents.
 
 ## Architectural constraints
 - SpeQL architecture and product behaviour must remain identical before and after this PR.
-- Lightweight tests must remain independent from CodeQL and database/spec clone prerequisites.
-- `requirements.txt` remains runtime-focused; pytest is installed in CI as a test tool.
+- `.gitignore` should document which script or pipeline owns each generated artefact area.
+- Generated inventory boundaries must cover flat, grouped, sharded, and packaged export outputs.
+- `requirements.txt` remains runtime-focused; pytest is installed separately as a test tool.
 
 ## Security constraints
 - No secrets, tokens, or credentials may be committed.
-- No new external network dependencies are introduced.
-- Workflow permissions must stay least-privilege.
+- No new external network dependencies or dependencies in-repo may be introduced.
+- Ignore rules must not accidentally hide committed source files outside the intended generated/local artefact boundaries.
 
 ## Files expected to change
 - `.github/carl/current-pr-contract.md` — amended scope for this PR
-- `README.md` and/or `CONTRIBUTING.md` — lightweight test entry-point documentation
-- `.github/workflows/lightweight-tests.yml` — new CI workflow for lightweight pytest suite
-- `.github/carl/memory.md` — workflow inventory/canonical command reconciliation (if needed)
+- `.gitignore` — hardened generated/local artefact boundaries
+- `inventory/api-index-23389057738.zip` — remove tracked generated export if confirmed in scope
+- `inventory/api-index-sharded-23390552485.zip` — remove tracked generated export if confirmed in scope
+- `README.md` and/or `CONTRIBUTING.md` — ownership clarification only if needed
+- `.github/carl/memory.md` — durable generated-artefact boundary reconciliation (if needed)
 
 ## Tests / validation
 - Install runtime deps: `pip3 install -r requirements.txt`
 - Install test tool explicitly: `pip3 install pytest`
 - Run lightweight tests: `python3 -m pytest tests/test_api_inventory_export.py tests/test_api_inventory_normalization.py -v`
-- Confirm no generated artefact churn in `git status`
+- Confirm `git status` contains only intended `.gitignore`, docs, cARL, and generated ZIP removal changes
+- Confirm no generated artefacts are newly added
 
 ## Stop conditions
 - Any change that alters product script or CodeQL query behaviour.
-- Any generated database, inventory, or results file appearing in git status.
+- Any generated database, SARIF, JSON inventory, or shard file appearing in git status as a new addition.
 - Any secret detected in changed files.
 
 ## Escalation triggers
-- If requested workflow scope expands beyond lightweight pytest tests.
-- If changes require touching queries, generated artefacts, or dependency policy.
+- If hardening requires modifying workflows, product code, or query logic.
+- If additional tracked generated artefacts outside the approved scope need deletion.
+- If ignore patterns would overlap committed source files or docs unexpectedly.
 
 ## Context reset notes
-This contract covers adding a lightweight test entry point and CI workflow only. Close/reset after merge.
+This contract covers generated-artefact boundary hardening only. Close/reset after merge.

.github/carl/memory.md

@@ -51,7 +51,7 @@ SpeQL is a Python + CodeQL analysis pipeline targeting Azure REST API JSON specs
 |---|---|---|
 | `database/` | Generated; not committed | Built by refresh scripts from azure-rest-api-specs |
 | `results/` | Generated; gitignored | SARIF output from CodeQL queries |
-| `inventory/` | Generated; `.json`/`.min.json` gitignored | Exported API index artefacts |
+| `inventory/` | Generated; only `.gitkeep` tracked | Exported API index artefacts (`.json`, `.min.json`, `shards/`, and packaged `.zip` outputs are gitignored) |
 | `azure-rest-api-specs/` | Cloned externally; gitignored | Source spec corpus managed by refresh scripts |
 | `queries/azure-security/` | Committed | CodeQL query pack with `qlpack.yml` |
 | `config/sources/` | Committed | One JSON per spec source (currently `azure.json`) |
@@ -71,6 +71,7 @@ Adding a new source requires only a source config JSON and optionally a platform
 - `api-index-grouped.json` — nested by provider/host/route (with `--grouped`)
 - `shards/<provider>.json` — per-provider shards (with `--sharded`)
 - Minified variants (`*.min.json`) with `--minified`
+- Packaged sharded export ZIPs for release publication
 
 Published to UndREST-APISpy via GitHub Releases.
 
@@ -133,7 +134,7 @@ requires a package, document it explicitly rather than silently adding.
 
 ## Core invariants
 
-- Do not commit database/, results/, azure-rest-api-specs/, or generated inventory JSON files.
+- Do not commit database/, results/, azure-rest-api-specs/, or generated inventory exports (JSON, shards, or packaged ZIPs).
 - Always pin CodeQL CLI to 2.20.1 or 2.20.2 in setup, docs, and CI.
 - requirements.txt lists only runtime-required Python packages; test tools (pytest) are documented separately.
 - Export pipeline output is deterministic given the same spec corpus input.

.gitignore

@@ -1,22 +1,26 @@
-# Results directory
+# Generated SARIF and query results
+# Owned by: `run-queries.sh`, local CodeQL CLI runs, and `.github/workflows/speql-security-scan.yml`
 results/
 *.sarif
 
-# CodeQL temporary files
+# CodeQL temporary files and downloaded local tool copies
+# Owned by: local CodeQL CLI setup and query execution
 .codeql/
 *.qls
-
-# CodeQL libraries (downloaded/extracted)
 codeql-javascript/
 codeql-shared/
 codeql/
 
-# Python cache
+# Local Python cache and build artefacts
 __pycache__/
 *.py[cod]
 *$py.class
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
 
-# Virtual environments
+# Local virtual environments
+.venv/
 venv/
 env/
 ENV/
@@ -34,16 +38,22 @@ Thumbs.db
 # Logs
 *.log
 
-# Azure REST API specs clone (managed by refresh scripts)
+# Cloned upstream API spec corpus
+# Owned by: `refresh_database.py`, `refresh-database.sh`, and export/scan workflows
 azure-rest-api-specs/
 
-# Database directory (created by refresh scripts, not tracked in git)
-# Empty directories can't be committed to Git
+# Generated CodeQL databases and build logs
+# Owned by: `refresh_database.py`, `refresh-database.sh`, and `.github/workflows/speql-security-scan.yml`
 database/
 
-# API inventory export output (generated by scripts/export/)
+# Generated API inventory exports, shards, and packaged release inputs
+# Owned by: `scripts/export/export_api_inventory.py`,
+# `.github/workflows/daily-api-index-export.yml`, and
+# `.github/workflows/daily-api-index-export-sharded.yml`
 inventory/*.json
 inventory/*.min.json
+inventory/shards/
+inventory/*.zip
 
 # APISpy portal sweep output files
 apispy/scripts/apispy-*.csv

CONTRIBUTING.md

@@ -23,7 +23,7 @@ This repository uses [cARL](https://github.com/goldjg/cARL) for AI coding agent
 
 Key rules for agents and contributors:
 - **CodeQL CLI must be pinned to 2.20.1 or 2.20.2.** 2.23.x+ breaks JSON-only database builds.
-- **Do not commit** `database/`, `results/`, `azure-rest-api-specs/`, or generated inventory JSON files.
+- **Do not commit** `database/`, `results/`, `azure-rest-api-specs/`, or generated inventory exports (`inventory/*.json`, `inventory/shards/`, packaged inventory ZIPs).
 - **Lightweight tests** (no CodeQL/DB required): `python3 -m pytest tests/test_api_inventory_export.py tests/test_api_inventory_normalization.py -v`
 - **requirements.txt** lists only `pyfiglet`. Install `pytest` separately for testing.