test(verify): language-neutral verification conformance vectors

[aditisingh02]

What

Adds a portable, signed test-vector suite (AM-VEC-001..AM-VEC-019) for the verification engine, plus a Python test asserting the reference SDK reproduces every vector.

Why

The spec targets multi-language SDK parity (TypeScript, Go, .NET, Rust per the roadmap), but there is currently no executable, language-neutral contract for the verification semantics in spec Section 5. Today a new SDK has no way to prove it agrees with the reference implementation.

This suite is that contract. Each vector is a self-contained JSON of {manifest, context, expected} that any implementation, in any language, can load and check its VerificationResult against. It also pins down the RFC 8785 (JCS) signing pre-image — the byte-exact detail most likely to diverge across languages — and is the natural foundation for a future TypeScript/Go verifier.

Coverage spans the full OverallResult space: VALID (incl. a valid signed delegation chain and a matching attestation report), MISMATCH (artifact hash / tampered signature / flagged RAG poisoning scan), EXPIRED, REVOKED, INCOMPATIBLE_VERSION, SIGNATURE_MISSING, UNVERIFIABLE (no trusted keys; and a delegation chain present without keys), INCOMPLETE, ATTESTATION_UNAVAILABLE, plus HITL (approved/missing/expired) and memory-baseline TTL expiry.

Design notes:

• All vectors are signed with one fixed Ed25519 key (seed 00..1f, published in keys.json — test-only) and use absolute timestamps, so the suite is byte-reproducible (generate.py regenerates identically) and stable over time.
• AM-VEC-013 intentionally returns overall VALID while memory_baseline is EXPIRED, faithfully encoding the reference engine's per-field behaviour (documented in the suite README).

Spec impact

None — SDK/test-only. No normative text changes; the vectors encode the existing behaviour of verify_manifest (spec Section 5).

Test plan

• pytest -v passes (53 passed: 19 vectors + index-sync check + existing test_verify)
• mypy src/agent_manifest passes (unchanged — no src/ files touched)
• ruff check src/ tests/ passes (verified with CI flags on the new files)
• New or updated tests cover the change (tests/test_vectors.py parametrized over every vector)
• If spec change: CHANGELOG.md updated (N/A — no spec change)

Note: validated locally against src/ on Python 3.9 (no 3.11+ interpreter on my machine); CI will run the full 3.11/3.12/3.13 matrix and mypy.

DCO

All commits in this PR are signed off (git commit -s). By submitting this PR I certify the Developer Certificate of Origin.

[github-actions]

🟡 Contributor Check: MEDIUM

Check	Result
Profile	MEDIUM
Credential	NONE
Overall	MEDIUM

Automated check by AGT Contributor Check.

[imran-siddique]

Thanks @aditisingh02. The failing Lint and AGT governance verify checks here are a stale base, not a problem with your change: main has since landed the AGT CLI install fix (the governance job now installs agent-governance-toolkit-core) and the lint fix. Please rebase on the latest main (or merge main into your branch) and those should clear. Happy to review the conformance vectors once CI is green.

[aditisingh02]

Re: CodeQL "Clear-text storage of sensitive information" — python/tests/vectors/generate.py

These alerts are false positives. The flagged values are a fixed, public Ed25519 key and its key_id, published intentionally in keys.json to verify the conformance vectors; no private key material is written (private_key_b64url was removed earlier).

The security-extended query py/clear-text-storage-sensitive-data matches on identifier names, so it flags trusted_keys, key_id, and public_key_b64url regardless of value. These names cannot be changed, as they are required by the verification schema.

Could you advise on your preferred resolution:

1. A scoped .github/codeql/codeql-config.yml with paths-ignore: python/tests/vectors (ready to push), or
2. Dismissing the two alerts as "false positive / used in tests".

Happy to proceed with either.

[carloshvp]

Review pass on current head (9c7d9c1). This is a useful conformance addition: the vectors are self-contained, the index matches the AM-VEC-*.json files, keys.json contains only verifier/public-key fields, and the focused Python reference test passes locally.

Validated locally:

• PYTHONPATH=python/src python3 -m pytest python/tests/test_vectors.py -q -> 20 passed.
• index.json lists all 19 vector files on disk.
• keys.json fields are only algorithm, note, key_id, and public_key_b64url; no private_* field is written.

On the CodeQL question: I would avoid adding a broad paths-ignore: python/tests/vectors unless maintainers explicitly want that policy. These fixtures exercise verification, crypto identities, and trust material, so suppressing the whole directory would also hide future real findings in the generator or vector format.

My preference would be one of these narrower resolutions:

1. If maintainers are comfortable handling this in GitHub Security, dismiss the current alerts as false positive / public conformance fixture data, with the rationale that these are public verifier keys and deterministic test identifiers, not deployable secrets.
2. If the PR should clear CodeQL from source, add query-specific suppression comments only at the alert anchors in generate.py, e.g. # codeql[py/clear-text-storage-sensitive-data], with a short rationale comment that the values are public conformance-vector key IDs/public keys. CodeQL supports query-specific Python suppression comments, and this keeps the rest of python/tests/vectors covered.

One wording nuance: because the deterministic signing seed is intentionally present in generate.py for reproducibility, I’d phrase the security claim as “no secret/private production key material is written to generated fixtures” rather than “no private key material exists.” The current README and comments are already close, but that precision may avoid future confusion.

Small non-blocking nit: the docstring in python/tests/test_vectors.py says Conformance IDs: AM-VEC-001 .. AM-VEC-015, but this PR now adds through AM-VEC-019.

No functional blocker from my pass; the main decision is how maintainers want to record the CodeQL false-positive handling.

[imran-siddique]

Reviewed the harness, the index, and the expected results across all 19 vectors. This is a valuable, well-structured contract and the direction is right — leaving a substantive comment rather than a blind approve since I can't independently regenerate every signed pre-image byte-for-byte, and there are a couple of gaps worth closing first.

Strengths:

• Coverage genuinely spans the full OverallResult space with real positive AND negative cases: VALID (001/018/019, incl. attestation + signed delegation), MISMATCH from artifact-hash tamper (002, signature still valid) vs. from post-signing agent_id tamper (008, signature_verified=false — a genuine crypto negative), EXPIRED (003), REVOKED (004), SIGNATURE_MISSING (005), UNVERIFIABLE (006 no-keys, 012 delegation-without-keys), INCOMPATIBLE_VERSION (007), INCOMPLETE (016), ATTESTATION_UNAVAILABLE (017), plus HITL approved/missing/expired (009/010/011).
• The harness (test_vectors.py) asserts signature_verified on the VALID/negative cases, so positive vectors must actually verify against the published test key rather than pass trivially — and the full 3.11/3.12/3.13 x ubuntu/macos/windows matrix is green, which is good evidence the signed vectors reproduce.
• test_index_lists_every_vector_file guards index/file drift. Deterministic fixed-seed signing + absolute timestamps make the suite reproducible via generate.py. Pinning the RFC 8785 (JCS) pre-image is exactly the right thing to lock down for cross-language parity.

Please address before merge:

1. All 19 vectors sign with Ed25519 only. The two most divergence-prone and security-critical algorithms across languages — ML-DSA-65 and hybrid-Ed25519-ML-DSA-65 — have zero conformance coverage. Given #220/#221 just hardened the hybrid combined-key-id binding, a hybrid VALID vector plus a hybrid negative (component-substitution / key_id-mismatch -> MISMATCH) would make this suite the regression anchor those fixes need.
2. No wrong-key vector: key_id present in trusted_keys but mapped to the wrong public-key bytes -> expect MISMATCH/signature_verified=false. Today 008 covers manifest tamper and 006 covers no-keys, but not a mis-registered trusted key.
3. AM-VEC-013 asserts overall VALID while memory_baseline is EXPIRED. It's documented in the README as faithful to current engine per-field behaviour, but as a language-neutral contract this hard-codes that an expired memory baseline does NOT taint the overall result. Please confirm with the spec owners that this is intended normative behaviour and not just an implementation artifact — a conformance vector will freeze it for every future SDK.

Also: the CodeQL check and the "Policy: Awaiting maintainer review" gate show as failing on the rollup (the CI test matrix itself is green). Please re-run / confirm the CodeQL failure is transient before merge.

Happy to approve once the hybrid/ML-DSA and wrong-key vectors are added and the VEC-013 semantics are confirmed.

[imran-siddique]

The vectors are correct, well-formed, and deterministic, and I verified all 19 expected outcomes against the reference engine semantics. No false-conformance risk. A few things to close before this can serve as the language-neutral interop contract it's aiming to be.

1. Document the signing pre-image for non-Python implementers (blocking the stated goal).
signing_pre_image applies normalization that isn't described anywhere the vectors are consumed: hitl_record.approvals is forced to [] before canonicalization, and absent/null fields are dropped (exclude_none). The README only points at "see signing_pre_image in the SDK". A Go/TS/Rust implementer working from the vectors + README alone will canonicalize the literal hitl_record and hit InvalidSignature on AM-VEC-009 and AM-VEC-011. Please document the SIGNED_FIELDS list, the exclude_none rule, and the approvals=[] normalization directly in the README.

2. AM-VEC-013 promotes a debatable behavior into the normative contract.
An expired memory_baseline yields overall VALID (only the per-field status is EXPIRED). This faithfully mirrors the engine and is documented, so it's not a vector bug, but as a conformance vector it forces every future SDK to reproduce VALID for an expired security-relevant artifact. Worth a maintainer decision on whether that's intended before it becomes a cross-language guarantee.

3. Coverage gaps (follow-ups, not blockers):

• Wrong/unknown key_id with non-empty trusted_keys (the classic wrong-key MISMATCH) is untested; distinct from AM-VEC-006's empty-keys UNVERIFIABLE.
• No malformed-signature vector (non-base64url / wrong-length signature_value).
• No invalid-delegation vector (present chain with a bad hop signature -> INVALID).
• No schema-malformed manifest vector, despite "malformed" being a core conformance case.
• Ed25519 only; no ML-DSA-65 or hybrid vectors.

CI note: the two CodeQL clear-text-storage-sensitive-data highs at generate.py:381,385 are false positives, only public key material and public vector JSON are written and the private seed is never serialized. A maintainer will need to dismiss them.