diff --git a/.gitignore b/.gitignore index 706837c37..f3d5a68e7 100644 --- a/.gitignore +++ b/.gitignore @@ -18,3 +18,7 @@ build/ plan/ .omo/ +*.env +secrets/ +*.pem +*.key diff --git a/README.md b/README.md index f8e56b796..3f22fa055 100644 --- a/README.md +++ b/README.md @@ -110,6 +110,14 @@ replay audits, and weight submission. - **Signed enrollment** — the miner signs a hotkey↔worker binding; provider keys (`LIUM_API_KEY` / `TARGON_API_KEY`) stay in the miner's environment and never reach the master. - **Anti-collusion** — a worker never evaluates its owner's submission; each unit replicates across **R=2 distinct-owner** workers and is reconciled by `ExecutionProof.manifest_sha256`. - **Proof tiers** — tier 0 (manifest hash + sr25519 signature), tier 1 (pinned image digest), tier 2 (in-guest attestation, gated off on Targon). Audit sampling is tier-modulated. +- **Agent Challenge Phala Intel TDX path (separate from the PRISM worker plane):** BASE carries + the Phala-tier `ExecutionProof` / `EvalExecutionProof` schema (including `vm_config` ≤ 256 KiB), + quote verification helpers, R=1 assignment for attested units, and public-proxy deny rules so + agent-challenge capability, internal, and direct result-ingestion routes stay challenge-direct + (default off). Flag-off: legacy byte-identical signed submission/env/launch proxy path and R=1 + own_runner behavior. Full attested mode uses **one miner-funded external eval (R=1)** with + **zero** BASE validator re-exec multi-replica assignment; score acceptance remains challenge- + owned. Details: [Architecture](docs/architecture.md#agent-challenge-phala-intel-tdx-path). - **Admission rule** — when enforced, a miner needs ≥1 active bound worker to submit to Prism, else `403 NO_ACTIVE_WORKER`. See the miner worker deployment guide. diff --git a/docs/architecture.md b/docs/architecture.md index b07ffc77e..70d40f572 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -120,3 +120,73 @@ Operator install and deeper cardinality rules: [Compose-only deployment](compose - Helm / Kubernetes - Per-challenge Postgres servers managed by BASE - Automated destructive challenge purge without explicit operator action + +## Agent Challenge Phala Intel TDX path + +Agent Challenge attestation is **separate from the PRISM miner-funded GPU worker plane**. BASE owns the shared proof, proxy, and assignment surfaces below; end-to-end self-deploy review→eval, RA-TLS key release, and score acceptance live in the agent-challenge service when its own attestation flags are on (cross-repo challenge docs are available after PR merge). + +### Flag-off vs flag-on + +| Surface | Flag off (default) | Flag on | +|---------|--------------------|---------| +| Master setting | `master.agent_challenge_attested_routes_enabled=false` | `master.agent_challenge_attested_routes_enabled=true` | +| Public proxy | Legacy signed submission / env / launch passthrough | Fail-closed **allowlist** of review/eval + status/SSE + benchmark metadata only | +| Evaluation ownership | Legacy R=1 `own_runner` on validators (reassign on failure, never multi-replica) | Full attested mode: **one miner-funded external eval (R=1)**; BASE creates **zero** agent-challenge validator assignment, retry, replica, reconciliation, audit, or fold rows for those units | +| Phala verifier | Not required on agent-challenge results | Challenge/operators use BASE Phala-tier schema + quote helpers; BASE does not invent challenge-side score policy | + +Legacy path is byte-identical when the master attested-routes flag stays off. PRISM worker-plane replication (default R=2, `compute.replication_factor`) is unchanged either way. + +```mermaid +flowchart TB + M[Miner] --> P[Public BASE proxy] + P -->|allowlisted review/eval only when flag on| AC[agent-challenge service] + AC -->|miner-funded external Eval R=1| TEE[Phala Intel TDX CVM] + TEE -->|EvalExecutionProof tier phala-tdx| AC + AC -->|internal weights / control plane only| MSTR[Master] + V[Validators] -.->|sampled replay audit only when scheduled| AC +``` + +### Public vs private challenge proxy boundary + +Public clients reach challenges only at `/challenges/{slug}/...`. The proxy always blocks: + +- `/internal/*`, `/health`, `/version` +- Generic benchmark-execution-shaped paths (for example `/benchmark-executions` and benchmark `run` / `execute` / `launch` leaves) + +With **attested routes enabled**, agent-challenge is additionally **allowlist fail-closed** (`src/base/master/app_proxy.py`): only the exact signed review/eval rows, signed `POST /submissions`, `GET .../status` and `GET .../events`, and `GET /benchmarks/tasks` are forwardable. Capability, assignment, evidence, key-release, direct result ingestion, results aliases, env/launch (legacy), wrong methods, neighboring paths, and non-canonical raw path bytes are denied **locally** (404 before any upstream call). Private routes must never fall through the public proxy. + +On allowlisted agent-challenge routes the proxy: + +- Preserves miner signature headers `X-Hotkey`, `X-Signature`, `X-Nonce`, `X-Timestamp` where the miner signs the challenge-local path +- Strips hop-by-hop, internal, and attested trust-shaped headers/prefixes (for example `x-base-*`, `x-attestation-*`, `x-ra-tls-*`, client-IP spoofs) so edge clients cannot inject trust + +### ExecutionProof Phala tier (BASE schema) + +Every worker/eval proof envelope is the shared `ExecutionProof` model in `src/base/schemas/worker.py`: + +- Integer tiers **0 / 1 / 2** remain the PRISM worker-plane tiers (manifest + sr25519; image digest; optional in-guest attestation). +- Phala Intel TDX uses string tier **`phala-tdx`** (`PHALA_TDX_TIER`), with an `attestation` block (`PhalaAttestation` / strict `EvalPhalaAttestation`). +- Canonical **Eval** wire equals schema-closed `EvalExecutionProof` (version 1, tier `phala-tdx`, digest-pinned `image_digest`, empty worker-signature placeholders for validator rebind, no extra fields). +- Bound examples (fail closed at parse): TDX quote ≤ 64 KiB, event log ≤ 4096 entries / 2 MiB, **`vm_config` ≤ 256 KiB** (`EVAL_MAX_VM_CONFIG_BYTES`), closed `vm_config` fields `{vcpu, memory_mb, os_image_hash}`. +- Measurement registers use fixed hex widths; `report_data` is 64-byte (128 hex) left-aligned binding with domain tag `base-agent-challenge-v1` (`src/base/worker/proof.py`). + +### Quote verify and park vs reject + +BASE quote helpers (`src/base/worker/phala_quote.py`, `phala_verify.py`, `proof.py`) verify Phala-tier proofs: + +1. **Tier-0** worker (or rebound) signature over `sha256("{manifest_sha256}:{unit_id}")` always required. +2. Quote structure + RTMR3 event-log replay from the hardware-signed TD report (compose bound by content, not trusted by value). +3. Cryptographic DCAP verification via the **`dcap-qvl`** CLI: quote bytes are written to a **temp file** and passed by path (not as an inline argv hex body). +4. Measurement must match a non-empty validator **allowlist** (fail-closed when empty or unloaded). +5. Nonce freshness via validator-issued state; acceptable TCB default includes `UpToDate`. + +Outcomes: + +- Cryptographic / structure failure → **reject** +- Transient `dcap-qvl` missing, timeout, exit-0 non-JSON, or missing TCB status → **park** (`VerifierUnavailableError`): never accept, never permanent fraud-reject + +BASE does not claim perfect hardware trust. Treat Phala as **cryptographically-anchored trust-but-audit**; residual TEE and collateral risks remain (see [Security](security.md)). + +### Replay audit seam + +Sampled validator replay audits for agent-challenge use the labelled transport in `src/base/master/replay_audit.py` (`agent-challenge.replay-audit.v1`). Requests carry the complete immutable Eval plan; plan digests and trial scores are validated on the BASE wire before forward or accept. This is not ordinary multi-replica worker reconciliation. diff --git a/docs/challenge-integration.md b/docs/challenge-integration.md index 6fd7bebf8..0041fb32c 100644 --- a/docs/challenge-integration.md +++ b/docs/challenge-integration.md @@ -125,6 +125,16 @@ connection pooling, storage resize workflows, challenge Alembic migration automation, or automated destructive purge beyond the explicit operator scripts documented for the master Compose project ([compose.md](compose.md)). +## Agent Challenge attestation surfaces (BASE-owned) + +Integrator notes for challenges that participate in the Phala / attested topology (today: agent-challenge). Implement these contracts only when your challenge ships the matching attested mode; generated demo challenges still use the weight + SQLite contract above. + +- **Public proxy.** BASE never publicly proxies `/internal/*`, result-ingestion, capability, assignment, evidence, or key-release neighbors. Opt-in review/eval allowlisting is `master.agent_challenge_attested_routes_enabled` (default off keeps legacy submission/env/launch). +- **ExecutionProof.** Prefer the schema-closed Eval wire (`EvalExecutionProof`, tier `phala-tdx`) documented in [Architecture](architecture.md#executionproof-phala-tier-base-schema). Bound `vm_config` JSON encoding to **256 KiB**; quotes, event logs, and string fields have fixed ceilings in `src/base/schemas/worker.py`. +- **R=1 full attested mode.** When the challenge exposes no assignable work units for a fully attested submission, BASE creates **zero** validator multi-replica work rows for that submission. Do not rely on BASE worker-plane R=2 reconciliation for that path; use challenge-owned miner-funded external eval and BASE shared proof helpers only where you integrate them. +- **Flag off.** Leave BASE and challenge attestation flags off for the legacy R=1 `own_runner` / env-launch path. Mixed topologies are unsupported. +- Challenge-owned review→eval and RA-TLS containers, images, and operator docs: **available after PR merge** in the agent-challenge repository. + ## Build and publish The generated CI workflow tests the challenge and pushes its Docker image to GHCR diff --git a/docs/challenges.md b/docs/challenges.md index 0e0d196c3..5e4ce3723 100644 --- a/docs/challenges.md +++ b/docs/challenges.md @@ -101,3 +101,8 @@ execution routes. Challenges export raw **hotkey** weights. The master aggregates and serves the final vector; validators fetch and call `set_weights`. Challenges never submit final UID vectors and never receive master database credentials. + +## Agent Challenge Phala attestation notes + +Private control-plane work (work units, fold, weights, bridge launch) stays on challenge-direct or master-internal channels, never on the public edge. Full attested mode evaluation is one miner-funded external eval (R=1) with **no** BASE validator multi-replica re-exec assignment for those units; cross-repo review→eval behavior is available after PR merge. See [Architecture: Agent Challenge Phala path](architecture.md#agent-challenge-phala-intel-tdx-path). +5. If attested routes are enabled, confirm the client hit an allowlisted review/eval/status path (not a private result or capability route). diff --git a/docs/master/README.md b/docs/master/README.md index 319e00185..5804fdd7f 100644 --- a/docs/master/README.md +++ b/docs/master/README.md @@ -96,3 +96,17 @@ Backup / restore / teardown scripts (Compose-only): unsupported** for new installs. They must not be documented as the required master path and must not be run against live production Swarm fabric from this guide. See [deploy/swarm/README.md](../../deploy/swarm/README.md) for the explicit non-target banner. + +## Agent Challenge attested proxy flag + +The master proxy setting `master.agent_challenge_attested_routes_enabled` (default +**false**) selects the public agent-challenge topology: + +- **Off:** legacy signed submission / env / launch passthrough (byte-identical). +- **On:** fail-closed allowlist for review/eval miner flows; private and result + routes never fall through the public edge. + +Full attested evaluation ownership and score policy stay in the agent-challenge +service. See [Architecture](../architecture.md#agent-challenge-phala-intel-tdx-path) +and [Challenges](../challenges.md). Foundation install diffs for challenge-side +CVM credentials are out of scope here and must not appear in this guide. diff --git a/docs/miner/worker-plane.md b/docs/miner/worker-plane.md index 469efcc7c..3f840eebd 100644 --- a/docs/miner/worker-plane.md +++ b/docs/miner/worker-plane.md @@ -17,9 +17,14 @@ This guide covers, for both **Lium** and **Targon**: - [Monitoring your fleet](#monitoring-your-fleet) - [Troubleshooting](#troubleshooting) -> The whole worker plane is gated behind a feature flag +> The whole (PRISM) worker plane is gated behind a feature flag > (`compute.worker_plane_enabled` on the master, `worker_plane` in the prism config). With > the flag off, nothing here applies and the subnet behaves exactly as before. +> +> **This guide is PRISM-only.** Agent Challenge Phala Intel TDX attestation (other +> challenge, separate flags, R=1 external eval, public-proxy allowlist) is **not** the +> miner GPU worker plane. See [Architecture: Agent Challenge Phala path](../architecture.md#agent-challenge-phala-intel-tdx-path) +> and the agent-challenge operator docs (available after PR merge). --- diff --git a/docs/security.md b/docs/security.md index a8608fd93..931cf694d 100644 --- a/docs/security.md +++ b/docs/security.md @@ -97,3 +97,16 @@ and must not display raw text such as `BASE request failed with status 502`. * LLM gateway services, routes, tokens, and provider clients * Swarm install path for greenfield hosts (`install-swarm.sh`, Swarm secrets, overlays) * Application-launched evaluator containers + +## Phala attestation residual risk + +BASE ships fail-closed Phala-tier verification helpers (schema bounds, measurement allowlist, `dcap-qvl` temp-file quote verify, park-vs-reject on verifier outage). Product posture is **cryptographically-anchored trust-but-audit**, not a claim of absolute TEE safety. + +Residual risks operators should plan for: + +* Hardware / microarchitectural TEE residual risk (including published TDX class attacks such as TEE.fail-style research) is not eliminated by software allowlists. +* Intel PCS / collateral freshness and `dcap-qvl` availability affect park vs reject; a parked result is intentionally not accepted. +* Challenge-owned score acceptance, RA-TLS key release, and CVM image pinning remain agent-challenge responsibilities (available after PR merge for cross-repo docs). BASE proxy must never expose those private challenge routes on the public edge. +* Flag-off (`agent_challenge_attested_routes_enabled=false`) preserves the legacy non-Phala public surfaces; operators enabling the flag must deploy the matching agent-challenge attested mode rather than mixing topologies. + +* Public proxy strips sensitive headers (and, for agent-challenge attested mode, strips attested trust-shaped headers and prefixes). diff --git a/docs/validator/README.md b/docs/validator/README.md index 1f401dfb6..437ea836e 100644 --- a/docs/validator/README.md +++ b/docs/validator/README.md @@ -174,3 +174,5 @@ uv run pytest tests/unit/test_validator_compose_artifact.py tests/unit/test_vali Start a live submit path only when wallet material on the host is intentionally authorized for chain use. CI publishes Docker images to GHCR only from trusted events. + +Agent Challenge Phala full-attested mode is also **not** a multi-replica validator re-exec path: evaluation is one miner-funded external eval (R=1) and BASE creates zero agent-challenge multi-replica work rows for that submission. Validators may still run sampled labelled replay audits and the legacy R=1 `own_runner` path when attestation flags are off. See [Architecture: Agent Challenge Phala path](../architecture.md#agent-challenge-phala-intel-tdx-path). diff --git a/src/base/cli_app/main.py b/src/base/cli_app/main.py index 3637773ec..cf26381be 100644 --- a/src/base/cli_app/main.py +++ b/src/base/cli_app/main.py @@ -49,6 +49,8 @@ from base.master.challenge_client import ChallengeClient from base.master.challenge_work_source import ( HttpChallengeFoldTrigger, + HttpChallengeReplayClient, + HttpChallengeReplaySource, HttpChallengeResultForwarder, HttpChallengeWorkSource, ) @@ -586,6 +588,16 @@ def _master_orchestration_driver( timeout_seconds=settings.master.challenge_timeout_seconds, retries=settings.master.challenge_retries, ), + replay_source=HttpChallengeReplaySource( + registry, + timeout_seconds=settings.master.challenge_timeout_seconds, + retries=settings.master.challenge_retries, + ), + replay_result_forwarder=HttpChallengeReplayClient( + registry, + timeout_seconds=settings.master.challenge_timeout_seconds, + retries=settings.master.challenge_retries, + ), fold_trigger=HttpChallengeFoldTrigger( registry, timeout_seconds=settings.master.challenge_timeout_seconds, @@ -1157,6 +1169,9 @@ def master_proxy(config: Path = typer.Option(Path("config/master.example.yaml")) ), identity_resolver=ValidatorIdentityResolver(cache=runtime.identity_cache), readiness_probes=(database_probe,), + agent_challenge_attested_routes_enabled=( + settings.master.agent_challenge_attested_routes_enabled + ), ) endpoint = f"{settings.master.proxy_host}:{settings.master.proxy_port}" typer.echo(f"Starting proxy API on {endpoint}") diff --git a/src/base/config/settings.py b/src/base/config/settings.py index 43656d9b1..b08dfd9f5 100644 --- a/src/base/config/settings.py +++ b/src/base/config/settings.py @@ -63,6 +63,9 @@ class MasterSettings(BaseModel): upload_require_registered_hotkey: bool = True # ss58 hotkeys accepted without on-chain registration (QA/allowlist; empty in prod) upload_extra_registered_hotkeys: list[str] = Field(default_factory=list) + # Opt-in canonical Agent Challenge review/eval proxy topology. OFF preserves + # the legacy signed submission/env/launch proxy behavior byte-for-byte. + agent_challenge_attested_routes_enabled: bool = False # Validator coordination plane (architecture.md sec 4). The proxy serves the # hotkey-signed register/heartbeat/pull/progress/result routes, returns # ``validator_heartbeat_interval_seconds`` to validators, marks a validator diff --git a/src/base/master/app_proxy.py b/src/base/master/app_proxy.py index a0428f089..f72e1b894 100644 --- a/src/base/master/app_proxy.py +++ b/src/base/master/app_proxy.py @@ -129,6 +129,34 @@ "x-base-request-hash", } +ATTESTED_SENSITIVE_REQUEST_HEADERS = { + "cf-connecting-ip", + "forwarded", + "true-client-ip", + "via", + "x-base-challenge-slug", + "x-base-proxy", + "x-internal-authorization", + "x-proxy-trust", + "x-real-ip", +} + +ATTESTED_SENSITIVE_REQUEST_HEADER_PREFIXES = ( + "x-admin-", + "x-allowlist-", + "x-attestation-", + "x-base-", + "x-forwarded-", + "x-internal-", + "x-measurement-", + "x-proxy-", + "x-ra-tls-", + "x-ratls-", + "x-review-", + "x-trust-", + "x-trusted-", +) + MINER_SIGNATURE_HEADERS = { "x-hotkey", "x-signature", @@ -259,27 +287,166 @@ def _is_agent_challenge_env_route(slug: str, method: str, path: str) -> bool: return False -def _is_agent_challenge_signed_route(slug: str, method: str, path: str) -> bool: +def _is_agent_challenge_signed_route( + slug: str, + method: str, + path: str, + *, + attested_routes_enabled: bool = False, +) -> bool: """Routes where the miner signs the challenge-local path. The miner's signature headers (``X-Hotkey``/``X-Signature``/``X-Nonce``/ ``X-Timestamp``) must survive the generic ``/challenges/{slug}`` passthrough - so the challenge can verify them. This covers the signed env actions plus the - JSON base64 submission upload (``POST /submissions``) per the frontend API - contract. + so the challenge can verify them. Fully legacy mode keeps its env/launch + actions. Full attested mode replaces those with the exact review/eval + allowlist while retaining signed ``POST /submissions``. """ - if _is_agent_challenge_env_route(slug, method, path): - return True if slug != "agent-challenge": return False + normalized = normpath(f"/{path.lstrip('/')}") parts = [part for part in normalized.split("/") if part] - return len(parts) == 1 and parts[0] == "submissions" and method.upper() == "POST" + normalized_method = method.upper() + if not attested_routes_enabled: + return _is_agent_challenge_env_route(slug, method, path) or ( + len(parts) == 1 + and parts[0] == "submissions" + and normalized_method == "POST" + ) + + canonical_path = f"/{path}" + if path.startswith("/") or canonical_path != normalized: + return False + + if len(parts) == 1 and parts[0] == "submissions": + return normalized_method == "POST" + if len(parts) != 4 or parts[0] != "submissions": + return False + + route = (parts[2], parts[3]) + signed_post_routes = { + ("review", "prepare"), + ("review", "retry"), + ("review", "deployed"), + ("review", "cancel"), + ("eval", "prepare"), + ("eval", "retry"), + ("eval", "cancel"), + ("eval", "failure"), + } + signed_get_routes = { + ("review", "report"), + ("eval", "status"), + } + return (normalized_method == "POST" and route in signed_post_routes) or ( + normalized_method == "GET" and route in signed_get_routes + ) + + +def _is_agent_challenge_enabled_mode_allowed_route( + slug: str, + method: str, + path: str, +) -> bool: + """Return whether an enabled-mode Agent Challenge proxy path is allowlisted. + + When attested routes are enabled the BASE proxy is fail-closed: only the + exact signed review/eval rows plus the normative status/SSE and benchmark + metadata surfaces may reach agent-challenge. Capability, assignment, + evidence, key-release, direct-result, results, and every other neighbor are + denied locally before any upstream call. + """ + + if slug != "agent-challenge": + return False + + normalized = normpath(f"/{path.lstrip('/')}") + parts = [part for part in normalized.split("/") if part] + normalized_method = method.upper() + canonical_path = f"/{path}" + if path.startswith("/") or canonical_path != normalized: + return False + + if _is_agent_challenge_signed_route( + slug, + method, + path, + attested_routes_enabled=True, + ): + return True + + if ( + len(parts) == 3 + and parts[0] == "submissions" + and parts[2] in {"status", "events"} + and normalized_method == "GET" + ): + return True + if ( + len(parts) == 2 + and parts[0] == "benchmarks" + and parts[1] == "tasks" + and normalized_method == "GET" + ): + return True + return False + + +def _is_blocked_agent_challenge_proxy_path( + slug: str, + method: str, + path: str, + *, + attested_routes_enabled: bool, +) -> bool: + """Deny every enabled-mode Agent Challenge route outside the allowlist. + + Capability, assignment, evidence, key-release, direct-result, and results + aliases never fall through to the challenge. Wrong methods and neighboring + review/eval routes are denied locally so an upstream catch-all can never + widen this allowlist. + """ + + if slug != "agent-challenge" or not attested_routes_enabled: + return False + + return not _is_agent_challenge_enabled_mode_allowed_route(slug, method, path) + + +def _has_canonical_agent_challenge_raw_path( + request: Request, *, slug: str, path: str +) -> bool: + """Require literal canonical BASE prefix/path bytes for attested routing.""" + + expected = f"/challenges/{slug}" + if path: + expected = f"{expected}/{path}" + try: + expected_bytes = expected.encode("ascii") + except UnicodeEncodeError: + return False + return request.scope.get("raw_path") == expected_bytes + + +def _is_sensitive_request_header( + lowered: str, *, strip_attested_trust_headers: bool +) -> bool: + return lowered in SENSITIVE_REQUEST_HEADERS or ( + strip_attested_trust_headers + and ( + lowered in ATTESTED_SENSITIVE_REQUEST_HEADERS + or lowered.startswith(ATTESTED_SENSITIVE_REQUEST_HEADER_PREFIXES) + ) + ) def _forward_headers( - request: Request, *, preserve_miner_signature_headers: bool = False + request: Request, + *, + preserve_miner_signature_headers: bool = False, + strip_attested_trust_headers: bool = False, ) -> dict[str, str]: """Copy safe request headers for forwarding to a public challenge route.""" @@ -291,7 +458,13 @@ def _forward_headers( ) if ( lowered in HOP_BY_HOP_HEADERS - or (lowered in SENSITIVE_REQUEST_HEADERS and not preserve_header) + or ( + _is_sensitive_request_header( + lowered, + strip_attested_trust_headers=strip_attested_trust_headers, + ) + and not preserve_header + ) or lowered == "host" ): continue @@ -447,6 +620,7 @@ def create_proxy_app( identity_resolver: ValidatorIdentityResolver | None = None, allowed_cors_origins: list[str] | None = None, readiness_probes: Sequence[ReadinessProbe] = (), + agent_challenge_attested_routes_enabled: bool = False, ) -> FastAPI: """Create the public proxy FastAPI app. @@ -686,6 +860,38 @@ async def proxy_request(slug: str, path: str, request: Request) -> Response: status_code=status.HTTP_404_NOT_FOUND, detail="Not Found", ) + if ( + agent_challenge_attested_routes_enabled + and slug == "agent-challenge" + and not _has_canonical_agent_challenge_raw_path( + request, + slug=slug, + path=path, + ) + ): + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail="Proxy path not found", + ) + if ( + agent_challenge_attested_routes_enabled + and slug != "agent-challenge" + and slug.casefold() in {"agent challenge", "agent-challenge"} + ): + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail="Proxy path not found", + ) + if _is_blocked_agent_challenge_proxy_path( + slug, + request.method, + path, + attested_routes_enabled=agent_challenge_attested_routes_enabled, + ): + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail="Proxy path not found", + ) if is_blocked_proxy_path(path): raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, @@ -701,7 +907,13 @@ async def proxy_request(slug: str, path: str, request: Request) -> Response: headers = _forward_headers( request, preserve_miner_signature_headers=_is_agent_challenge_signed_route( - slug, request.method, path + slug, + request.method, + path, + attested_routes_enabled=agent_challenge_attested_routes_enabled, + ), + strip_attested_trust_headers=( + agent_challenge_attested_routes_enabled and slug == "agent-challenge" ), ) headers["X-Base-Challenge-Slug"] = slug diff --git a/src/base/master/assignment.py b/src/base/master/assignment.py index eeac1bd40..0aa796a66 100644 --- a/src/base/master/assignment.py +++ b/src/base/master/assignment.py @@ -25,8 +25,15 @@ ValidatorStatus, WorkAssignment, WorkAssignmentStatus, + WorkResult, ) from base.db.session import session_scope +from base.master.replay_audit import ( + REPLAY_AUDIT_ASSIGNMENT_KIND, + REPLAY_AUDIT_FORWARDED_KEY, + ReplayAuditRequest, + replay_assignment_payload, +) CAPABILITY_CPU = "cpu" CAPABILITY_GPU = "gpu" @@ -264,6 +271,104 @@ async def create_prism_work_unit( ) return work_unit_id + async def create_replay_audit_work_unit( + self, + *, + request: ReplayAuditRequest, + max_attempts: int | None = None, + challenge_slug: str = AGENT_CHALLENGE_SLUG, + ) -> str: + """Create one CPU assignment for one labelled replay audit. + + A replay is one full-plan unit, not one ordinary task unit. The exact + request body is copied into the payload and is never rebuilt from local + scoring configuration. Repeated discovery returns the same unit. + """ + + now = self._now_fn() + max_att = self._default_max_attempts if max_attempts is None else max_attempts + work_unit_id = request.work_unit_id + async with session_scope(self._session_factory) as session: + existing = ( + await session.execute( + select(WorkAssignment).where( + WorkAssignment.challenge_slug == challenge_slug, + WorkAssignment.work_unit_id == work_unit_id, + ) + ) + ).scalar_one_or_none() + if existing is not None: + if existing.payload.get("replay_audit_request") != request.to_dict(): + raise ValueError("replay assignment identity changed") + return work_unit_id + session.add( + WorkAssignment( + challenge_slug=challenge_slug, + work_unit_id=work_unit_id, + submission_ref=request.submission_id, + payload=replay_assignment_payload(request), + required_capability=CAPABILITY_CPU, + status=WorkAssignmentStatus.PENDING, + attempt_count=0, + max_attempts=max_att, + created_at=now, + updated_at=now, + ) + ) + return work_unit_id + + async def get_unforwarded_replay_results( + self, *, challenge_slug: str = AGENT_CHALLENGE_SLUG + ) -> list[tuple[WorkAssignment, WorkResult]]: + """Return completed labelled replay results awaiting challenge delivery.""" + + async with self._session_factory() as session: + assignments = ( + ( + await session.execute( + select(WorkAssignment) + .where(WorkAssignment.challenge_slug == challenge_slug) + .where(WorkAssignment.status == WorkAssignmentStatus.COMPLETED) + ) + ) + .scalars() + .all() + ) + output: list[tuple[WorkAssignment, WorkResult]] = [] + for assignment in assignments: + if ( + assignment.payload.get("assignment_kind") + != REPLAY_AUDIT_ASSIGNMENT_KIND + or not assignment.result_ref + ): + continue + result = ( + await session.execute( + select(WorkResult).where( + WorkResult.assignment_id == assignment.id + ) + ) + ).scalar_one_or_none() + if result is None or (result.payload or {}).get( + REPLAY_AUDIT_FORWARDED_KEY + ): + continue + output.append((assignment, result)) + return output + + async def mark_replay_result_forwarded(self, result_id: str) -> None: + import uuid + + parsed = uuid.UUID(result_id) + async with session_scope(self._session_factory) as session: + result = ( + await session.execute(select(WorkResult).where(WorkResult.id == parsed)) + ).scalar_one_or_none() + if result is not None: + payload = dict(result.payload or {}) + payload[REPLAY_AUDIT_FORWARDED_KEY] = True + result.payload = payload + def transaction(self) -> AbstractAsyncContextManager[AsyncSession]: """Open a single committed transaction over the control-plane DB. diff --git a/src/base/master/challenge_work_source.py b/src/base/master/challenge_work_source.py index 8ece58c85..dd2c3b5c1 100644 --- a/src/base/master/challenge_work_source.py +++ b/src/base/master/challenge_work_source.py @@ -29,6 +29,12 @@ WORK_UNIT_MAX_ATTEMPTS_REASON, ChallengePendingWork, ) +from base.master.replay_audit import ( + ReplayAuditRequest, + ReplayAuditResult, + ReplayAuditWireError, + parse_replay_json, +) logger = logging.getLogger(__name__) @@ -105,6 +111,184 @@ async def _fetch_work_units( return None +class HttpChallengeReplayClient: + """Fetch labelled replay requests and post raw replay trials. + + This client is intentionally separate from :class:`HttpChallengeWorkSource`. + The normal ``/internal/v1/work_units`` route is never used for replay audits, + and a replay body is accepted only after the explicit protocol label and + immutable-plan checks pass. + """ + + def __init__( + self, + registry: Any, + *, + timeout_seconds: float = 10.0, + retries: int = 3, + transport: httpx.AsyncBaseTransport | None = None, + ) -> None: + self._registry = registry + self._timeout_seconds = timeout_seconds + self._retries = retries + self._transport = transport + + async def fetch_request( + self, *, challenge_slug: str, eval_run_id: str + ) -> ReplayAuditRequest: + record = await _resolve(self._registry.get(challenge_slug)) + token = await _resolve(self._registry.get_token(challenge_slug)) + if not token: + raise RuntimeError( + f"challenge {challenge_slug!r} has no token for replay request" + ) + url = ( + f"{record.internal_base_url.rstrip('/')}/internal/v1/replay-audits/" + f"{eval_run_id}/request" + ) + headers = { + "Authorization": f"Bearer {token}", + "X-Base-Challenge-Slug": challenge_slug, + "Accept": "application/json", + } + last_error: Exception | None = None + for _attempt in range(max(self._retries, 1)): + try: + async with httpx.AsyncClient( + timeout=self._timeout_seconds, transport=self._transport + ) as client: + response = await client.get(url, headers=headers) + response.raise_for_status() + raw = response.content + parsed = parse_replay_json(raw) + if not isinstance(parsed, dict): + raise ValueError("replay request response must be an object") + request = ReplayAuditRequest.from_mapping(parsed, raw_body=raw) + if request.eval_run_id != eval_run_id: + raise ValueError("replay request run identity mismatch") + return request + except ReplayAuditWireError: + raise + except (ValueError, TypeError): + raise + except Exception as exc: # noqa: BLE001 - retry transport failures + last_error = exc + assert last_error is not None + raise RuntimeError( + f"failed to fetch replay request for {eval_run_id} on " + f"{challenge_slug}: {last_error}" + ) from last_error + + async def post_result( + self, + *, + challenge_slug: str, + result: ReplayAuditResult, + ) -> dict[str, Any]: + record = await _resolve(self._registry.get(challenge_slug)) + token = await _resolve(self._registry.get_token(challenge_slug)) + if not token: + raise RuntimeError( + f"challenge {challenge_slug!r} has no token for replay result" + ) + url = ( + f"{record.internal_base_url.rstrip('/')}/internal/v1/replay-audits/" + f"{result.eval_run_id}/result" + ) + headers = { + "Authorization": f"Bearer {token}", + "X-Base-Challenge-Slug": challenge_slug, + "Accept": "application/json", + "Content-Type": "application/json", + } + body = result.to_dict() + last_error: Exception | None = None + for _attempt in range(max(self._retries, 1)): + try: + async with httpx.AsyncClient( + timeout=self._timeout_seconds, transport=self._transport + ) as client: + response = await client.post(url, json=body, headers=headers) + response.raise_for_status() + payload = parse_replay_json(response.content) + if not isinstance(payload, dict): + raise ValueError("replay result response must be an object") + return payload + except (ReplayAuditWireError, ValueError, TypeError): + raise + except Exception as exc: # noqa: BLE001 - retry transient HTTP failures + last_error = exc + assert last_error is not None + raise RuntimeError( + f"failed to post replay result {result.audit_id} on " + f"{challenge_slug}: {last_error}" + ) from last_error + + async def forward( + self, *, challenge_slug: str, result: ReplayAuditResult + ) -> dict[str, Any]: + """Forward one validated replay result to the challenge comparator.""" + + return await self.post_result(challenge_slug=challenge_slug, result=result) + + +class HttpChallengeReplaySource: + """Discover sampled replay requests without touching normal work units.""" + + def __init__( + self, + registry: Any, + *, + timeout_seconds: float = 10.0, + retries: int = 3, + transport: httpx.AsyncBaseTransport | None = None, + ) -> None: + self._registry = registry + self._client = HttpChallengeReplayClient( + registry, + timeout_seconds=timeout_seconds, + retries=retries, + transport=transport, + ) + + async def fetch_sampled_requests(self) -> list[ReplayAuditRequest]: + records = await _resolve(self._registry.list(active_only=True)) + requests: list[ReplayAuditRequest] = [] + for record in records: + if record.slug != "agent-challenge": + continue + token = await _resolve(self._registry.get_token(record.slug)) + if not token: + continue + url = ( + f"{record.internal_base_url.rstrip('/')}" + "/internal/v1/replay-audits/requests" + ) + headers = { + "Authorization": f"Bearer {token}", + "X-Base-Challenge-Slug": record.slug, + "Accept": "application/json", + } + try: + async with httpx.AsyncClient( + timeout=self._client._timeout_seconds, + transport=self._client._transport, + ) as client: + response = await client.get(url, headers=headers) + response.raise_for_status() + body = parse_replay_json(response.content) + raw_requests = body.get("requests") if isinstance(body, dict) else None + if not isinstance(raw_requests, list): + raise ValueError("replay request list must contain requests") + for raw in raw_requests: + requests.append(ReplayAuditRequest.from_mapping(raw)) + except ReplayAuditWireError: + raise + except Exception as exc: # noqa: BLE001 - audit is best effort + logger.warning("failed to discover replay requests: %s", exc) + return requests + + def _parse_work_units(slug: str, payload: dict[str, Any]) -> list[ChallengePendingWork]: """Map a challenge ``work_units`` response into bridgeable pending work. @@ -298,6 +482,8 @@ async def forward_result( __all__ = [ "HttpChallengeFoldTrigger", + "HttpChallengeReplayClient", + "HttpChallengeReplaySource", "HttpChallengeResultForwarder", "HttpChallengeWorkSource", ] diff --git a/src/base/master/orchestration.py b/src/base/master/orchestration.py index 329332842..689c4f745 100644 --- a/src/base/master/orchestration.py +++ b/src/base/master/orchestration.py @@ -49,6 +49,10 @@ challenge_spec_from_registry, ) from base.master.reassignment import ReassignmentPassResult, run_reassignment_pass +from base.master.replay_audit import ( + ReplayAuditRequest, + ReplayAuditResult, +) from base.master.validator_coordination import ValidatorCoordinationService from base.master.worker_assignment_engine import ( WorkerAssignmentEngine, @@ -94,6 +98,12 @@ class ChallengePendingWork: payload: Mapping[str, Any] = field(default_factory=dict) +class ReplayAuditSource(Protocol): + """Source of sampled, separately-labelled replay requests.""" + + async def fetch_sampled_requests(self) -> Sequence[Any]: ... + + class ChallengeWorkSource(Protocol): """Source of each challenge's currently-assignable pending work units.""" @@ -145,6 +155,8 @@ def __init__( assignment_service: AssignmentService, validator_service: ValidatorCoordinationService, work_source: ChallengeWorkSource, + replay_source: ReplayAuditSource | None = None, + replay_result_forwarder: Any | None = None, fold_trigger: ChallengeFoldTrigger | None = None, worker_assignment_engine: WorkerAssignmentEngine | None = None, worker_reconciler: WorkerReconciliationService | None = None, @@ -153,6 +165,8 @@ def __init__( self._assignment_service = assignment_service self._validator_service = validator_service self._work_source = work_source + self._replay_source = replay_source + self._replay_result_forwarder = replay_result_forwarder self._fold_trigger = fold_trigger self._worker_assignment_engine = worker_assignment_engine self._worker_reconciler = worker_reconciler @@ -197,6 +211,56 @@ async def bridge_pending_work(self) -> dict[str, list[str]]: bridged.setdefault(work.challenge_slug, []).append(work_unit_id) return bridged + async def bridge_replay_requests(self) -> list[str]: + """Materialize only sampled labelled replay requests as assignments.""" + + if self._replay_source is None: + return [] + requests = await self._replay_source.fetch_sampled_requests() + created: list[str] = [] + for request in requests: + created.append( + await self._assignment_service.create_replay_audit_work_unit( + request=request, + ) + ) + return created + + async def forward_replay_results(self) -> list[str]: + """Deliver completed replay trials and mark forwarding durably.""" + + if self._replay_result_forwarder is None: + return [] + forwarded: list[str] = [] + for ( + assignment, + result_row, + ) in await self._assignment_service.get_unforwarded_replay_results(): + payload = dict(result_row.payload or {}).get("replay_audit_result") + if not isinstance(payload, Mapping): + logger.warning( + "replay assignment %s returned malformed result", + assignment.work_unit_id, + ) + continue + result = ReplayAuditResult.from_mapping(payload) + result.validate_against( + # The request was validated at assignment creation and preserved + # byte-for-byte in the assignment payload. + ReplayAuditRequest.from_mapping( + assignment.payload["replay_audit_request"] + ) + ) + await self._replay_result_forwarder.forward( + challenge_slug=assignment.challenge_slug, + result=result, + ) + await self._assignment_service.mark_replay_result_forwarded( + str(result_row.id) + ) + forwarded.append(assignment.work_unit_id) + return forwarded + async def run_once(self) -> OrchestrationPassResult: """Bridge work, reassign, replicate + reconcile worker units, then fold. @@ -207,6 +271,7 @@ async def run_once(self) -> OrchestrationPassResult: """ bridged = await self.bridge_pending_work() + replayed = await self.bridge_replay_requests() reassignment = await run_reassignment_pass( validator_service=self._validator_service, assignment_service=self._assignment_service, @@ -222,6 +287,9 @@ async def run_once(self) -> OrchestrationPassResult: if self._worker_reconciler is not None: reconciliation = await self._worker_reconciler.reconcile_once() folded = await self._fold_failed() + await self.forward_replay_results() + if replayed: + bridged.setdefault(AGENT_CHALLENGE_SLUG, []).extend(replayed) return OrchestrationPassResult( bridged=bridged, reassignment=reassignment, diff --git a/src/base/master/replay_audit.py b/src/base/master/replay_audit.py new file mode 100644 index 000000000..344b52fce --- /dev/null +++ b/src/base/master/replay_audit.py @@ -0,0 +1,746 @@ +"""BASE transport and validation for the labelled replay-audit seam. + +Replay audits are deliberately not ordinary challenge work. A request is +eligible only when the challenge labels it with the replay-audit protocol and +includes the complete immutable Eval plan. This module keeps that discriminator +and the plan bytes together while the request crosses BASE's assignment plane. +It also validates raw ordered trial scores before a result can be forwarded back +to the challenge comparator. +""" + +from __future__ import annotations + +import json +import math +import re +import struct +import unicodedata +from collections.abc import Mapping, Sequence +from dataclasses import dataclass, field +from hashlib import sha256 +from typing import Any + +REPLAY_AUDIT_LABEL = "agent-challenge.replay-audit.v1" +REPLAY_AUDIT_REQUEST_KIND = "replay_audit_request" +REPLAY_AUDIT_RESULT_KIND = "replay_audit_result" +REPLAY_AUDIT_ASSIGNMENT_KIND = "replay_audit" + +REPLAY_AUDIT_LABEL_KEY = "replay_audit_label" +REPLAY_AUDIT_REQUEST_KEY = "replay_audit_request" +REPLAY_AUDIT_RESULT_KEY = "replay_audit_result" +REPLAY_AUDIT_FORWARDED_KEY = "replay_audit_result_forwarded" + +_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") +_REGISTER_RE = re.compile(r"^[0-9a-f]{96}$") +_F64_RE = re.compile(r"^[0-9a-f]{16}$") +_IMAGE_RE = re.compile(r"^[^@\s]+@sha256:[0-9a-f]{64}$") + + +class ReplayAuditWireError(ValueError): + """A replay payload is malformed or fails its immutable identity checks.""" + + +def _require_id(value: Any, name: str) -> str: + if ( + not isinstance(value, str) + or not (1 <= len(value) <= 128) + or any(not ("!" <= char <= "~") for char in value) + ): + raise ReplayAuditWireError(f"{name} must be a 1-128 character visible ASCII id") + return value + + +def _require_sha256(value: Any, name: str) -> str: + if not isinstance(value, str) or _SHA256_RE.fullmatch(value) is None: + raise ReplayAuditWireError(f"{name} must be lowercase sha256 hex") + return value + + +def _require_register(value: Any, name: str) -> str: + if not isinstance(value, str) or _REGISTER_RE.fullmatch(value) is None: + raise ReplayAuditWireError(f"{name} must be lowercase 48-byte hex") + return value + + +def _require_f64(value: Any, name: str) -> float: + if not isinstance(value, str) or _F64_RE.fullmatch(value) is None: + raise ReplayAuditWireError(f"{name} must be lowercase binary64 hex") + try: + raw = bytes.fromhex(value) + score = struct.unpack(">d", raw)[0] + except (ValueError, TypeError): + raise ReplayAuditWireError(f"{name} must be lowercase binary64 hex") from None + if raw == b"\x80" + b"\0" * 7: + raise ReplayAuditWireError(f"{name} must not be negative zero") + if not math.isfinite(score) or not 0.0 <= score <= 1.0: + raise ReplayAuditWireError(f"{name} must be finite and in [0, 1]") + return score + + +def _require_positive_int(value: Any, name: str) -> int: + if not isinstance(value, int) or isinstance(value, bool) or value < 1: + raise ReplayAuditWireError(f"{name} must be a positive integer") + return value + + +def _canonical_json(value: Any) -> bytes: + try: + return json.dumps( + _normalize_canonical(value), + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + except (TypeError, ValueError, UnicodeEncodeError) as exc: + raise ReplayAuditWireError("immutable Eval plan is not canonical JSON") from exc + + +def _validate_raw_body( + raw_body: bytes, value: Mapping[str, Any], *, kind: str = "replay request" +) -> None: + """Reject duplicate/alternate JSON object representations at the HTTP seam.""" + + try: + parsed = json.loads( + raw_body.decode("utf-8"), + object_pairs_hook=_reject_duplicate_json_keys, + parse_constant=lambda constant: (_ for _ in ()).throw( + ReplayAuditWireError(f"unsupported JSON constant: {constant}") + ), + ) + except ReplayAuditWireError: + raise + except (UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ReplayAuditWireError(f"{kind} body is not valid UTF-8 JSON") from exc + if not isinstance(parsed, Mapping) or dict(parsed) != dict(value): + raise ReplayAuditWireError(f"{kind} body differs from parsed request") + + +def _reject_duplicate_json_keys(items: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, item in items: + if key in result: + raise ReplayAuditWireError(f"duplicate JSON key: {key!r}") + result[key] = item + return result + + +def parse_replay_json(raw_body: bytes | str) -> Any: + """Parse replay JSON while rejecting duplicate object keys.""" + + try: + if isinstance(raw_body, bytes): + raw_body = raw_body.decode("utf-8") + if not isinstance(raw_body, str): + raise TypeError("replay JSON must be bytes or text") + return json.loads( + raw_body, + object_pairs_hook=_reject_duplicate_json_keys, + parse_constant=lambda constant: (_ for _ in ()).throw( + ReplayAuditWireError(f"unsupported JSON constant: {constant}") + ), + ) + except ReplayAuditWireError: + raise + except (TypeError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ReplayAuditWireError("replay JSON is not valid UTF-8 JSON") from exc + + +def _normalize_canonical(value: Any) -> Any: + """Match agent-challenge's canonical_json_v1 profile exactly.""" + + if value is None or isinstance(value, bool) or isinstance(value, int): + return value + if isinstance(value, float): + raise ReplayAuditWireError("canonical Eval plan JSON forbids floats") + if isinstance(value, str): + if any(0xD800 <= ord(char) <= 0xDFFF for char in value): + raise ReplayAuditWireError("canonical Eval plan JSON forbids surrogates") + return unicodedata.normalize("NFC", value) + if isinstance(value, Mapping): + normalized: dict[str, Any] = {} + for key, item in value.items(): + if not isinstance(key, str): + raise ReplayAuditWireError( + "canonical Eval plan object keys must be strings" + ) + normalized_key = unicodedata.normalize("NFC", key) + if normalized_key in normalized: + raise ReplayAuditWireError("canonical Eval plan has duplicate keys") + normalized[normalized_key] = _normalize_canonical(item) + return normalized + if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)): + return [_normalize_canonical(item) for item in value] + raise ReplayAuditWireError("canonical Eval plan contains unsupported JSON value") + + +def _object(value: Any, name: str, fields: Sequence[str]) -> dict[str, Any]: + if not isinstance(value, Mapping): + raise ReplayAuditWireError(f"{name} must be an object") + actual = set(value) + expected = set(fields) + if actual != expected: + raise ReplayAuditWireError( + f"{name} has invalid fields: missing={sorted(expected - actual)}, " + f"unknown={sorted(actual - expected)}" + ) + return dict(value) + + +def _require_image(value: Any, name: str) -> str: + if not isinstance(value, str) or _IMAGE_RE.fullmatch(value) is None: + raise ReplayAuditWireError(f"{name} must be digest-pinned image reference") + return value + + +def _validate_scoring_policy(value: Any) -> dict[str, Any]: + policy = _object( + value, + "eval_plan.scoring_policy", + ( + "schema_version", + "per_task_aggregation", + "keep_policy", + "drop_lowest_n", + "threshold_f64be", + ), + ) + if policy["schema_version"] != 1: + raise ReplayAuditWireError("scoring policy schema_version must be 1") + if policy["per_task_aggregation"] not in {"mean", "best_of_k"}: + raise ReplayAuditWireError("invalid scoring policy aggregation") + if policy["keep_policy"] not in {"off", "drop_lowest_n", "threshold_band"}: + raise ReplayAuditWireError("invalid scoring policy keep_policy") + drop_lowest_n = policy["drop_lowest_n"] + if ( + not isinstance(drop_lowest_n, int) + or isinstance(drop_lowest_n, bool) + or drop_lowest_n < 0 + ): + raise ReplayAuditWireError( + "scoring policy drop_lowest_n must be non-negative integer" + ) + threshold = policy["threshold_f64be"] + if policy["keep_policy"] == "threshold_band": + _require_f64(threshold, "scoring policy threshold_f64be") + if drop_lowest_n != 0: + raise ReplayAuditWireError("threshold_band requires neutral drop_lowest_n") + elif threshold is not None: + raise ReplayAuditWireError( + "threshold_f64be must be null outside threshold_band" + ) + if policy["keep_policy"] != "drop_lowest_n" and drop_lowest_n != 0: + raise ReplayAuditWireError( + "drop_lowest_n must be neutral outside drop_lowest_n" + ) + return { + "schema_version": 1, + "per_task_aggregation": policy["per_task_aggregation"], + "keep_policy": policy["keep_policy"], + "drop_lowest_n": drop_lowest_n, + "threshold_f64be": threshold, + } + + +def scoring_policy_digest(policy: Mapping[str, Any]) -> str: + """Compute the digest over the complete canonical Scoring policy v1 bytes.""" + + return sha256(_canonical_json(_validate_scoring_policy(policy))).hexdigest() + + +def _validate_plan(value: Any) -> dict[str, Any]: + """Validate the complete, schema-closed Eval plan v1 independently.""" + + plan = _object( + value, + "eval_plan", + ( + "schema_version", + "eval_run_id", + "submission_id", + "submission_version", + "authorizing_review_digest", + "agent_hash", + "selected_tasks", + "k", + "scoring_policy", + "scoring_policy_digest", + "eval_app", + "key_release_endpoint", + "result_endpoint", + "key_release_nonce", + "score_nonce", + "run_token_sha256", + "issued_at_ms", + "expires_at_ms", + ), + ) + if plan["schema_version"] != 1: + raise ReplayAuditWireError("eval_plan schema_version must be 1") + eval_run_id = _require_id(plan["eval_run_id"], "eval_plan.eval_run_id") + submission_id = _require_id(plan["submission_id"], "eval_plan.submission_id") + submission_version = plan["submission_version"] + if ( + not isinstance(submission_version, int) + or isinstance(submission_version, bool) + or submission_version < 1 + ): + raise ReplayAuditWireError( + "eval_plan.submission_version must be positive integer" + ) + review_digest = _require_sha256( + plan["authorizing_review_digest"], "eval_plan.authorizing_review_digest" + ) + agent_hash = _require_sha256(plan["agent_hash"], "eval_plan.agent_hash") + k = _require_positive_int(plan["k"], "eval_plan.k") + policy = _validate_scoring_policy(plan["scoring_policy"]) + policy_digest = _require_sha256( + plan["scoring_policy_digest"], "eval_plan.scoring_policy_digest" + ) + if policy_digest != scoring_policy_digest(policy): + raise ReplayAuditWireError("scoring_policy_digest does not match policy bytes") + + selected_raw = plan["selected_tasks"] + if not isinstance(selected_raw, list) or not selected_raw: + raise ReplayAuditWireError("eval_plan.selected_tasks must be non-empty array") + selected: list[dict[str, str]] = [] + previous = "" + for item in selected_raw: + task = _object( + item, + "eval_plan.selected_tasks[]", + ("task_id", "image_ref", "task_config_sha256"), + ) + task_id = _require_id(task["task_id"], "selected_tasks[].task_id") + if previous >= task_id: + raise ReplayAuditWireError("selected_tasks must be sorted and unique") + previous = task_id + selected.append( + { + "task_id": task_id, + "image_ref": _require_image( + task["image_ref"], "selected_tasks[].image_ref" + ), + "task_config_sha256": _require_sha256( + task["task_config_sha256"], "selected_tasks[].task_config_sha256" + ), + } + ) + + app = _object( + plan["eval_app"], + "eval_plan.eval_app", + ( + "image_ref", + "compose_hash", + "app_identity", + "kms_key_algorithm", + "kms_public_key_hex", + "kms_public_key_sha256", + "measurement", + ), + ) + measurement = _object( + app["measurement"], + "eval_plan.eval_app.measurement", + ( + "mrtd", + "rtmr0", + "rtmr1", + "rtmr2", + "os_image_hash", + "key_provider", + "vm_shape", + ), + ) + public_key = _require_sha256( + app["kms_public_key_hex"], "eval_plan.eval_app.kms_public_key_hex" + ) + public_key_digest = _require_sha256( + app["kms_public_key_sha256"], "eval_plan.eval_app.kms_public_key_sha256" + ) + if sha256(bytes.fromhex(public_key)).hexdigest() != public_key_digest: + raise ReplayAuditWireError("kms_public_key_sha256 does not match public key") + if app["kms_key_algorithm"] != "x25519": + raise ReplayAuditWireError("eval_app.kms_key_algorithm must be x25519") + valid_measurement = { + "mrtd": _require_register(measurement["mrtd"], "measurement.mrtd"), + "rtmr0": _require_register(measurement["rtmr0"], "measurement.rtmr0"), + "rtmr1": _require_register(measurement["rtmr1"], "measurement.rtmr1"), + "rtmr2": _require_register(measurement["rtmr2"], "measurement.rtmr2"), + "os_image_hash": _require_sha256( + measurement["os_image_hash"], "measurement.os_image_hash" + ), + "key_provider": _require_id( + measurement["key_provider"], "measurement.key_provider" + ), + "vm_shape": _require_id(measurement["vm_shape"], "measurement.vm_shape"), + } + + key_nonce = _require_id(plan["key_release_nonce"], "eval_plan.key_release_nonce") + score_nonce = _require_id(plan["score_nonce"], "eval_plan.score_nonce") + if key_nonce == score_nonce: + raise ReplayAuditWireError("Eval plan nonces must be distinct") + expected_result_endpoint = f"/evaluation/v1/runs/{eval_run_id}/result" + if plan["result_endpoint"] != expected_result_endpoint: + raise ReplayAuditWireError("eval_plan.result_endpoint does not target run") + endpoint = plan["key_release_endpoint"] + if not isinstance(endpoint, str) or not endpoint or len(endpoint) > 16_384: + raise ReplayAuditWireError("eval_plan.key_release_endpoint is invalid") + for plan_field in ("issued_at_ms", "expires_at_ms"): + timestamp = plan[plan_field] + if ( + not isinstance(timestamp, int) + or isinstance(timestamp, bool) + or timestamp < 0 + ): + raise ReplayAuditWireError( + f"eval_plan.{plan_field} must be non-negative integer" + ) + if plan["expires_at_ms"] <= plan["issued_at_ms"]: + raise ReplayAuditWireError("eval_plan expiry must be after issue time") + + return { + "schema_version": 1, + "eval_run_id": eval_run_id, + "submission_id": submission_id, + "submission_version": submission_version, + "authorizing_review_digest": review_digest, + "agent_hash": agent_hash, + "selected_tasks": selected, + "k": k, + "scoring_policy": policy, + "scoring_policy_digest": policy_digest, + "eval_app": { + "image_ref": _require_image(app["image_ref"], "eval_app.image_ref"), + "compose_hash": _require_sha256( + app["compose_hash"], "eval_app.compose_hash" + ), + "app_identity": _require_id(app["app_identity"], "eval_app.app_identity"), + "kms_key_algorithm": "x25519", + "kms_public_key_hex": public_key, + "kms_public_key_sha256": public_key_digest, + "measurement": valid_measurement, + }, + "key_release_endpoint": endpoint, + "result_endpoint": expected_result_endpoint, + "key_release_nonce": key_nonce, + "score_nonce": score_nonce, + "run_token_sha256": _require_sha256( + plan["run_token_sha256"], "eval_plan.run_token_sha256" + ), + "issued_at_ms": plan["issued_at_ms"], + "expires_at_ms": plan["expires_at_ms"], + } + + +def plan_sha256(plan: Mapping[str, Any]) -> str: + """Digest the exact canonical bytes of a complete Eval plan v1.""" + + return sha256(_canonical_json(_validate_plan(plan))).hexdigest() + + +def _require_plan(value: Any) -> dict[str, Any]: + return _validate_plan(value) + + +def _replay_audit_id(eval_run_id: str, replay_attempt: int) -> str: + return f"replay:{eval_run_id}:{replay_attempt}" + + +@dataclass(frozen=True) +class ReplayAuditRequest: + """One labelled request for a full-plan legacy own_runner replay.""" + + audit_id: str + submission_id: str + eval_run_id: str + replay_attempt: int + plan_sha256: str + eval_plan: Mapping[str, Any] + k: int + selected_tasks: tuple[Mapping[str, Any], ...] + scoring_policy: Mapping[str, Any] + scoring_policy_digest: str + attested_score: float + raw_body: bytes | None = field(default=None, compare=False, repr=False) + + @classmethod + def from_mapping( + cls, value: Mapping[str, Any], *, raw_body: bytes | None = None + ) -> ReplayAuditRequest: + expected = { + "schema_version", + "audit_label", + "kind", + "audit_id", + "submission_id", + "eval_run_id", + "replay_attempt", + "plan_sha256", + "eval_plan", + "k", + "selected_tasks", + "scoring_policy", + "scoring_policy_digest", + "attested_score", + } + if not isinstance(value, Mapping) or set(value) != expected: + raise ReplayAuditWireError("replay request has unknown or missing fields") + if ( + value["schema_version"] != 1 + or value["audit_label"] != REPLAY_AUDIT_LABEL + or value["kind"] != REPLAY_AUDIT_REQUEST_KIND + ): + raise ReplayAuditWireError("replay request is not separately labelled") + audit_id = _require_id(value["audit_id"], "audit_id") + submission_id = _require_id(value["submission_id"], "submission_id") + eval_run_id = _require_id(value["eval_run_id"], "eval_run_id") + replay_attempt = _require_positive_int( + value["replay_attempt"], "replay_attempt" + ) + if audit_id != _replay_audit_id(eval_run_id, replay_attempt): + raise ReplayAuditWireError("audit_id does not match replay identity") + digest = _require_sha256(value["plan_sha256"], "plan_sha256") + plan = _require_plan(value["eval_plan"]) + if raw_body is not None: + _validate_raw_body(raw_body, value) + if plan["eval_run_id"] != eval_run_id: + raise ReplayAuditWireError("request/run Eval plan identity mismatch") + if plan["submission_id"] != submission_id: + raise ReplayAuditWireError("request/submission Eval plan identity mismatch") + if digest != plan_sha256(plan): + raise ReplayAuditWireError( + "plan_sha256 does not match canonical Eval plan bytes" + ) + k = _require_positive_int(value["k"], "k") + if k != plan["k"]: + raise ReplayAuditWireError("request k differs from immutable Eval plan") + selected = value["selected_tasks"] + if not isinstance(selected, list) or selected != plan["selected_tasks"]: + raise ReplayAuditWireError( + "selected task bytes differ from immutable Eval plan" + ) + policy = value["scoring_policy"] + if not isinstance(policy, Mapping) or dict(policy) != plan["scoring_policy"]: + raise ReplayAuditWireError( + "scoring policy bytes differ from immutable Eval plan" + ) + policy_digest = _require_sha256( + value["scoring_policy_digest"], "scoring_policy_digest" + ) + if policy_digest != scoring_policy_digest(policy): + raise ReplayAuditWireError( + "scoring_policy_digest does not match canonical policy bytes" + ) + if not isinstance(value["attested_score"], (int, float)) or isinstance( + value["attested_score"], bool + ): + raise ReplayAuditWireError("attested_score must be numeric") + if ( + not math.isfinite(float(value["attested_score"])) + or not 0.0 <= float(value["attested_score"]) <= 1.0 + ): + raise ReplayAuditWireError("attested_score must be finite in [0, 1]") + return cls( + audit_id=audit_id, + submission_id=submission_id, + eval_run_id=eval_run_id, + replay_attempt=replay_attempt, + plan_sha256=digest, + eval_plan=plan, + k=k, + selected_tasks=tuple(dict(item) for item in selected), + scoring_policy=dict(policy), + scoring_policy_digest=policy_digest, + attested_score=float(value["attested_score"]), + raw_body=raw_body, + ) + + def to_dict(self) -> dict[str, Any]: + """Return the labelled wire object without dropping plan/policy fields.""" + + result = { + "schema_version": 1, + "audit_label": REPLAY_AUDIT_LABEL, + "kind": REPLAY_AUDIT_REQUEST_KIND, + "audit_id": self.audit_id, + "submission_id": self.submission_id, + "eval_run_id": self.eval_run_id, + "replay_attempt": self.replay_attempt, + "plan_sha256": self.plan_sha256, + "eval_plan": dict(self.eval_plan), + "k": self.k, + "selected_tasks": [dict(item) for item in self.selected_tasks], + "scoring_policy": dict(self.scoring_policy), + "scoring_policy_digest": self.scoring_policy_digest, + "attested_score": self.attested_score, + } + return result + + @property + def work_unit_id(self) -> str: + return self.audit_id + + +@dataclass(frozen=True) +class ReplayAuditResult: + """Raw ordered replay trials returned to the challenge comparator.""" + + audit_id: str + submission_id: str + eval_run_id: str + replay_attempt: int + plan_sha256: str + trial_scores_by_task: Mapping[str, tuple[float, ...]] + raw_body: bytes | None = field(default=None, compare=False, repr=False) + + @classmethod + def from_mapping( + cls, value: Mapping[str, Any], *, raw_body: bytes | None = None + ) -> ReplayAuditResult: + expected = { + "schema_version", + "audit_label", + "kind", + "audit_id", + "submission_id", + "eval_run_id", + "replay_attempt", + "plan_sha256", + "trial_scores_by_task", + } + if not isinstance(value, Mapping) or set(value) != expected: + raise ReplayAuditWireError("replay result has unknown or missing fields") + if ( + value["schema_version"] != 1 + or value["audit_label"] != REPLAY_AUDIT_LABEL + or value["kind"] != REPLAY_AUDIT_RESULT_KIND + ): + raise ReplayAuditWireError("replay result is not separately labelled") + if raw_body is not None: + _validate_raw_body(raw_body, value, kind="replay result") + scores_by_task = value["trial_scores_by_task"] + if not isinstance(scores_by_task, Mapping) or not scores_by_task: + raise ReplayAuditWireError("replay result requires raw task trial scores") + parsed: dict[str, tuple[float, ...]] = {} + for task_id, scores in scores_by_task.items(): + _require_id(task_id, "trial task_id") + if ( + not isinstance(scores, Sequence) + or isinstance(scores, (str, bytes, bytearray)) + or not scores + ): + raise ReplayAuditWireError("each replay task requires ordered trials") + values: list[float] = [] + for score in scores: + if ( + not isinstance(score, (int, float)) + or isinstance(score, bool) + or not math.isfinite(float(score)) + or not 0.0 <= float(score) <= 1.0 + ): + raise ReplayAuditWireError( + "replay trial scores must be finite numbers" + ) + values.append(float(score)) + parsed[task_id] = tuple(values) + return cls( + audit_id=_require_id(value["audit_id"], "audit_id"), + submission_id=_require_id(value["submission_id"], "submission_id"), + eval_run_id=_require_id(value["eval_run_id"], "eval_run_id"), + replay_attempt=_require_positive_int( + value["replay_attempt"], "replay_attempt" + ), + plan_sha256=_require_sha256(value["plan_sha256"], "plan_sha256"), + trial_scores_by_task=parsed, + raw_body=raw_body, + ) + + def validate_against(self, request: ReplayAuditRequest) -> None: + # Revalidate the immutable request before comparing result identity. + ReplayAuditRequest.from_mapping(request.to_dict()) + if ( + self.audit_id != request.audit_id + or self.submission_id != request.submission_id + or self.eval_run_id != request.eval_run_id + or self.replay_attempt != request.replay_attempt + or self.plan_sha256 != request.plan_sha256 + ): + raise ReplayAuditWireError("replay result identity differs from request") + task_ids = [str(item["task_id"]) for item in request.selected_tasks] + if list(self.trial_scores_by_task) != task_ids: + raise ReplayAuditWireError( + "replay result task order differs from immutable plan" + ) + if any( + len(scores) != request.k for scores in self.trial_scores_by_task.values() + ): + raise ReplayAuditWireError( + "replay result trial count differs from immutable k" + ) + if set(self.trial_scores_by_task) != { + str(item["task_id"]) for item in request.selected_tasks + }: + raise ReplayAuditWireError( + "replay result task set differs from immutable plan" + ) + + def to_dict(self) -> dict[str, Any]: + return { + "schema_version": 1, + "audit_label": REPLAY_AUDIT_LABEL, + "kind": REPLAY_AUDIT_RESULT_KIND, + "audit_id": self.audit_id, + "submission_id": self.submission_id, + "eval_run_id": self.eval_run_id, + "replay_attempt": self.replay_attempt, + "plan_sha256": self.plan_sha256, + "trial_scores_by_task": { + task_id: list(scores) + for task_id, scores in self.trial_scores_by_task.items() + }, + } + + +def replay_assignment_payload(request: ReplayAuditRequest) -> dict[str, Any]: + """Build the only assignment payload that may invoke replay dispatch.""" + + # Validate at the assignment boundary as well as at HTTP ingestion. This + # prevents a forged/mutated in-memory request from becoming broker work. + validated = ReplayAuditRequest.from_mapping(request.to_dict()) + return { + "assignment_kind": REPLAY_AUDIT_ASSIGNMENT_KIND, + REPLAY_AUDIT_LABEL_KEY: REPLAY_AUDIT_LABEL, + REPLAY_AUDIT_REQUEST_KEY: validated.to_dict(), + } + + +def is_replay_assignment_payload(payload: Mapping[str, Any] | None) -> bool: + return ( + isinstance(payload, Mapping) + and payload.get("assignment_kind") == REPLAY_AUDIT_ASSIGNMENT_KIND + and payload.get(REPLAY_AUDIT_LABEL_KEY) == REPLAY_AUDIT_LABEL + and isinstance(payload.get(REPLAY_AUDIT_REQUEST_KEY), Mapping) + ) + + +__all__ = [ + "REPLAY_AUDIT_ASSIGNMENT_KIND", + "REPLAY_AUDIT_FORWARDED_KEY", + "REPLAY_AUDIT_LABEL", + "REPLAY_AUDIT_LABEL_KEY", + "REPLAY_AUDIT_REQUEST_KEY", + "REPLAY_AUDIT_RESULT_KEY", + "REPLAY_AUDIT_RESULT_KIND", + "REPLAY_AUDIT_REQUEST_KIND", + "ReplayAuditRequest", + "ReplayAuditResult", + "ReplayAuditWireError", + "is_replay_assignment_payload", + "parse_replay_json", + "plan_sha256", + "replay_assignment_payload", + "scoring_policy_digest", +] diff --git a/src/base/schemas/worker.py b/src/base/schemas/worker.py index 689f7c156..8b42af149 100644 --- a/src/base/schemas/worker.py +++ b/src/base/schemas/worker.py @@ -2,10 +2,18 @@ from __future__ import annotations +import json +from collections.abc import Mapping from datetime import datetime -from typing import Any +from typing import Any, Literal -from pydantic import BaseModel, Field +from pydantic import ( + AliasChoices, + BaseModel, + ConfigDict, + Field, + model_validator, +) class WorkerFaultView(BaseModel): @@ -137,6 +145,269 @@ class WorkerSignature(BaseModel): sig: str +#: Tier value marking an ExecutionProof as a Phala Intel TDX attestation +#: (architecture.md sec 6). Distinct from the integer worker-plane tiers so the +#: Phala envelope is self-describing without disturbing tier 0/1/2. +PHALA_TDX_TIER = "phala-tdx" + + +class PhalaMeasurement(BaseModel): + """TDX measurement registers for a canonical Phala eval image (arch sec 6/7). + + The static, allowlist-pinnable ``canonical_measurement`` is the subset + ``{mrtd, rtmr0, rtmr1, rtmr2, compose_hash, os_image_hash}``. ``rtmr3`` is the + runtime event-log register (it carries the live compose-hash event); it is + kept on the record for completeness but excluded from the pinned canonical + measurement bound into ``report_data``. + """ + + mrtd: str + rtmr0: str + rtmr1: str + rtmr2: str + rtmr3: str + compose_hash: str + os_image_hash: str + + def canonical(self) -> dict[str, str]: + """The static, allowlist-pinnable subset (excludes runtime ``rtmr3``).""" + + return { + "mrtd": self.mrtd, + "rtmr0": self.rtmr0, + "rtmr1": self.rtmr1, + "rtmr2": self.rtmr2, + "compose_hash": self.compose_hash, + "os_image_hash": self.os_image_hash, + } + + +class PhalaAttestation(BaseModel): + """Phala Intel TDX attestation payload carried by a Phala-tier ExecutionProof. + + Populates the ``attestation`` block of an ExecutionProof whose ``tier`` is + :data:`PHALA_TDX_TIER` (architecture.md sec 6). The architecture's + ``tdx_quote_b64`` / ``report_data_hex`` spellings are accepted as input + aliases; serialization always uses the canonical field names. + """ + + model_config = ConfigDict(populate_by_name=True) + + tdx_quote: str = Field(validation_alias=AliasChoices("tdx_quote", "tdx_quote_b64")) + event_log: list[dict[str, Any]] = Field(default_factory=list) + report_data: str = Field( + validation_alias=AliasChoices("report_data", "report_data_hex") + ) + measurement: PhalaMeasurement + vm_config: dict[str, Any] = Field(default_factory=dict) + + +_SHA256_PATTERN = r"^[0-9a-f]{64}$" +_REGISTER_PATTERN = r"^[0-9a-f]{96}$" +_REPORT_DATA_PATTERN = r"^[0-9a-f]{128}$" +_EVEN_HEX_PATTERN = r"^(?:[0-9a-f]{2})*$" +_NONEMPTY_EVEN_HEX_PATTERN = r"^(?:[0-9a-f]{2})+$" +_VISIBLE_ID_PATTERN = r"^[!-~]{1,128}$" + +# Eval result wire limits are deliberately fixed at the schema boundary. The +# direct result endpoint uses the same defaults, but the BASE conformance model +# must remain safe when it is used independently of that endpoint. +EVAL_MAX_QUOTE_BYTES = 64 * 1024 +EVAL_MAX_EVENT_LOG_ENTRIES = 4096 +EVAL_MAX_EVENT_LOG_BYTES = 2 * 1024 * 1024 +EVAL_MAX_VM_CONFIG_BYTES = 256 * 1024 +EVAL_MAX_STRING_BYTES = 16 * 1024 +EVAL_MAX_PAYLOAD_BYTES = EVAL_MAX_STRING_BYTES +EVAL_MAX_INTEGER = (1 << 63) - 1 + + +class EvalPhalaMeasurement(BaseModel): + """Exact canonical measurement wire schema for Eval Phala attestations.""" + + model_config = ConfigDict(extra="forbid", strict=True) + + mrtd: str = Field(pattern=_REGISTER_PATTERN) + rtmr0: str = Field(pattern=_REGISTER_PATTERN) + rtmr1: str = Field(pattern=_REGISTER_PATTERN) + rtmr2: str = Field(pattern=_REGISTER_PATTERN) + rtmr3: str = Field(pattern=_REGISTER_PATTERN) + compose_hash: str = Field(pattern=_SHA256_PATTERN) + os_image_hash: str = Field(pattern=_SHA256_PATTERN) + + def canonical(self) -> dict[str, str]: + """The static, allowlist-pinnable subset, excluding runtime ``rtmr3``.""" + + return { + "mrtd": self.mrtd, + "rtmr0": self.rtmr0, + "rtmr1": self.rtmr1, + "rtmr2": self.rtmr2, + "compose_hash": self.compose_hash, + "os_image_hash": self.os_image_hash, + } + + +class EvalPhalaEventLogEntry(BaseModel): + """One schema-closed event-log entry on the canonical Eval wire.""" + + model_config = ConfigDict(extra="forbid", strict=True) + + imr: int = Field(ge=0, le=EVAL_MAX_INTEGER) + event_type: int = Field(ge=0, le=EVAL_MAX_INTEGER) + digest: str = Field(pattern=_REGISTER_PATTERN) + event: str = Field(pattern=_VISIBLE_ID_PATTERN, max_length=EVAL_MAX_STRING_BYTES) + event_payload: str = Field( + pattern=_EVEN_HEX_PATTERN, + max_length=EVAL_MAX_PAYLOAD_BYTES, + ) + + +class EvalPhalaVmConfig(BaseModel): + """Evidence-only VM configuration carried in a canonical Eval attestation.""" + + model_config = ConfigDict(extra="forbid", strict=True) + + vcpu: int = Field(ge=1, le=EVAL_MAX_INTEGER) + memory_mb: int = Field(ge=1, le=EVAL_MAX_INTEGER) + os_image_hash: str | None = Field(pattern=_SHA256_PATTERN) + + +class EvalPhalaAttestation(BaseModel): + """Strict Eval Phala attestation boundary, with no legacy field aliases.""" + + model_config = ConfigDict(extra="forbid", strict=True) + + tdx_quote: str = Field( + pattern=_NONEMPTY_EVEN_HEX_PATTERN, + max_length=2 * EVAL_MAX_QUOTE_BYTES, + ) + event_log: list[EvalPhalaEventLogEntry] + report_data: str = Field(pattern=_REPORT_DATA_PATTERN) + measurement: EvalPhalaMeasurement + vm_config: EvalPhalaVmConfig + + @model_validator(mode="before") + @classmethod + def validate_transport_bounds(cls, value: Any) -> Any: + """Reject large nested transports before quote verification/allocation.""" + + if not isinstance(value, Mapping): + return value + event_log = value.get("event_log") + if isinstance(event_log, list): + if len(event_log) > EVAL_MAX_EVENT_LOG_ENTRIES: + raise ValueError("event_log exceeds its entry bound") + encoded_bytes = 2 + max(0, len(event_log) - 1) + for event in event_log: + if not isinstance(event, Mapping): + continue + if set(event) != { + "imr", + "event_type", + "digest", + "event", + "event_payload", + }: + raise ValueError("event_log entry has invalid fields") + for field, limit in ( + ("digest", len("a" * 96)), + ("event", EVAL_MAX_STRING_BYTES), + ("event_payload", EVAL_MAX_PAYLOAD_BYTES), + ): + field_value = event.get(field) + if isinstance(field_value, str) and len(field_value) > limit: + raise ValueError(f"event_log.{field} exceeds its string bound") + try: + encoded_bytes += len( + json.dumps( + event, + ensure_ascii=False, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ) + except (TypeError, ValueError, UnicodeEncodeError) as exc: + raise ValueError("event_log is not encodable") from exc + if encoded_bytes > EVAL_MAX_EVENT_LOG_BYTES: + raise ValueError("event_log exceeds its byte bound") + vm_config = value.get("vm_config") + if isinstance(vm_config, Mapping): + if set(vm_config) != {"vcpu", "memory_mb", "os_image_hash"}: + raise ValueError("vm_config has invalid fields") + os_image_hash = vm_config.get("os_image_hash") + if isinstance(os_image_hash, str) and len(os_image_hash) > 64: + raise ValueError("vm_config.os_image_hash exceeds its string bound") + try: + encoded = json.dumps( + vm_config, + ensure_ascii=False, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + except (TypeError, ValueError, UnicodeEncodeError) as exc: + raise ValueError("vm_config is not encodable") from exc + if len(encoded) > EVAL_MAX_VM_CONFIG_BYTES: + raise ValueError("vm_config exceeds its byte bound") + return value + + +class EvalWorkerSignature(BaseModel): + """The sole in-CVM worker-signature placeholder accepted on the Eval wire.""" + + model_config = ConfigDict(extra="forbid", strict=True) + + worker_pubkey: Literal[""] + sig: Literal[""] + + +class EvalExecutionProof(BaseModel): + """Schema-closed canonical Eval ``ExecutionProof`` wire envelope. + + This is intentionally separate from the permissive legacy + :class:`ExecutionProof` model. The direct Eval result endpoint validates this + model before replacing its exact empty signature placeholder with a + validator-owned signature. + """ + + model_config = ConfigDict(extra="forbid", strict=True) + + version: Literal[1] + tier: Literal["phala-tdx"] + manifest_sha256: str = Field(pattern=_SHA256_PATTERN) + image_digest: str = Field( + pattern=r"^[^@\s]+@sha256:[0-9a-f]{64}$", + max_length=EVAL_MAX_STRING_BYTES, + ) + provider: Literal[None] + worker_signature: EvalWorkerSignature + attestation: EvalPhalaAttestation + + def to_execution_proof(self) -> ExecutionProof: + """Convert validated canonical wire data to the legacy transport model.""" + + return ExecutionProof.model_validate(self.model_dump(mode="json")) + + +def _reject_duplicate_json_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + """JSON object hook that rejects duplicate member names before Pydantic.""" + + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON key: {key}") + result[key] = value + return result + + +def parse_eval_execution_proof_json( + data: str | bytes | bytearray, +) -> EvalExecutionProof: + """Parse a canonical Eval proof while rejecting duplicate JSON object keys.""" + + decoded = json.loads(data, object_pairs_hook=_reject_duplicate_json_keys) + return EvalExecutionProof.model_validate(decoded) + + class ExecutionProof(BaseModel): """Proof envelope attached to every worker result (architecture 3.4). @@ -145,11 +416,14 @@ class ExecutionProof(BaseModel): pinned message (``sha256(f"{manifest_sha256}:{unit_id}")``). Tier 1 adds ``image_digest`` + a populated ``provider`` block; tier 2 adds a non-null ``attestation``. The base worker plane emits tier 0; prism fills the higher - tiers. + tiers. The Phala TDX tier (``tier == PHALA_TDX_TIER``) carries a + :class:`PhalaAttestation` payload in ``attestation`` (architecture.md sec 6); + hence ``tier`` accepts the string Phala value in addition to the integer + worker-plane tiers. """ version: int = 1 - tier: int = 0 + tier: int | str = 0 manifest_sha256: str image_digest: str | None = None provider: ProviderInfo | None = None diff --git a/src/base/validator/agent/adapters/agent_challenge.py b/src/base/validator/agent/adapters/agent_challenge.py index 8c2bcc1f2..168f46597 100644 --- a/src/base/validator/agent/adapters/agent_challenge.py +++ b/src/base/validator/agent/adapters/agent_challenge.py @@ -12,12 +12,21 @@ from collections.abc import Awaitable, Callable, Mapping from typing import Any +from base.master.replay_audit import ( + REPLAY_AUDIT_REQUEST_KEY, + ReplayAuditRequest, + ReplayAuditResult, + is_replay_assignment_payload, +) +from base.schemas.worker import PHALA_TDX_TIER, ExecutionProof, WorkerSignature from base.validator.agent.executor import ( AssignmentContext, AssignmentExecutionError, ExecutionResult, ProgressCallback, ) +from base.validator.agent.signing import RequestSigner +from base.worker.proof import execution_proof_signing_payload CHALLENGE_SLUG = "agent-challenge" @@ -29,12 +38,20 @@ class AgentChallengeCycleExecutor: """Run a pulled agent-challenge assignment via the sibling validator cycle.""" - def __init__(self, *, dispatch: DispatchFn | None = None) -> None: + def __init__( + self, + *, + dispatch: DispatchFn | None = None, + dispatch_replay: DispatchFn | None = None, + ) -> None: self._dispatch = dispatch + self._dispatch_replay = dispatch_replay async def execute( self, context: AssignmentContext, *, progress: ProgressCallback ) -> ExecutionResult: + if is_replay_assignment_payload(context.assignment.payload): + return await self._execute_replay(context) dispatch = self._dispatch or _load_dispatch() broker = context.broker result = await dispatch( @@ -47,6 +64,36 @@ async def execute( ) return ExecutionResult(success=True, payload=dict(result)) + async def _execute_replay(self, context: AssignmentContext) -> ExecutionResult: + payload = context.assignment.payload + raw_request = payload.get(REPLAY_AUDIT_REQUEST_KEY) + if not isinstance(raw_request, Mapping): + raise AssignmentExecutionError("replay assignment has no labelled request") + request = ReplayAuditRequest.from_mapping(raw_request) + dispatch = self._dispatch_replay or _load_replay_dispatch() + broker = context.broker + result = await dispatch( + request=request.to_dict(), + work_unit_id=context.assignment.work_unit_id, + payload=dict(payload), + broker_url=broker.broker_url, + broker_token=broker.broker_token, + broker_token_file=broker.broker_token_file, + broker_allowed_images=tuple(broker.allowed_images), + ) + if not isinstance(result, Mapping): + raise AssignmentExecutionError( + "replay dispatch returned a non-object result" + ) + replay_result = ReplayAuditResult.from_mapping( + result.get("replay_audit_result", result) + ) + replay_result.validate_against(request) + return ExecutionResult( + success=True, + payload={"replay_audit_result": replay_result.to_dict()}, + ) + def _load_dispatch() -> DispatchFn: try: @@ -58,4 +105,59 @@ def _load_dispatch() -> DispatchFn: return dispatch_assignment -__all__ = ["CHALLENGE_SLUG", "AgentChallengeCycleExecutor"] +def _load_replay_dispatch() -> DispatchFn: + try: + from agent_challenge.validator_dispatch import dispatch_replay_audit + except Exception as exc: # noqa: BLE001 - surfaced as a dispatch failure + raise AssignmentExecutionError( + f"agent-challenge replay dispatch adapter is unavailable: {exc}" + ) from exc + return dispatch_replay_audit + + +def rebind_worker_signature( + proof: ExecutionProof, *, signer: RequestSigner, unit_id: str +) -> ExecutionProof: + """Rebind a Phala-tier envelope's tier-0 worker signature to ``unit_id``. + + The canonical eval image runs a LEAN CVM image with no bittensor/sr25519 + keypair, so its emitted Phala-tier ``ExecutionProof`` carries only a + schema-valid PLACEHOLDER ``worker_signature`` (empty pubkey/sig). When the + validator ingests the attested ``BASE_BENCHMARK_RESULT`` payload it re-signs + the tier-0 layer over the pinned ``sha256(f"{manifest_sha256}:{unit_id}")`` + message with its OWN signer, so ``verify_execution_proof`` enforces a real + signature bound to this unit (no cross-unit replay, VAL-VERIFY-013). + + The trust root remains the **attestation** (the hardware-signed TDX quote), + which the validator verifies cryptographically; this rebind only anchors the + worker-plane envelope layer to a real key -- it never substitutes for quote + verification. The attestation payload is carried through unchanged. + """ + + if proof.tier != PHALA_TDX_TIER: + raise AssignmentExecutionError( + "rebind_worker_signature only applies to Phala-tier proofs" + ) + if proof.worker_signature.worker_pubkey != "" or proof.worker_signature.sig != "": + raise AssignmentExecutionError( + "rebind_worker_signature requires the exact empty Eval placeholder" + ) + signature = signer.sign( + execution_proof_signing_payload( + manifest_sha256=proof.manifest_sha256, unit_id=unit_id + ) + ) + return proof.model_copy( + update={ + "worker_signature": WorkerSignature( + worker_pubkey=signer.hotkey, sig=signature + ) + } + ) + + +__all__ = [ + "CHALLENGE_SLUG", + "AgentChallengeCycleExecutor", + "rebind_worker_signature", +] diff --git a/src/base/worker/phala_quote.py b/src/base/worker/phala_quote.py new file mode 100644 index 000000000..09d521719 --- /dev/null +++ b/src/base/worker/phala_quote.py @@ -0,0 +1,538 @@ +"""TDX quote parsing, event-log replay, and signature/TCB verification (M4). + +The validator/master Phala-tier verifier (``verify_execution_proof``) must, before +accepting an attested result, confirm that its Intel TDX quote: + +1. is cryptographically valid on genuine hardware with an acceptable TCB posture + (delegated to a :class:`QuoteVerifier` -- ``dcap-qvl`` / Phala verify); and +2. carries the expected measurement registers + ``report_data`` (parsed here + structurally from the hardware-signed TD report); and +3. has an RTMR3 that a replay of its event log reproduces, yielding the canonical + ``compose_hash`` (so the compose is bound by content, not trusted by value). + +Structural parsing (register/report_data offsets) and the dstack ``cc-eventlog`` +RTMR replay follow the same hardware/dstack facts the agent-challenge in-CVM +key-release path uses, so a live dstack event log verifies here unmodified. The +crucial base-side addition is the **park vs reject** distinction: a *cryptographic* +failure raises :class:`QuoteVerificationError` (the verifier returns False), while +a *transient* dependency outage/timeout raises :class:`VerifierUnavailableError` +so the caller PARKS the result (never accepts, never fraud-rejects) -- VAL-VERIFY-014. +""" + +from __future__ import annotations + +import hashlib +import json +import subprocess +from collections.abc import Callable, Iterable, Mapping, Sequence +from dataclasses import dataclass, field +from typing import Any, Protocol, runtime_checkable + +# --------------------------------------------------------------------------- # +# TDX quote v4 (ECDSA) layout: 48-byte header + TD report (TDREPORT10) +# --------------------------------------------------------------------------- # +#: Quote header length (bytes) preceding the TD report body. +QUOTE_HEADER_LEN = 48 +#: SHA-384 measurement register width (bytes). +REGISTER_LEN = 48 +#: TDX ``report_data`` field width (bytes). +REPORT_DATA_LEN = 64 + +# Absolute byte offsets of the fields within a v4 quote (header + TD report body). +_MRTD_OFFSET = QUOTE_HEADER_LEN + 136 +_RTMR0_OFFSET = QUOTE_HEADER_LEN + 328 +_RTMR1_OFFSET = QUOTE_HEADER_LEN + 376 +_RTMR2_OFFSET = QUOTE_HEADER_LEN + 424 +_RTMR3_OFFSET = QUOTE_HEADER_LEN + 472 +_REPORT_DATA_OFFSET = QUOTE_HEADER_LEN + 520 + +#: Minimum length (bytes) a quote must have to contain the full TD report. +MIN_QUOTE_LEN = _REPORT_DATA_OFFSET + REPORT_DATA_LEN + +# --------------------------------------------------------------------------- # +# dstack event-log constants (cc-eventlog) +# --------------------------------------------------------------------------- # +#: dstack runtime event type (RTMR3 app events). Not a TCG-defined value. +DSTACK_RUNTIME_EVENT_TYPE = 0x08000001 +#: RTMR3 event name carrying the normalized compose hash. +COMPOSE_HASH_EVENT = "compose-hash" +#: RTMR3 event name carrying the KMS key-provider identity. +KEY_PROVIDER_EVENT = "key-provider" +#: The RTMR index whose event log binds the app compose (RTMR3). +APP_IMR = 3 + + +class QuoteError(Exception): + """Base error for quote parsing / verification failures (fail closed).""" + + +class QuoteStructureError(QuoteError): + """The quote bytes are malformed / too short to contain a TD report.""" + + +class QuoteVerificationError(QuoteError): + """The quote is cryptographically invalid, or its event log is inconsistent. + + A *cryptographic* verdict: the caller should REJECT the result (verify False). + """ + + +class VerifierUnavailableError(QuoteError): + """The verification dependency is transiently unreachable / timed out. + + NOT a cryptographic verdict: the caller should PARK the result (retry later), + never accept it and never permanently fraud-reject it (VAL-VERIFY-014). + """ + + +@dataclass(frozen=True) +class TdReport: + """Measurement registers + ``report_data`` read from a TDX quote's TD report.""" + + mrtd: str + rtmr0: str + rtmr1: str + rtmr2: str + rtmr3: str + report_data: bytes + + +def _coerce_register(value: bytes | str, *, field_name: str) -> bytes: + if isinstance(value, str): + try: + raw = bytes.fromhex(value) + except ValueError as exc: + raise ValueError(f"{field_name} is not valid hex: {exc}") from exc + elif isinstance(value, bytes | bytearray): + raw = bytes(value) + else: + raise TypeError( + f"{field_name} must be hex str or bytes, not {type(value).__name__}" + ) + if len(raw) != REGISTER_LEN: + raise ValueError(f"{field_name} must be {REGISTER_LEN} bytes, got {len(raw)}") + return raw + + +def parse_quote_hex(quote_hex: str) -> bytes: + """Decode a hex quote string to bytes; fail closed on malformed hex.""" + + if not isinstance(quote_hex, str) or not quote_hex: + raise QuoteStructureError("quote must be a non-empty hex string") + text = quote_hex.strip() + if text.startswith(("0x", "0X")): + text = text[2:] + try: + return bytes.fromhex(text) + except ValueError as exc: + raise QuoteStructureError(f"quote is not valid hex: {exc}") from exc + + +def parse_td_report(quote: bytes | str) -> TdReport: + """Parse the measurement registers + ``report_data`` from a TDX quote. + + Reads MRTD/RTMR0-3 (SHA-384) and the 64-byte ``report_data`` at their fixed + v4-quote offsets. Raises :class:`QuoteStructureError` if the quote is too + short to contain a full TD report. + """ + + raw = parse_quote_hex(quote) if isinstance(quote, str) else bytes(quote) + if len(raw) < MIN_QUOTE_LEN: + raise QuoteStructureError( + f"quote is {len(raw)} bytes, need at least {MIN_QUOTE_LEN} for a TD report" + ) + + def _reg(offset: int) -> str: + return raw[offset : offset + REGISTER_LEN].hex() + + return TdReport( + mrtd=_reg(_MRTD_OFFSET), + rtmr0=_reg(_RTMR0_OFFSET), + rtmr1=_reg(_RTMR1_OFFSET), + rtmr2=_reg(_RTMR2_OFFSET), + rtmr3=_reg(_RTMR3_OFFSET), + report_data=raw[_REPORT_DATA_OFFSET : _REPORT_DATA_OFFSET + REPORT_DATA_LEN], + ) + + +def os_image_hash_from_registers(mrtd: str, rtmr1: str, rtmr2: str) -> str: + """The dstack ``mr_image`` OS identity: ``sha256(MRTD ∥ RTMR1 ∥ RTMR2)``. + + Matches the canonical eval image's ``os_image_hash`` (dstack ``mr_image``) so + a quote's registers reproduce the value a validator pins in the allowlist. + """ + + preimage = ( + _coerce_register(mrtd, field_name="mrtd") + + _coerce_register(rtmr1, field_name="rtmr1") + + _coerce_register(rtmr2, field_name="rtmr2") + ) + return hashlib.sha256(preimage).hexdigest() + + +# --------------------------------------------------------------------------- # +# Event-log replay (RTMR3) +# --------------------------------------------------------------------------- # +def runtime_event_digest(event_name: str, payload: bytes) -> bytes: + """SHA-384 digest of a dstack runtime (RTMR3) event (cc-eventlog format). + + ``SHA384( event_type(le32) ∥ b":" ∥ event_name ∥ b":" ∥ payload )`` with + ``event_type == DSTACK_RUNTIME_EVENT_TYPE``. Binding the digest to the payload + means a forged compose-hash payload cannot keep a matching RTMR3. + """ + + hasher = hashlib.sha384() + hasher.update(DSTACK_RUNTIME_EVENT_TYPE.to_bytes(4, "little")) + hasher.update(b":") + hasher.update(event_name.encode("utf-8")) + hasher.update(b":") + hasher.update(payload) + return hasher.digest() + + +def _rtmr_extend(mr: bytes, digest: bytes) -> bytes: + return hashlib.sha384(mr + digest).digest() + + +@dataclass(frozen=True) +class Rtmr3Replay: + """Result of replaying an event log into RTMR3.""" + + rtmr3: str + compose_hash: str | None + key_provider: str | None + + +def _payload_bytes(entry: Mapping[str, Any]) -> bytes: + payload = entry.get("event_payload", "") + if isinstance(payload, bytes | bytearray): + return bytes(payload) + if not isinstance(payload, str): + raise QuoteVerificationError("event_payload must be a hex string or bytes") + if payload == "": + return b"" + try: + return bytes.fromhex(payload) + except ValueError as exc: + raise QuoteVerificationError(f"event_payload is not valid hex: {exc}") from exc + + +def replay_rtmr3(event_log: Iterable[Mapping[str, Any]]) -> Rtmr3Replay: + """Replay the RTMR3 (``imr == 3``) events, binding each digest to its payload. + + For dstack runtime events the digest is recomputed from the event contents + and must equal the logged digest (else the log is inconsistent and rejected), + then folded ``RTMR = SHA384(RTMR || digest)``. The ``compose-hash`` and + ``key-provider`` event payloads are surfaced for the allowlist check. + """ + + mr = bytes(REGISTER_LEN) + compose_hash: str | None = None + key_provider: str | None = None + + for entry in event_log: + if not isinstance(entry, Mapping): + raise QuoteVerificationError("event log entries must be objects") + if entry.get("imr") != APP_IMR: + continue + event_name = entry.get("event", "") + if not isinstance(event_name, str): + raise QuoteVerificationError("event 'event' name must be a string") + payload = _payload_bytes(entry) + event_type = entry.get("event_type") + + if event_type == DSTACK_RUNTIME_EVENT_TYPE: + digest = runtime_event_digest(event_name, payload) + logged = entry.get("digest") + if isinstance(logged, str) and logged: + try: + logged_bytes = bytes.fromhex(logged) + except ValueError as exc: + raise QuoteVerificationError( + f"event digest is not valid hex: {exc}" + ) from exc + if logged_bytes != digest: + raise QuoteVerificationError( + f"event '{event_name}' digest does not match its payload" + ) + else: + logged = entry.get("digest") + if not isinstance(logged, str) or not logged: + raise QuoteVerificationError( + "non-runtime RTMR3 event is missing a digest" + ) + try: + digest = bytes.fromhex(logged) + except ValueError as exc: + raise QuoteVerificationError( + f"event digest is not valid hex: {exc}" + ) from exc + if len(digest) != REGISTER_LEN: + raise QuoteVerificationError("RTMR3 event digest must be 48 bytes") + + mr = _rtmr_extend(mr, digest) + if event_name == COMPOSE_HASH_EVENT: + compose_hash = payload.hex() + elif event_name == KEY_PROVIDER_EVENT: + key_provider = payload.hex() + + return Rtmr3Replay( + rtmr3=mr.hex(), compose_hash=compose_hash, key_provider=key_provider + ) + + +# --------------------------------------------------------------------------- # +# Cryptographic verification (signature + TCB) +# --------------------------------------------------------------------------- # +@dataclass(frozen=True) +class QuoteVerdict: + """A verified quote's TCB posture (its bytes are Intel-signed / authentic).""" + + tcb_status: str + advisory_ids: tuple[str, ...] = () + + +@runtime_checkable +class QuoteVerifier(Protocol): + """Verifies a TDX quote's DCAP signature + certificate chain. + + Returns a :class:`QuoteVerdict` (carrying the TCB status) when the quote is + cryptographically valid, raises :class:`QuoteVerificationError` when it is not + (invalid signature / broken cert chain / malformed), and raises + :class:`VerifierUnavailableError` when the verification dependency itself is + transiently unreachable (so the caller parks rather than rejects). + """ + + def verify(self, quote_hex: str) -> QuoteVerdict: # pragma: no cover - protocol + ... + + +@dataclass +class DcapQvlVerifier: + """Trustless quote verification via the ``dcap-qvl`` CLI (Intel PCS). + + ``dcap-qvl`` verifies the quote against Intel collateral and reports the TCB + status; this adapter shells out and parses that verdict. ``runner`` is + injectable for testing. The accept / reject / park mapping is deliberate + (VAL-VERIFY-014): + + * a **non-zero exit** is a cryptographic verdict -- the tool judged the quote + invalid / its TCB unacceptable -- so it PERMANENTLY rejects + (:class:`QuoteVerificationError`); and + * a **timeout / missing binary / subprocess error** is a transient outage, so + it PARKS (:class:`VerifierUnavailableError`, retryable); and + * an **exit-0-but-unparseable / non-object / missing-TCB-status** output is a + *tooling* regression (dcap-qvl accepted the quote's cryptography but changed + its stdout format), NOT a fraud verdict -- so it PARKS + (:class:`VerifierUnavailableError`) rather than permanently fraud-rejecting a + legitimate result. It is never accepted (no verdict is returned). + """ + + binary: str = "dcap-qvl" + timeout: float = 30.0 + runner: Callable[[list[str]], subprocess.CompletedProcess[str]] | None = None + + def _run(self, args: list[str]) -> subprocess.CompletedProcess[str]: + if self.runner is not None: + return self.runner(args) + return subprocess.run( # pragma: no cover - real CLI invoked live (M6) + args, capture_output=True, text=True, timeout=self.timeout + ) + + def verify(self, quote_hex: str) -> QuoteVerdict: + import tempfile + from pathlib import Path + + # dcap-qvl's third argument is a *file path*, not the hex body. + # Live TDX quotes are ~10k hex chars; passing them as argv raises + # "File name too long (os error 36)" and collapses to permanent reject. + if not isinstance(quote_hex, str) or not quote_hex: + raise QuoteVerificationError("quote_hex must be a non-empty hex string") + quote_path: Path | None = None + try: + with tempfile.NamedTemporaryFile( + mode="w", + encoding="ascii", + prefix="dcap-qvl-quote-", + suffix=".hex", + delete=False, + ) as handle: + handle.write(quote_hex) + quote_path = Path(handle.name) + args = [self.binary, "verify", "--hex", str(quote_path)] + try: + proc = self._run(args) + except FileNotFoundError as exc: + raise VerifierUnavailableError( + f"dcap-qvl not available: {exc}" + ) from exc + except subprocess.TimeoutExpired as exc: + raise VerifierUnavailableError(f"dcap-qvl timed out: {exc}") from exc + except subprocess.SubprocessError as exc: + raise VerifierUnavailableError( + f"dcap-qvl invocation failed: {exc}" + ) from exc + finally: + if quote_path is not None: + try: + quote_path.unlink(missing_ok=True) + except OSError: # pragma: no cover - best-effort cleanup + pass + + if proc.returncode != 0: + detail = (proc.stderr or proc.stdout or "").strip() + raise QuoteVerificationError(f"dcap-qvl rejected the quote: {detail}") + + try: + report = json.loads(proc.stdout) + except json.JSONDecodeError as exc: + raise VerifierUnavailableError( + f"dcap-qvl exited 0 but its output is not JSON " + f"(tooling regression -- park, do not reject): {exc}" + ) from exc + if not isinstance(report, Mapping): + raise VerifierUnavailableError( + "dcap-qvl exited 0 but its output was not a JSON object " + "(tooling regression -- park, do not reject)" + ) + + status = ( + report.get("status") or report.get("tcbStatus") or report.get("tcb_status") + ) + if not isinstance(status, str) or not status: + raise VerifierUnavailableError( + "dcap-qvl exited 0 but its output is missing a TCB status " + "(tooling regression -- park, do not reject)" + ) + advisories = report.get("advisory_ids") or report.get("advisoryIDs") or [] + if not isinstance(advisories, Sequence) or isinstance(advisories, str | bytes): + advisories = [] + return QuoteVerdict( + tcb_status=status, advisory_ids=tuple(str(a) for a in advisories) + ) + + +@dataclass(frozen=True) +class StaticQuoteVerifier: + """A :class:`QuoteVerifier` with a fixed verdict (tests / offline harness). + + ``tcb_status`` is the posture reported for any quote; ``valid=False`` rejects + every quote as if its signature did not verify (:class:`QuoteVerificationError`); + ``unavailable=True`` models a transient verifier outage + (:class:`VerifierUnavailableError`, park). + """ + + tcb_status: str = "UpToDate" + valid: bool = True + unavailable: bool = False + advisory_ids: tuple[str, ...] = field(default_factory=tuple) + + def verify(self, quote_hex: str) -> QuoteVerdict: + if self.unavailable: + raise VerifierUnavailableError("quote verifier is unavailable") + if not self.valid: + raise QuoteVerificationError("quote signature verification failed") + return QuoteVerdict(tcb_status=self.tcb_status, advisory_ids=self.advisory_ids) + + +# --------------------------------------------------------------------------- # +# Quote / event-log assembly (tooling + test-vector generation) +# --------------------------------------------------------------------------- # +def build_tdx_quote( + *, + mrtd: bytes | str, + rtmr0: bytes | str, + rtmr1: bytes | str, + rtmr2: bytes | str, + rtmr3: bytes | str, + report_data: bytes | str, + header: bytes = b"", + tail: bytes = b"", +) -> str: + """Assemble a minimal v4-layout TDX quote (hex) with the given fields. + + The inverse of :func:`parse_td_report`: places each register + ``report_data`` + at its fixed offset. Used to generate deterministic test vectors / fixtures + (the real quote is produced by dstack ``get_quote`` on a live CVM). + """ + + buf = bytearray(MIN_QUOTE_LEN) + if header: + buf[: min(len(header), QUOTE_HEADER_LEN)] = header[:QUOTE_HEADER_LEN] + buf[_MRTD_OFFSET : _MRTD_OFFSET + REGISTER_LEN] = _coerce_register( + mrtd, field_name="mrtd" + ) + buf[_RTMR0_OFFSET : _RTMR0_OFFSET + REGISTER_LEN] = _coerce_register( + rtmr0, field_name="rtmr0" + ) + buf[_RTMR1_OFFSET : _RTMR1_OFFSET + REGISTER_LEN] = _coerce_register( + rtmr1, field_name="rtmr1" + ) + buf[_RTMR2_OFFSET : _RTMR2_OFFSET + REGISTER_LEN] = _coerce_register( + rtmr2, field_name="rtmr2" + ) + buf[_RTMR3_OFFSET : _RTMR3_OFFSET + REGISTER_LEN] = _coerce_register( + rtmr3, field_name="rtmr3" + ) + if isinstance(report_data, str): + rd = bytes.fromhex(report_data) + else: + rd = bytes(report_data) + rd = rd[:REPORT_DATA_LEN].ljust(REPORT_DATA_LEN, b"\x00") + buf[_REPORT_DATA_OFFSET : _REPORT_DATA_OFFSET + REPORT_DATA_LEN] = rd + return (bytes(buf) + tail).hex() + + +def build_rtmr3_event_log( + events: Sequence[tuple[str, bytes]], +) -> tuple[list[dict[str, Any]], str]: + """Build a consistent RTMR3 event log + its replayed RTMR3 hex. + + ``events`` is an ordered sequence of ``(event_name, payload_bytes)``; each is + emitted as a dstack runtime event with the correctly derived digest, and the + folded RTMR3 is returned so a caller can pin it into a matching quote. + """ + + log: list[dict[str, Any]] = [] + mr = bytes(REGISTER_LEN) + for name, payload in events: + digest = runtime_event_digest(name, payload) + log.append( + { + "imr": APP_IMR, + "event_type": DSTACK_RUNTIME_EVENT_TYPE, + "digest": digest.hex(), + "event": name, + "event_payload": payload.hex(), + } + ) + mr = _rtmr_extend(mr, digest) + return log, mr.hex() + + +__all__ = [ + "APP_IMR", + "COMPOSE_HASH_EVENT", + "DSTACK_RUNTIME_EVENT_TYPE", + "KEY_PROVIDER_EVENT", + "MIN_QUOTE_LEN", + "REGISTER_LEN", + "REPORT_DATA_LEN", + "DcapQvlVerifier", + "QuoteError", + "QuoteStructureError", + "QuoteVerdict", + "QuoteVerificationError", + "QuoteVerifier", + "Rtmr3Replay", + "StaticQuoteVerifier", + "TdReport", + "VerifierUnavailableError", + "build_rtmr3_event_log", + "build_tdx_quote", + "os_image_hash_from_registers", + "parse_quote_hex", + "parse_td_report", + "replay_rtmr3", + "runtime_event_digest", +] diff --git a/src/base/worker/phala_verify.py b/src/base/worker/phala_verify.py new file mode 100644 index 000000000..73f1a2f99 --- /dev/null +++ b/src/base/worker/phala_verify.py @@ -0,0 +1,251 @@ +"""Validator-owned measurement allowlist, nonce freshness, and run binding (M4). + +Supporting types for the Phala-tier verifier in +:func:`base.worker.proof.verify_execution_proof` (architecture.md sec 4 C4 / 6 / 7): + +* :class:`MeasurementAllowlist` -- the **validator-owned** set of canonical + measurements a genuine quote must match. Membership (not mere quote validity) + governs acceptance; an EMPTY allowlist fails closed (matches nothing), never + accept-any. No requester-supplied value can widen it. +* :class:`NonceValidator` / :class:`InMemoryNonceValidator` -- validator-issued, + single-use, TTL-bounded nonces. The nonce bound into a quote's ``report_data`` + must be one the validator issued and has not consumed/expired, defeating + quote replay and cross-submission repurposing. +* :class:`PhalaBinding` -- the run identity the VALIDATOR expects for a submission + (agent_hash, task_ids, scores_digest, validator_nonce) that ``report_data`` must + bind. It is the validator's own record, never trusted from the attested payload. +""" + +from __future__ import annotations + +import json +import os +import secrets +import time +from collections.abc import Callable, Iterable, Mapping +from dataclasses import dataclass, field +from enum import StrEnum +from pathlib import Path +from typing import Protocol, runtime_checkable + +from base.schemas.worker import PhalaMeasurement + +#: The static, allowlist-pinnable measurement fields (excludes runtime ``rtmr3``). +CANONICAL_MEASUREMENT_FIELDS: tuple[str, ...] = ( + "mrtd", + "rtmr0", + "rtmr1", + "rtmr2", + "compose_hash", + "os_image_hash", +) + +#: Env var naming a JSON file with the validator's canonical measurement +#: allowlist. Unset/empty ⇒ an unconfigured validator ⇒ empty (fail-closed). +MEASUREMENT_ALLOWLIST_FILE_ENV = "BASE_PHALA_MEASUREMENT_ALLOWLIST_FILE" + + +def canonical_measurement_mapping( + measurement: PhalaMeasurement | Mapping[str, object], +) -> dict[str, str]: + """The static canonical subset of a measurement as a ``{field: str}`` mapping.""" + + if isinstance(measurement, PhalaMeasurement): + return measurement.canonical() + return {field: str(measurement[field]) for field in CANONICAL_MEASUREMENT_FIELDS} + + +@dataclass(frozen=True) +class MeasurementAllowlist: + """A validator-owned set of canonical measurements a quote must match. + + Matching is exact across ALL canonical registers. The allowlist can hold + MORE THAN ONE entry so that during a canonical-image rotation both the + outgoing and incoming measurements are trusted simultaneously -- a quote + matching ANY entry passes, one matching none is rejected. An empty allowlist + matches nothing (fail closed) -- an unconfigured validator never accepts a + quote, and every load path below fails closed (to empty) on a missing, + unreadable, or unparseable source rather than defaulting to accept-any. + """ + + entries: tuple[dict[str, str], ...] = () + + @classmethod + def from_measurements( + cls, measurements: Iterable[PhalaMeasurement | Mapping[str, object]] + ) -> MeasurementAllowlist: + return cls(tuple(canonical_measurement_mapping(m) for m in measurements)) + + @classmethod + def from_json(cls, text: str) -> MeasurementAllowlist: + """Parse a JSON allowlist, FAILING CLOSED (empty) on any malformed input. + + Accepts either a bare ``[entry, ...]`` list or a ``{"entries": [...]}`` + object. Invalid JSON, an unexpected top-level shape, a non-mapping entry, + or an entry missing a canonical register yields an EMPTY allowlist (which + rejects everything) -- never an accept-any allowlist and never an + exception a caller might mistake for success (VAL-VERIFY-025). + """ + + try: + data = json.loads(text) + except (json.JSONDecodeError, TypeError, ValueError): + return cls() + if isinstance(data, Mapping): + data = data.get("entries", []) + if not isinstance(data, list): + return cls() + entries: list[dict[str, str]] = [] + for item in data: + if not isinstance(item, Mapping): + return cls() + try: + entries.append(canonical_measurement_mapping(item)) + except (KeyError, TypeError): + return cls() + return cls(tuple(entries)) + + @classmethod + def from_file(cls, path: str | Path) -> MeasurementAllowlist: + """Load an allowlist from a JSON file, FAILING CLOSED on any I/O error. + + A missing or unreadable file yields an EMPTY allowlist (fail closed) + rather than raising or accepting anything. + """ + + file_path = Path(path) + try: + text = file_path.read_text(encoding="utf-8") + except OSError: + return cls() + return cls.from_json(text) + + @classmethod + def from_env(cls, env: Mapping[str, str] | None = None) -> MeasurementAllowlist: + """Load the allowlist named by :data:`MEASUREMENT_ALLOWLIST_FILE_ENV`. + + When the env var is unset/empty the validator is UNCONFIGURED, which + fails closed to an empty allowlist (accepts nothing). + """ + + environ = os.environ if env is None else env + path = environ.get(MEASUREMENT_ALLOWLIST_FILE_ENV) + if not path: + return cls() + return cls.from_file(path) + + def __bool__(self) -> bool: + return bool(self.entries) + + def contains(self, measurement: PhalaMeasurement | Mapping[str, object]) -> bool: + """Whether ``measurement`` exactly matches a canonical allowlist entry.""" + + candidate = canonical_measurement_mapping(measurement) + return any(candidate == entry for entry in self.entries) + + +class NonceState(StrEnum): + """Outcome of consuming a validator nonce.""" + + OK = "ok" + UNKNOWN = "unknown" + EXPIRED = "expired" + CONSUMED = "consumed" + + +@runtime_checkable +class NonceValidator(Protocol): + """Consumes a validator-issued nonce, reporting its freshness state. + + ``consume`` is single-use: the first consume of a known, unexpired nonce + returns :attr:`NonceState.OK` and marks it consumed; any later consume of the + same nonce returns :attr:`NonceState.CONSUMED`. + """ + + def consume(self, nonce: str) -> NonceState: # pragma: no cover - protocol + ... + + +@dataclass +class InMemoryNonceValidator: + """A single-use, TTL-bounded :class:`NonceValidator` (reference / tests). + + Nonces are 256-bit ``secrets``-random unless a value is supplied to + :meth:`issue`. ``clock`` is injectable so expiry can be exercised + deterministically. + """ + + ttl_seconds: float = 120.0 + clock: Callable[[], float] = time.time + _issued: dict[str, float] = field(default_factory=dict) + _consumed: set[str] = field(default_factory=set) + + def issue(self, nonce: str | None = None) -> str: + value = nonce if nonce is not None else secrets.token_urlsafe(32) + self._issued[value] = self.clock() + return value + + def is_outstanding(self, nonce: str) -> bool: + if not nonce or nonce in self._consumed or nonce not in self._issued: + return False + return (self.clock() - self._issued[nonce]) <= self.ttl_seconds + + def consume(self, nonce: str) -> NonceState: + if not nonce or nonce not in self._issued: + return NonceState.UNKNOWN + if nonce in self._consumed: + return NonceState.CONSUMED + if (self.clock() - self._issued[nonce]) > self.ttl_seconds: + return NonceState.EXPIRED + self._consumed.add(nonce) + return NonceState.OK + + +@dataclass(frozen=True) +class PhalaBinding: + """The run identity a validator expects a submission's ``report_data`` to bind. + + Sourced from the validator's OWN records (the submission's agent hash, the + accepted work unit's task ids, the scores the result reports, and the nonce + the validator issued) -- never trusted from the attested payload. ``task_ids`` + are compared order-insensitively (sorted before hashing). + """ + + agent_hash: str + task_ids: tuple[str, ...] + scores_digest: str + validator_nonce: str = "" + eval_run_id: str | None = None + score_nonce: str | None = None + + def __post_init__(self) -> None: + if (self.eval_run_id is None) != (self.score_nonce is None): + raise ValueError("eval_run_id and score_nonce must be supplied together") + if self.is_eval_v2 and self.validator_nonce: + raise ValueError("schema-version-2 bindings do not use validator_nonce") + + @property + def is_eval_v2(self) -> bool: + """Whether this immutable binding uses the schema-version-2 Eval shape.""" + + return self.eval_run_id is not None or self.score_nonce is not None + + @property + def nonce(self) -> str: + """The purpose-scoped nonce consumed after a successful verification.""" + + return ( + self.score_nonce if self.score_nonce is not None else self.validator_nonce + ) + + +__all__ = [ + "CANONICAL_MEASUREMENT_FIELDS", + "MEASUREMENT_ALLOWLIST_FILE_ENV", + "InMemoryNonceValidator", + "MeasurementAllowlist", + "NonceState", + "NonceValidator", + "PhalaBinding", + "canonical_measurement_mapping", +] diff --git a/src/base/worker/proof.py b/src/base/worker/proof.py index 39d40bf5f..4c041c557 100644 --- a/src/base/worker/proof.py +++ b/src/base/worker/proof.py @@ -12,28 +12,66 @@ from __future__ import annotations +import hashlib +import json +import re +from collections.abc import Collection, Iterable, Mapping from typing import Any +from pydantic import ValidationError + from base.challenge_sdk.proof import ( EXECUTION_PROOF_VERSION, execution_proof_signing_payload, ) -from base.schemas.worker import ExecutionProof, ProviderInfo, WorkerSignature +from base.schemas.worker import ( + PHALA_TDX_TIER, + ExecutionProof, + PhalaAttestation, + PhalaMeasurement, + ProviderInfo, + WorkerSignature, +) from base.security.miner_auth import SignatureVerifier, verify_substrate_signature from base.validator.agent.signing import RequestSigner +from base.worker.phala_quote import ( + QuoteStructureError, + QuoteVerificationError, + QuoteVerifier, + os_image_hash_from_registers, + parse_td_report, + replay_rtmr3, +) +from base.worker.phala_verify import ( + MeasurementAllowlist, + NonceState, + NonceValidator, + PhalaBinding, +) + +#: TCB statuses the Phala-tier verifier accepts by default (architecture sec 7). +#: A quote whose collateral reports any other posture is rejected. +ACCEPTABLE_TCB_DEFAULT: tuple[str, ...] = ("UpToDate",) #: Result-payload key carrying the serialized :class:`ExecutionProof`. PROOF_PAYLOAD_KEY = "execution_proof" #: Result-payload key carrying the deterministic prism manifest hash. MANIFEST_SHA256_PAYLOAD_KEY = "manifest_sha256" +#: Domain-separation tag bound into a Phala attestation's ``report_data`` +#: (architecture.md sec 6). Prevents a quote minted for this purpose from being +#: repurposed under a different protocol tag. +PHALA_REPORT_DATA_TAG = "base-agent-challenge-v1" +#: Byte width of the TDX ``report_data`` field a quote carries. +PHALA_REPORT_DATA_BYTES = 64 + def build_execution_proof( *, signer: RequestSigner, manifest_sha256: str, unit_id: str, - tier: int = 0, + tier: int | str = 0, provider: ProviderInfo | None = None, image_digest: str | None = None, attestation: dict[str, Any] | None = None, @@ -64,27 +102,355 @@ def verify_execution_proof( proof: ExecutionProof, *, unit_id: str, + expected_binding: PhalaBinding | None = None, + quote_verifier: QuoteVerifier | None = None, + allowlist: MeasurementAllowlist | None = None, + nonce_validator: NonceValidator | None = None, + acceptable_tcb: Collection[str] = ACCEPTABLE_TCB_DEFAULT, signature_verifier: SignatureVerifier = verify_substrate_signature, ) -> bool: - """Whether ``proof``'s worker signature verifies for ``unit_id`` (sr25519). + """Whether ``proof`` verifies for ``unit_id``. + + Without ``expected_binding`` this is the tier-0 check only: the worker sr25519 + signature over ``sha256(f"{manifest_sha256}:{unit_id}")`` must verify, and a + proof signed for a DIFFERENT unit is rejected (no cross-unit replay). This is + the unchanged behavior every existing (tier 0/1/2) caller relies on. + + With ``expected_binding`` the full **Phala-tier** verification runs on top of + the tier-0 check (architecture sec 4 C4 / 6 / 7): the proof must be the Phala + tier with an attestation whose TDX quote (a) is DCAP-valid with an acceptable + TCB, (b) reconstructs a measurement in the validator ``allowlist``, (c) whose + ``report_data`` binds exactly ``expected_binding``, and (d) whose nonce is a + fresh, validator-issued, unconsumed one. The tier-0 worker signature is STILL + required, so the attested envelope's trust root is the quote while a real + (validator-rebound) signature prevents cross-unit replay -- an absent/wrong + signature is rejected even for a valid quote (VAL-VERIFY-013). - Rejects a proof presented with a DIFFERENT ``unit_id`` than the one signed, - so a proof cannot be replayed across units. + Returns ``True`` to ACCEPT and ``False`` to REJECT. When the quote-verification + dependency is transiently unavailable it raises + :class:`base.worker.phala_quote.VerifierUnavailableError` so the caller PARKS + the result rather than accepting or fraud-rejecting it (VAL-VERIFY-014). """ payload = execution_proof_signing_payload( manifest_sha256=proof.manifest_sha256, unit_id=unit_id ) - return signature_verifier( + signature_ok = signature_verifier( proof.worker_signature.worker_pubkey, payload, proof.worker_signature.sig ) + if expected_binding is None: + return signature_ok + + if not signature_ok: + return False + if proof.tier != PHALA_TDX_TIER or proof.attestation is None: + return False + if quote_verifier is None or allowlist is None or nonce_validator is None: + return False + try: + attestation = PhalaAttestation.model_validate(proof.attestation) + except ValidationError: + return False + return _verify_phala_attestation( + attestation, + expected_binding=expected_binding, + quote_verifier=quote_verifier, + allowlist=allowlist, + nonce_validator=nonce_validator, + acceptable_tcb=acceptable_tcb, + ) + + +def _verify_phala_attestation( + attestation: PhalaAttestation, + *, + expected_binding: PhalaBinding, + quote_verifier: QuoteVerifier, + allowlist: MeasurementAllowlist, + nonce_validator: NonceValidator, + acceptable_tcb: Collection[str], +) -> bool: + """Verify a Phala attestation against the validator's expectations. + + Fail-closed and conjunctive: every check must pass. The trust root is the + hardware-signed quote -- the measurement and ``report_data`` are read from the + parsed TD report (not the untrusted attestation block), and the event log must + replay to the quote's signed RTMR3. Raises + :class:`base.worker.phala_quote.VerifierUnavailableError` (park) if the quote + verifier is transiently unavailable. + """ + + if not allowlist: + return False + if not expected_binding.nonce: + return False + + try: + report = parse_td_report(attestation.tdx_quote) + except QuoteStructureError: + return False + + try: + verdict = quote_verifier.verify(attestation.tdx_quote) + except QuoteVerificationError: + return False + if verdict.tcb_status not in acceptable_tcb: + return False + + try: + replay = replay_rtmr3(attestation.event_log) + except QuoteVerificationError: + return False + if replay.rtmr3 != report.rtmr3 or replay.compose_hash is None: + return False + + measurement = { + "mrtd": report.mrtd, + "rtmr0": report.rtmr0, + "rtmr1": report.rtmr1, + "rtmr2": report.rtmr2, + "compose_hash": replay.compose_hash, + "os_image_hash": os_image_hash_from_registers( + report.mrtd, report.rtmr1, report.rtmr2 + ), + } + if not allowlist.contains(measurement): + return False + + binding_args: dict[str, str | None] = ( + { + "eval_run_id": expected_binding.eval_run_id, + "score_nonce": expected_binding.score_nonce, + } + if expected_binding.is_eval_v2 + else {"validator_nonce": expected_binding.validator_nonce} + ) + expected_report_data = phala_report_data_hex( + canonical_measurement=measurement, + agent_hash=expected_binding.agent_hash, + task_ids=expected_binding.task_ids, + scores_digest=expected_binding.scores_digest, + **binding_args, + ) + if report.report_data != bytes.fromhex(expected_report_data): + return False + + return nonce_validator.consume(expected_binding.nonce) is NonceState.OK + + +def _canonical_measurement_mapping( + canonical_measurement: PhalaMeasurement | Mapping[str, Any], +) -> dict[str, str]: + """The static, allowlist-pinnable measurement subset (excludes ``rtmr3``).""" + + if isinstance(canonical_measurement, PhalaMeasurement): + return canonical_measurement.canonical() + static_fields = ( + "mrtd", + "rtmr0", + "rtmr1", + "rtmr2", + "compose_hash", + "os_image_hash", + ) + return {field: str(canonical_measurement[field]) for field in static_fields} + + +def _is_visible_id(value: object) -> bool: + """Whether an Eval wire identifier is one to 128 visible ASCII bytes.""" + + return ( + isinstance(value, str) + and 1 <= len(value) <= 128 + and all("!" <= char <= "~" for char in value) + ) + + +def _is_lower_hex(value: object, width: int) -> bool: + """Whether ``value`` has an exact-width lowercase hexadecimal encoding.""" + + return ( + isinstance(value, str) + and len(value) == width + and re.fullmatch(r"[0-9a-f]+", value) is not None + ) + + +def _schema_v2_measurement( + canonical_measurement: PhalaMeasurement | Mapping[str, Any], +) -> dict[str, str]: + """Validate and return the strict static Eval measurement object.""" + + try: + measurement = _canonical_measurement_mapping(canonical_measurement) + except (KeyError, TypeError) as exc: + raise ValueError( + "schema-version-2 canonical measurement is incomplete" + ) from exc + + register_fields = ("mrtd", "rtmr0", "rtmr1", "rtmr2") + if any(not _is_lower_hex(measurement[field], 96) for field in register_fields): + raise ValueError("schema-version-2 measurement registers must be 96 hex chars") + if not _is_lower_hex(measurement["compose_hash"], 64) or not _is_lower_hex( + measurement["os_image_hash"], 64 + ): + raise ValueError("schema-version-2 measurement hashes must be 64 hex chars") + return measurement + + +def phala_report_data( + *, + canonical_measurement: PhalaMeasurement | Mapping[str, Any], + agent_hash: str, + task_ids: Iterable[str], + scores_digest: str, + validator_nonce: str | None = None, + eval_run_id: str | None = None, + score_nonce: str | None = None, +) -> bytes: + """The 32-byte ``report_data`` digest binding a Phala run (architecture sec 6). + + The schema-version-2 Eval boundary hashes canonical JSON over + ``{agent_hash, canonical_measurement, domain, eval_run_id, schema_version, + score_nonce, scores_digest, task_ids}``; its sorted task IDs are a + duplicate-free set and its measurement excludes runtime ``rtmr3``. Legacy + callers that still supply only ``validator_nonce`` retain the original v1 + derivation until they migrate to the direct Eval boundary. + + This is the single source of truth for the derivation shared by the image + emitter (M1) and the validator/master verifier (M4): both MUST call this + function rather than re-implementing sec 6. + """ + + supplied_tasks = tuple(task_ids) + if (eval_run_id is None) != (score_nonce is None): + raise ValueError("eval_run_id and score_nonce must be supplied together") + + if eval_run_id is not None and score_nonce is not None: + if validator_nonce is not None: + raise ValueError("schema-version-2 binding does not use validator_nonce") + if ( + not _is_visible_id(eval_run_id) + or not _is_visible_id(score_nonce) + or any(not _is_visible_id(task_id) for task_id in supplied_tasks) + ): + raise ValueError("schema-version-2 ids must be visible ASCII") + if not _is_lower_hex(agent_hash, 64) or not _is_lower_hex(scores_digest, 64): + raise ValueError("schema-version-2 digests must be 64 lowercase hex chars") + task_set = tuple(sorted(supplied_tasks)) + if len(task_set) != len(set(task_set)): + raise ValueError("task_ids must be unique") + if supplied_tasks != task_set: + raise ValueError("schema-version-2 task_ids must be sorted") + preimage = { + "agent_hash": agent_hash, + "canonical_measurement": _schema_v2_measurement(canonical_measurement), + "domain": PHALA_REPORT_DATA_TAG, + "eval_run_id": eval_run_id, + "schema_version": 2, + "score_nonce": score_nonce, + "scores_digest": scores_digest, + "task_ids": list(supplied_tasks), + } + else: + if validator_nonce is None: + raise ValueError("validator_nonce is required for legacy bindings") + preimage = { + "tag": PHALA_REPORT_DATA_TAG, + "canonical_measurement": _canonical_measurement_mapping( + canonical_measurement + ), + "agent_hash": agent_hash, + "task_ids": sorted(supplied_tasks), + "scores_digest": scores_digest, + "validator_nonce": validator_nonce, + } + encoded = json.dumps( + preimage, + ensure_ascii=eval_run_id is None, + sort_keys=True, + separators=(",", ":"), + ).encode() + return hashlib.sha256(encoded).digest() + + +def phala_report_data_hex( + *, + canonical_measurement: PhalaMeasurement | Mapping[str, Any], + agent_hash: str, + task_ids: Iterable[str], + scores_digest: str, + validator_nonce: str | None = None, + eval_run_id: str | None = None, + score_nonce: str | None = None, +) -> str: + """``report_data`` as a 64-byte TDX field (128 hex chars, left-aligned). + + The 32-byte :func:`phala_report_data` digest occupies the leading bytes and + the trailing bytes are zero, matching Phala's observed left-aligned zero-pad + round-trip for the fixed-width quote field. + """ + + digest = phala_report_data( + canonical_measurement=canonical_measurement, + agent_hash=agent_hash, + task_ids=task_ids, + scores_digest=scores_digest, + validator_nonce=validator_nonce, + eval_run_id=eval_run_id, + score_nonce=score_nonce, + ) + return digest.ljust(PHALA_REPORT_DATA_BYTES, b"\x00").hex() + + +def build_phala_execution_proof( + *, + signer: RequestSigner, + manifest_sha256: str, + unit_id: str, + attestation: PhalaAttestation | Mapping[str, Any], + provider: ProviderInfo | None = None, + image_digest: str | None = None, +) -> ExecutionProof: + """Build a Phala-tier ExecutionProof carrying a TDX attestation payload. + + The envelope keeps the tier-0 worker sr25519 signature over the pinned + ``sha256(f"{manifest_sha256}:{unit_id}")`` message, so + :func:`verify_execution_proof` still validates the worker-signature layer; + ``tier`` is set to :data:`PHALA_TDX_TIER` and ``attestation`` carries the + serialized :class:`PhalaAttestation`. Cryptographic quote verification is + layered on top by the validator/master (milestone M4); this helper does not + fork a parallel envelope. + """ + + payload = ( + attestation + if isinstance(attestation, PhalaAttestation) + else PhalaAttestation.model_validate(dict(attestation)) + ) + return build_execution_proof( + signer=signer, + manifest_sha256=manifest_sha256, + unit_id=unit_id, + tier=PHALA_TDX_TIER, + provider=provider, + image_digest=image_digest, + attestation=payload.model_dump(mode="json"), + ) __all__ = [ + "ACCEPTABLE_TCB_DEFAULT", "EXECUTION_PROOF_VERSION", "MANIFEST_SHA256_PAYLOAD_KEY", + "PHALA_REPORT_DATA_BYTES", + "PHALA_REPORT_DATA_TAG", + "PHALA_TDX_TIER", "PROOF_PAYLOAD_KEY", "build_execution_proof", + "build_phala_execution_proof", "execution_proof_signing_payload", + "phala_report_data", + "phala_report_data_hex", "verify_execution_proof", ] diff --git a/tests/unit/eval_execution_proof_v2_vectors.json b/tests/unit/eval_execution_proof_v2_vectors.json new file mode 100644 index 000000000..5e89a51c6 --- /dev/null +++ b/tests/unit/eval_execution_proof_v2_vectors.json @@ -0,0 +1,205 @@ +{ + "description": "Shared positive and malformed vectors for Eval ExecutionProof v1 and its schema-version-2 Phala score report-data binding. The agent-challenge implementation must assert byte-identical results.", + "positive": { + "binding": { + "agent_hash": "f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0", + "canonical_measurement": { + "compose_hash": "c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3", + "mrtd": "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1", + "os_image_hash": "0fbc46b4f3c1c019f20d7985f653a46a0303e106edeed801b0ab29b0660d3b2e", + "rtmr0": "b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0", + "rtmr1": "b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1", + "rtmr2": "b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2" + }, + "eval_run_id": "eval-run-001", + "score_nonce": "score-nonce-001", + "scores_digest": "9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a9a", + "task_ids": [ + "task-a", + "task-b", + "task-c" + ] + }, + "canonical_json_utf8_hex": "7b226167656e745f68617368223a2266306630663066306630663066306630663066306630663066306630663066306630663066306630663066306630663066306630663066306630663066306630222c2263616e6f6e6963616c5f6d6561737572656d656e74223a7b22636f6d706f73655f68617368223a2263336333633363336333633363336333633363336333633363336333633363336333633363336333633363336333633363336333633363336333633363336333222c226d727464223a22613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131613161316131222c226f735f696d6167655f68617368223a2230666263343662346633633163303139663230643739383566363533613436613033303365313036656465656438303162306162323962303636306433623265222c2272746d7230223a22623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230623062306230222c2272746d7231223a22623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231623162316231222c2272746d7232223a22623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232623262326232227d2c22646f6d61696e223a22626173652d6167656e742d6368616c6c656e67652d7631222c226576616c5f72756e5f6964223a226576616c2d72756e2d303031222c22736368656d615f76657273696f6e223a322c2273636f72655f6e6f6e6365223a2273636f72652d6e6f6e63652d303031222c2273636f7265735f646967657374223a2239613961396139613961396139613961396139613961396139613961396139613961396139613961396139613961396139613961396139613961396139613961222c227461736b5f696473223a5b227461736b2d61222c227461736b2d62222c227461736b2d63225d7d", + "report_data_hex": "360ef521e929f903e1e93db74e4c3462d0622d21cc6b2c8f7ee8ff783389ebed0000000000000000000000000000000000000000000000000000000000000000", + "execution_proof": { + "version": 1, + "tier": "phala-tdx", + "manifest_sha256": "1111111111111111111111111111111111111111111111111111111111111111", + "image_digest": "registry.example/eval@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", + "provider": null, + "worker_signature": { + "worker_pubkey": "", + "sig": "" + }, + "attestation": { + "tdx_quote": "abababababababab", + "event_log": [ + { + "imr": 3, + "event_type": 134217729, + "digest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "event": "compose-hash", + "event_payload": "c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3" + } + ], + "report_data": "360ef521e929f903e1e93db74e4c3462d0622d21cc6b2c8f7ee8ff783389ebed0000000000000000000000000000000000000000000000000000000000000000", + "measurement": { + "compose_hash": "c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3", + "mrtd": "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1", + "os_image_hash": "0fbc46b4f3c1c019f20d7985f653a46a0303e106edeed801b0ab29b0660d3b2e", + "rtmr0": "b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0", + "rtmr1": "b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1", + "rtmr2": "b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2", + "rtmr3": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" + }, + "vm_config": { + "vcpu": 1, + "memory_mb": 2048, + "os_image_hash": "0fbc46b4f3c1c019f20d7985f653a46a0303e106edeed801b0ab29b0660d3b2e" + } + } + } + }, + "malformed": [ + { + "name": "duplicate_json_key", + "kind": "raw_json", + "value": "{\"version\":1,\"version\":1}" + }, + { + "name": "tdx_quote_alias", + "kind": "rename", + "from": "attestation.tdx_quote", + "to": "attestation.tdx_quote_b64" + }, + { + "name": "report_data_alias", + "kind": "rename", + "from": "attestation.report_data", + "to": "attestation.report_data_hex" + }, + { + "name": "unknown_attestation_field", + "kind": "set", + "path": "attestation.unknown", + "value": "forbidden" + }, + { + "name": "unknown_execution_proof_field", + "kind": "set", + "path": "unknown", + "value": "forbidden" + }, + { + "name": "unknown_measurement_field", + "kind": "set", + "path": "attestation.measurement.unknown", + "value": "forbidden" + }, + { + "name": "unknown_event_field", + "kind": "set", + "path": "attestation.event_log.0.unknown", + "value": "forbidden" + }, + { + "name": "unknown_vm_config_field", + "kind": "set", + "path": "attestation.vm_config.unknown", + "value": "forbidden" + }, + { + "name": "uppercase_quote", + "kind": "set", + "path": "attestation.tdx_quote", + "value": "ABAB" + }, + { + "name": "odd_length_quote", + "kind": "set", + "path": "attestation.tdx_quote", + "value": "abc" + }, + { + "name": "short_report_data", + "kind": "set", + "path": "attestation.report_data", + "value": "aa" + }, + { + "name": "long_report_data", + "kind": "set", + "path": "attestation.report_data", + "value": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + }, + { + "name": "short_mrtd", + "kind": "set", + "path": "attestation.measurement.mrtd", + "value": "aa" + }, + { + "name": "long_mrtd", + "kind": "set", + "path": "attestation.measurement.mrtd", + "value": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + }, + { + "name": "short_compose_hash", + "kind": "set", + "path": "attestation.measurement.compose_hash", + "value": "aa" + }, + { + "name": "long_os_image_hash", + "kind": "set", + "path": "attestation.measurement.os_image_hash", + "value": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" + }, + { + "name": "short_event_digest", + "kind": "set", + "path": "attestation.event_log.0.digest", + "value": "cc" + }, + { + "name": "long_event_digest", + "kind": "set", + "path": "attestation.event_log.0.digest", + "value": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" + }, + { + "name": "odd_event_payload", + "kind": "set", + "path": "attestation.event_log.0.event_payload", + "value": "c" + }, + { + "name": "uppercase_manifest", + "kind": "set", + "path": "manifest_sha256", + "value": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + }, + { + "name": "invalid_image_digest", + "kind": "set", + "path": "image_digest", + "value": "registry.example/eval:latest" + }, + { + "name": "provider_present", + "kind": "set", + "path": "provider", + "value": { + "name": "forbidden" + } + }, + { + "name": "partial_placeholder_signature", + "kind": "set", + "path": "worker_signature.worker_pubkey", + "value": "not-empty" + } + ] +} diff --git a/tests/unit/phala_quote_golden_vector.json b/tests/unit/phala_quote_golden_vector.json new file mode 100644 index 000000000..5def83417 --- /dev/null +++ b/tests/unit/phala_quote_golden_vector.json @@ -0,0 +1,31 @@ +{ + "description": "Shared cross-repo golden TDX quote/measurement/RTMR3 anti-drift vector. A fixed v4 TDX quote + dstack RTMR3 event log with the exact registers, os_image_hash (sha256(MRTD||RTMR1||RTMR2)) and RTMR3/compose-hash a correct parser must reproduce. Asserted BYTE-IDENTICALLY in both base (tests/unit/test_phala_quote_golden_vector.py) and agent-challenge (tests/test_quote_golden_vector.py). A one-sided offset/hash tweak to either base.worker.phala_quote or agent_challenge.keyrelease.quote makes that repo's parse of this fixed quote diverge from these pinned values, failing a test on that side. Do NOT edit one repo's copy without the other (see AGENTS.md 'TDX-quote-parse / measurement / RTMR3 anti-drift').", + "event_log": [ + { + "digest": "2851989bdb17f204310f1c0ef0ed4bf8ffbd3ed455af43ce0f438e94356c858b8746a15516c7c0102a58bb71d67f51e2", + "event": "compose-hash", + "event_payload": "abababababababababababababababababababababababababababababababab", + "event_type": 134217729, + "imr": 3 + }, + { + "digest": "0c365c23d8f1ad06b7bb3c843b1d3d7b0b8347a0f175e71c118241d8048005fe3538cc05eb89c52250bad3244646b739", + "event": "key-provider", + "event_payload": "cdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcd", + "event_type": 134217729, + "imr": 3 + } + ], + "expected": { + "compose_hash": "abababababababababababababababababababababababababababababababab", + "key_provider": "cdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcd", + "mrtd": "111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111", + "os_image_hash": "90ac6e18901b222f3a58a45231a39545d14a9cd59854cc3eafa683083f0e11d0", + "report_data_hex": "55555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555", + "rtmr0": "222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222", + "rtmr1": "333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333", + "rtmr2": "444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444", + "rtmr3": "7934b424e98de2a6fba5c468ee804fd575de5d6cb6201f78416a976b23e616e1c74aa9e28a1d4e0c55a65b849007ae68" + }, + "quote_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222223333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333334444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444447934b424e98de2a6fba5c468ee804fd575de5d6cb6201f78416a976b23e616e1c74aa9e28a1d4e0c55a65b849007ae6855555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555" +} diff --git a/tests/unit/test_agent_challenge_attested_proxy.py b/tests/unit/test_agent_challenge_attested_proxy.py new file mode 100644 index 000000000..181b24bb2 --- /dev/null +++ b/tests/unit/test_agent_challenge_attested_proxy.py @@ -0,0 +1,572 @@ +from __future__ import annotations + +from contextlib import asynccontextmanager +from dataclasses import dataclass +from decimal import Decimal +from typing import Any + +import httpx +import pytest +from fastapi.testclient import TestClient + +from base.config.settings import MasterSettings +from base.master.app_proxy import create_proxy_app +from base.master.registry import ChallengeRegistry +from base.schemas.challenge import ChallengeCreate, ChallengeStatus +from base.security.miner_auth import NonceReplayError + + +class _NonceStore: + def __init__(self) -> None: + self.keys: set[tuple[int, str, str, str]] = set() + + async def reserve(self, **kwargs: Any) -> None: + key = ( + int(kwargs["netuid"]), + str(kwargs["challenge_slug"]), + str(kwargs["hotkey"]), + str(kwargs["nonce"]), + ) + if key in self.keys: + raise NonceReplayError("nonce already used") + self.keys.add(key) + + +class _Cache: + def get(self) -> dict[str, int]: + return {} + + +@dataclass(frozen=True) +class _SignedRoute: + method: str + path: str + upstream_path: str + upstream_status: int + + +SIGNED_ROUTES = ( + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions", + "/submissions", + 201, + ), + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions/sub-1/review/prepare", + "/submissions/sub-1/review/prepare", + 200, + ), + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions/sub-1/review/retry", + "/submissions/sub-1/review/retry", + 201, + ), + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions/sub-1/review/deployed", + "/submissions/sub-1/review/deployed", + 200, + ), + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions/sub-1/review/cancel", + "/submissions/sub-1/review/cancel", + 200, + ), + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions/sub-1/eval/prepare", + "/submissions/sub-1/eval/prepare", + 200, + ), + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions/sub-1/eval/retry", + "/submissions/sub-1/eval/retry", + 201, + ), + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions/sub-1/eval/cancel", + "/submissions/sub-1/eval/cancel", + 200, + ), + _SignedRoute( + "POST", + "/challenges/agent-challenge/submissions/sub-1/eval/failure", + "/submissions/sub-1/eval/failure", + 200, + ), + _SignedRoute( + "GET", + "/challenges/agent-challenge/submissions/sub-1/review/report", + "/submissions/sub-1/review/report", + 200, + ), + _SignedRoute( + "GET", + "/challenges/agent-challenge/submissions/sub-1/eval/status", + "/submissions/sub-1/eval/status", + 200, + ), +) + + +def _registry() -> ChallengeRegistry: + registry = ChallengeRegistry() + registry.create( + ChallengeCreate( + slug="agent-challenge", + name="Agent Challenge", + image="ghcr.io/baseintelligence/agent-challenge:latest@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + version="1.0.0", + emission_percent=Decimal("100"), + status=ChallengeStatus.ACTIVE, + internal_base_url="http://challenge-agent-challenge:8000", + ) + ) + return registry + + +def _proxy_client( + handler: httpx.AsyncBaseTransport | Any, + *, + attested_routes_enabled: bool = True, +) -> TestClient: + @asynccontextmanager + async def client_factory(): + transport = ( + handler + if isinstance(handler, httpx.AsyncBaseTransport) + else httpx.MockTransport(handler) + ) + async with httpx.AsyncClient( + transport=transport, + base_url="http://challenge-agent-challenge:8000", + ) as client: + yield client + + return TestClient( + create_proxy_app( + registry=_registry(), + nonce_store=_NonceStore(), + metagraph_cache=_Cache(), # type: ignore[arg-type] + client_factory=client_factory, + agent_challenge_attested_routes_enabled=attested_routes_enabled, + ) + ) + + +@pytest.mark.parametrize("route", SIGNED_ROUTES) +def test_exact_attested_signed_route_preserves_canonical_upstream_bytes( + route: _SignedRoute, +) -> None: + captured: dict[str, Any] = {} + upstream_body = ( + b'{"schema_version":1,"opaque":"upstream\\u0000bytes","route":"' + + route.upstream_path.encode() + + b'"}' + ) + + async def handler(request: httpx.Request) -> httpx.Response: + captured["method"] = request.method + captured["path"] = request.url.path + captured["query"] = request.url.query + captured["body"] = await request.aread() + captured["headers"] = request.headers + return httpx.Response( + route.upstream_status, + content=upstream_body, + headers={"content-type": "application/vnd.base.attested+json"}, + ) + + client = _proxy_client(handler) + request_body = ( + b'{"schema_version":1,"expected_id":"opaque","approval_id":"operator-1",' + b'"binary":"\\u0000\\u00ff"}' + ) + response = client.request( + route.method, + f"{route.path}?z=last&a=first", + content=request_body, + headers={ + "Content-Type": "application/vnd.base.signed+json", + "X-Hotkey": "miner-hotkey", + "X-Signature": "miner-signature", + "X-Nonce": "miner-nonce", + "X-Timestamp": "1700000000", + }, + ) + + assert response.status_code == route.upstream_status + assert response.content == upstream_body + assert response.headers["content-type"] == "application/vnd.base.attested+json" + assert captured["method"] == route.method + assert captured["path"] == route.upstream_path + assert captured["query"] == b"z=last&a=first" + assert captured["body"] == request_body + headers = captured["headers"] + assert headers["content-type"] == "application/vnd.base.signed+json" + assert headers["x-hotkey"] == "miner-hotkey" + assert headers["x-signature"] == "miner-signature" + assert headers["x-nonce"] == "miner-nonce" + assert headers["x-timestamp"] == "1700000000" + + +def test_attested_signed_route_strips_caller_authority_and_proxy_headers() -> None: + captured: dict[str, Any] = {} + + async def handler(request: httpx.Request) -> httpx.Response: + captured["headers"] = request.headers + return httpx.Response(200, json={"ok": True}) + + client = _proxy_client(handler) + response = client.post( + "/challenges/agent-challenge/submissions/sub-1/review/retry", + content=b'{"expected_assignment_id":"assignment-1","approval_id":"approval-1"}', + headers={ + "X-Hotkey": "miner-hotkey", + "X-Signature": "miner-signature", + "X-Nonce": "miner-nonce", + "X-Timestamp": "1700000000", + "Authorization": "Bearer caller-capability", + "Proxy-Authorization": "Basic caller-proxy", + "X-Admin-Token": "caller-admin", + "X-Base-Admin-Token": "caller-base-admin", + "X-Base-Internal-Token": "caller-internal", + "X-Internal-Authorization": "caller-internal-auth", + "X-Base-Verified-Hotkey": "caller-verified", + "X-Base-Verified-Future": "caller-future-trust", + "X-Base-Request-Hash": "caller-hash", + "X-Trust-Level": "caller-trust", + "X-Trusted-Proxy": "caller-trusted-proxy", + "X-Base-Trust-Result": "caller-base-trust", + "X-RA-TLS-Peer-Key": "caller-peer", + "X-RATLS-Peer-Certificate": "caller-peer-cert", + "X-Review-Verified": "true", + "X-Review-Verification": "passed", + "X-Attestation-Verified": "true", + "X-Allowlist-Digest": "caller-allowlist", + "X-Measurement-MRTD": "caller-measurement", + "Forwarded": "for=caller", + "Via": "caller-proxy", + "X-Forwarded-For": "198.51.100.7", + "X-Forwarded-Host": "evil.example", + "X-Forwarded-Proto": "https", + "X-Real-IP": "198.51.100.8", + "X-Proxy-Trust": "caller-proxy-trust", + "X-Base-Proxy": "false", + "X-Base-Challenge-Slug": "prism", + "X-Public-Header": "preserved", + }, + ) + + assert response.status_code == 200 + headers: httpx.Headers = captured["headers"] + assert headers["x-hotkey"] == "miner-hotkey" + assert headers["x-signature"] == "miner-signature" + assert headers["x-nonce"] == "miner-nonce" + assert headers["x-timestamp"] == "1700000000" + assert headers["x-public-header"] == "preserved" + assert headers.get_list("x-base-proxy") == ["true"] + assert headers.get_list("x-base-challenge-slug") == ["agent-challenge"] + forbidden = { + "authorization", + "proxy-authorization", + "x-admin-token", + "x-base-admin-token", + "x-base-internal-token", + "x-internal-authorization", + "x-base-verified-hotkey", + "x-base-verified-future", + "x-base-request-hash", + "x-trust-level", + "x-trusted-proxy", + "x-base-trust-result", + "x-ra-tls-peer-key", + "x-ratls-peer-certificate", + "x-review-verified", + "x-review-verification", + "x-attestation-verified", + "x-allowlist-digest", + "x-measurement-mrtd", + "forwarded", + "via", + "x-forwarded-for", + "x-forwarded-host", + "x-forwarded-proto", + "x-real-ip", + "x-proxy-trust", + } + assert forbidden.isdisjoint(headers) + + +@pytest.mark.parametrize( + "path", + ( + "/challenges/agent-challenge/submissions/sub-1/review/prepare/", + "/challenges/agent-challenge/submissions/sub-1/review//prepare", + "/challenges/agent-challenge/submissions//sub-1/review/prepare", + "/challenges/agent-challenge/submissions/sub-1/review/%70repare", + "/challenges/agent-challenge/submissions/%73ub-1/review/prepare", + "/challenges/%61gent-challenge/submissions/sub-1/review/prepare", + "/challenges/agent-challenge/submissions/sub-1/eval/status/", + ), +) +def test_attested_signed_route_rejects_noncanonical_path_neighbors(path: str) -> None: + upstream_calls: list[str] = [] + + async def handler(request: httpx.Request) -> httpx.Response: + upstream_calls.append(request.url.path) + return httpx.Response(200, json={"unexpected": True}) + + response = _proxy_client(handler).request( + "GET" if "status" in path else "POST", + path, + headers={ + "X-Hotkey": "miner-hotkey", + "X-Signature": "miner-signature", + "X-Nonce": "miner-nonce", + "X-Timestamp": "1700000000", + }, + ) + + assert response.status_code == 404 + assert upstream_calls == [] + + +@pytest.mark.parametrize("slug_alias", ("Agent%20Challenge", "AGENT-CHALLENGE")) +def test_attested_private_routes_reject_agent_challenge_name_aliases( + slug_alias: str, +) -> None: + upstream_calls: list[str] = [] + + async def handler(request: httpx.Request) -> httpx.Response: + upstream_calls.append(request.url.path) + return httpx.Response(200, json={"unexpected": True}) + + response = _proxy_client(handler).get( + f"/challenges/{slug_alias}/review/v1/assignments/assignment-1/artifact", + headers={"Authorization": "Bearer caller-capability"}, + ) + + assert response.status_code == 404 + assert upstream_calls == [] + + +@pytest.mark.parametrize( + ("method", "path"), + ( + ("GET", "/submissions/sub-1/review/prepare"), + ("PUT", "/submissions/sub-1/review/retry"), + ("POST", "/submissions/sub-1/review/report"), + ("GET", "/submissions/sub-1/review/history"), + ("GET", "/submissions/sub-1/eval/prepare"), + ("POST", "/submissions/sub-1/eval/status"), + ("POST", "/submissions/sub-1/eval/result"), + ("POST", "/submissions/sub-1/eval/key-release"), + ("GET", "/submissions/sub-1/env"), + ("PUT", "/submissions/sub-1/env"), + ("POST", "/submissions/sub-1/env/confirm-empty"), + ("POST", "/submissions/sub-1/launch"), + ("GET", "/review/v1/assignments/assignment-1/artifact"), + ("GET", "/review/v1/assignments/assignment-1/rules"), + ("POST", "/review/v1/assignments/assignment-1/model-call-started"), + ("POST", "/review/v1/assignments/assignment-1/failure"), + ("POST", "/review/v1/assignments/assignment-1/report"), + ("GET", "/internal/v1/reviews/session-1/report"), + ("GET", "/internal/v1/reviews/session-1/evidence/object-1"), + ("POST", "/internal/v1/reviews/session-1/approvals"), + ("POST", "/evaluation/v1/runs/run-1/result"), + ("GET", "/key-release/nonce"), + ("POST", "/key-release/release"), + ("GET", "/keyrelease/nonce"), + ("POST", "/keyrelease/release"), + ("GET", "/nonce"), + ("POST", "/release"), + # Fall-through aliases that a deny-list leave-behind would still forward. + ("GET", "/results"), + ("POST", "/results"), + ("GET", "/result"), + ("POST", "/result"), + ("GET", "/submissions/sub-1/results"), + ("POST", "/submissions/sub-1/results"), + ("GET", "/submissions/sub-1/result"), + ("POST", "/submissions/sub-1/result"), + ("GET", "/capability"), + ("POST", "/capability/token"), + ("GET", "/capabilities/token"), + ("POST", "/assignments/assignment-1"), + ("GET", "/assignment/assignment-1"), + ("GET", "/evidence/object-1"), + ("GET", "/submissions/sub-1/evidence/object-1"), + ("POST", "/key_release/release"), + ("GET", "/direct-result"), + ("POST", "/direct/result"), + ("GET", "/anything-private"), + ("POST", "/evals/run-1/result"), + ), +) +@pytest.mark.parametrize( + "prefix", + ( + "/challenges/agent-challenge", + "/v1/challenges/agent-challenge", + ), +) +def test_attested_private_neighbors_and_aliases_are_local_404( + method: str, + path: str, + prefix: str, +) -> None: + upstream_calls: list[str] = [] + + async def handler(request: httpx.Request) -> httpx.Response: + upstream_calls.append(request.url.path) + return httpx.Response(200, json={"unexpected": True}) + + client = _proxy_client(handler) + response = client.request( + method, + f"{prefix}{path}", + content=b'{"caller_trust":true}', + headers={ + "Authorization": "Bearer caller-capability", + "X-Hotkey": "miner-hotkey", + "X-Signature": "miner-signature", + "X-Nonce": "miner-nonce", + "X-Timestamp": "1700000000", + "X-Allowlist-Digest": "caller-allowlist", + "X-Measurement-MRTD": "caller-measurement", + "X-RA-TLS-Peer-Key": "caller-peer", + "X-Review-Verified": "true", + "X-Base-Verified-Hotkey": "caller-verified", + }, + ) + + assert response.status_code == 404 + assert upstream_calls == [] + + +@pytest.mark.parametrize( + ("method", "path", "upstream_path"), + ( + ( + "GET", + "/challenges/agent-challenge/submissions/sub-1/status", + "/submissions/sub-1/status", + ), + ( + "GET", + "/challenges/agent-challenge/submissions/sub-1/events", + "/submissions/sub-1/events", + ), + ( + "GET", + "/challenges/agent-challenge/benchmarks/tasks", + "/benchmarks/tasks", + ), + ), +) +def test_attested_public_status_and_benchmark_routes_remain_forwardable( + method: str, + path: str, + upstream_path: str, +) -> None: + captured: dict[str, Any] = {} + + async def handler(request: httpx.Request) -> httpx.Response: + captured["method"] = request.method + captured["path"] = request.url.path + captured["headers"] = request.headers + return httpx.Response( + 200, + content=b'{"schema_version":1,"safe":true}', + headers={"content-type": "application/json"}, + ) + + response = _proxy_client(handler).request( + method, + path, + headers={ + "Authorization": "Bearer caller-capability", + "X-Allowlist-Digest": "caller-allowlist", + "X-Measurement-MRTD": "caller-measurement", + "X-RA-TLS-Peer-Key": "caller-peer", + "X-Review-Verified": "true", + "X-Public-Header": "preserved", + }, + ) + + assert response.status_code == 200 + assert response.content == b'{"schema_version":1,"safe":true}' + assert captured["method"] == method + assert captured["path"] == upstream_path + headers: httpx.Headers = captured["headers"] + assert headers["x-public-header"] == "preserved" + assert headers.get_list("x-base-proxy") == ["true"] + assert headers.get_list("x-base-challenge-slug") == ["agent-challenge"] + assert "authorization" not in headers + assert "x-allowlist-digest" not in headers + assert "x-measurement-mrtd" not in headers + assert "x-ra-tls-peer-key" not in headers + assert "x-review-verified" not in headers + + +def test_attested_signed_upstream_auth_error_is_preserved_without_rewriting() -> None: + upstream_body = b'{"detail":{"code":"invalid_signed_request"}}' + + async def handler(request: httpx.Request) -> httpx.Response: + assert "x-signature" not in request.headers + return httpx.Response( + 401, + content=upstream_body, + headers={"content-type": "application/problem+json"}, + ) + + client = _proxy_client(handler) + response = client.post( + "/challenges/agent-challenge/submissions/sub-1/eval/prepare", + content=b'{"schema_version":1}', + ) + + assert response.status_code == 401 + assert response.content == upstream_body + assert response.headers["content-type"] == "application/problem+json" + + +def test_attested_proxy_flag_defaults_off_and_keeps_generic_legacy_behavior() -> None: + assert MasterSettings().agent_challenge_attested_routes_enabled is False + captured: dict[str, Any] = {} + + async def handler(request: httpx.Request) -> httpx.Response: + captured["path"] = request.url.path + captured["headers"] = request.headers + return httpx.Response(404, content=b'{"detail":"Not Found"}') + + client = _proxy_client(handler, attested_routes_enabled=False) + response = client.post( + "/challenges/agent-challenge/submissions/sub-1/review/prepare", + content=b'{"schema_version":1}', + headers={ + "X-Hotkey": "legacy-miner", + "X-Signature": "legacy-signature", + "X-Nonce": "legacy-nonce", + "X-Timestamp": "1700000000", + "X-Forwarded-For": "198.51.100.7", + "X-Review-Legacy-Metadata": "legacy-value", + }, + ) + + assert response.status_code == 404 + assert response.content == b'{"detail":"Not Found"}' + assert captured["path"] == "/submissions/sub-1/review/prepare" + assert captured["headers"]["x-forwarded-for"] == "198.51.100.7" + assert captured["headers"]["x-review-legacy-metadata"] == "legacy-value" + assert "x-hotkey" not in captured["headers"] + assert "x-signature" not in captured["headers"] + assert "x-nonce" not in captured["headers"] + assert "x-timestamp" not in captured["headers"] diff --git a/tests/unit/test_client_service_cli_config.py b/tests/unit/test_client_service_cli_config.py index 46937be2a..fb7fe8c05 100644 --- a/tests/unit/test_client_service_cli_config.py +++ b/tests/unit/test_client_service_cli_config.py @@ -483,6 +483,7 @@ def test_cli_master_proxy_builds_single_port_app_with_admin_deps( "master:", " proxy_host: 127.0.0.1", " proxy_port: 0", + " agent_challenge_attested_routes_enabled: true", "docker:", f" secret_dir: {tmp_path / 'secrets'}", "security:", @@ -554,6 +555,7 @@ def fake_create_proxy_app(**kwargs: object) -> object: assert token_provider() == "top-secret" assert captured["weight_service_cache"] is runtime.metagraph_cache assert proxy_kwargs["enforce_production_policy"] is False + assert proxy_kwargs["agent_challenge_attested_routes_enabled"] is True def test_cli_master_proxy_wires_coordination_plane_and_gateway( diff --git a/tests/unit/test_cross_integration_carry_chain.py b/tests/unit/test_cross_integration_carry_chain.py new file mode 100644 index 000000000..fa8a4ef70 --- /dev/null +++ b/tests/unit/test_cross_integration_carry_chain.py @@ -0,0 +1,641 @@ +"""Base-side cross-integration: zero full-attested assignment + carry-chain integrity. + +Offline complements to the agent-challenge cross-integration suite +(``agent-challenge/tests/test_cross_integration_e2e_offline.py``). These cover +the two legs that live in the base repo: + +* **VAL-CROSS-003** -- in full attested mode R=1 means one miner-funded external + ``eval_run_id`` with the challenge exposing **zero** assignable work units. + The master therefore creates zero agent-challenge validator assignment, retry, + replica, reconciliation, audit, or fold rows and does not dispatch + ``AgentChallengeCycleExecutor``. A sibling prism gpu unit in the SAME pass IS + still replicated to R=2 (worker plane genuinely active; zero agent-challenge + rows are not vacuum). Only sampled replay audit may invoke the legacy broker. + +* **VAL-CROSS-007 (base leg)** -- the TDX quote / ``report_data`` / measurement + emitted alongside the ``BASE_BENCHMARK_RESULT=`` line by the in-CVM backend is + mapped onto the BASE ``ExecutionProof`` Phala tier and carried to the master + BYTE-FOR-BYTE across the JSON serialization boundary and the validator + tier-0 signature rebind. The carried envelope still verifies, and a single + flipped quote byte / re-encoded ``report_data`` breaks it (the carry is a real + discriminator, not a constant pass). The challenge leg (envelope emitted + alongside the result line and parsed by the host normalizer) is asserted in the + agent-challenge repo. + +* **VAL-CROSS-021 (base leg)** -- with the flag OFF the BASE validator adapter + + master handle an agent-challenge unit exactly as legacy: no Phala tier is + required on results, no attestation gate is applied, and master orchestration + keeps the legacy reassign-on-failure (NEVER replicate) path for the cpu units. +""" + +from __future__ import annotations + +import hashlib +from dataclasses import dataclass, field +from datetime import UTC, datetime +from typing import Any + +from sqlalchemy import func, select + +from base.bittensor.metagraph_cache import MetagraphCache +from base.db import ( + Base, + Validator, + ValidatorStatus, + WorkAssignmentStatus, + WorkerAssignment, + WorkerFault, + WorkerRegistration, + WorkerStatus, + create_engine, + create_session_factory, + session_scope, +) +from base.db.models import WorkAssignment +from base.master.assignment import CAPABILITY_GPU, AssignmentService +from base.master.assignment_coordination import AssignmentCoordinationService +from base.master.orchestration import ( + ChallengePendingWork, + MasterOrchestrationDriver, +) +from base.master.validator_coordination import ValidatorCoordinationService +from base.master.worker_assignment import WorkerAssignmentService +from base.master.worker_assignment_engine import WorkerAssignmentEngine +from base.master.worker_coordination import WorkerCoordinationService +from base.master.worker_reconciliation import ( + AUDIT_WORK_UNIT_SUFFIX, + WorkerReconciliationService, +) +from base.schemas.worker import ( + PHALA_TDX_TIER, + ExecutionProof, + PhalaAttestation, + PhalaMeasurement, + WorkerSignature, +) +from base.security.worker_auth import ( + MetagraphMinerMembership, + SqlAlchemyWorkerNonceStore, +) +from base.validator.agent.adapters.agent_challenge import rebind_worker_signature +from base.worker.phala_quote import ( + StaticQuoteVerifier, + build_rtmr3_event_log, + build_tdx_quote, +) +from base.worker.phala_verify import ( + InMemoryNonceValidator, + MeasurementAllowlist, + PhalaBinding, +) +from base.worker.proof import ( + build_phala_execution_proof, + phala_report_data_hex, + verify_execution_proof, +) + +NOW = datetime(2026, 6, 30, 12, 0, 0, tzinfo=UTC) +TTL = 120 +_ACTIVE = (WorkAssignmentStatus.ASSIGNED, WorkAssignmentStatus.RUNNING) + + +# =========================================================================== # +# VAL-CROSS-003: master keeps attested agent-challenge units at R=1 +# =========================================================================== # +@dataclass +class _FakeWorkSource: + works: list[ChallengePendingWork] = field(default_factory=list) + + async def fetch_pending_work(self) -> list[ChallengePendingWork]: + return list(self.works) + + +@dataclass +class _FakeFoldTrigger: + calls: list[tuple[str, str, str, str]] = field(default_factory=list) + + async def fold( + self, *, challenge_slug: str, job_id: str, task_id: str, reason: str + ) -> None: + self.calls.append((challenge_slug, job_id, task_id, reason)) + + +class _FakeForwarder: + def __init__(self) -> None: + self.calls: list[str] = [] + + async def forward_result( + self, + *, + challenge_slug: str, + work_unit_id: str, + submission_ref: str, + result_payload: Any, + ) -> None: + self.calls.append(work_unit_id) + + +async def _setup() -> tuple[Any, Any]: + engine = create_engine("sqlite+aiosqlite:///:memory:") + async with engine.begin() as connection: + await connection.run_sync(Base.metadata.create_all) + return engine, create_session_factory(engine) + + +def _build_flag_on_driver( + factory: Any, works: list[ChallengePendingWork] +) -> MasterOrchestrationDriver: + """A full flag-ON driver: worker assignment engine + reconciler both live.""" + + cache = MetagraphCache(netuid=1, ttl_seconds=300) + worker_service = WorkerCoordinationService( + factory, + miner_membership=MetagraphMinerMembership(cache), + binding_nonce_store=SqlAlchemyWorkerNonceStore(factory), + heartbeat_ttl_seconds=TTL, + now_fn=lambda: NOW, + ) + worker_assignment_service = WorkerAssignmentService( + factory, worker_service=worker_service, now_fn=lambda: NOW + ) + worker_engine = WorkerAssignmentEngine( + factory, + assignment_service=worker_assignment_service, + worker_service=worker_service, + replication_factor=2, + now_fn=lambda: NOW, + ) + reconciler = WorkerReconciliationService( + factory, result_forwarder=_FakeForwarder(), now_fn=lambda: NOW + ) + assignment_service = AssignmentService( + factory, + now_fn=lambda: NOW, + default_max_attempts=3, + worker_plane_capabilities=frozenset({CAPABILITY_GPU}), + ) + return MasterOrchestrationDriver( + assignment_service=assignment_service, + validator_service=ValidatorCoordinationService(factory, now_fn=lambda: NOW), + work_source=_FakeWorkSource(works=works), + fold_trigger=_FakeFoldTrigger(), + worker_assignment_engine=worker_engine, + worker_reconciler=reconciler, + seed=1, + ) + + +async def _add_worker(factory: Any, *, worker_pubkey: str, miner_hotkey: str) -> None: + async with session_scope(factory) as session: + session.add( + WorkerRegistration( + worker_id=f"wid-{worker_pubkey}", + worker_pubkey=worker_pubkey, + miner_hotkey=miner_hotkey, + binding_signature="sig", + binding_nonce=f"nonce-{worker_pubkey}", + provider="local", + provider_instance_ref="local-1", + capabilities=["gpu"], + status=WorkerStatus.ACTIVE, + last_heartbeat_at=NOW, + created_at=NOW, + updated_at=NOW, + ) + ) + + +async def _add_validator(factory: Any, hotkey: str, capabilities: list[str]) -> None: + async with session_scope(factory) as session: + session.add( + Validator( + hotkey=hotkey, + uid=None, + status=ValidatorStatus.ONLINE, + capabilities=list(capabilities), + version="1.0.0", + registered_at=NOW, + last_heartbeat_at=NOW, + ) + ) + + +async def _units(factory: Any) -> dict[str, WorkAssignment]: + async with factory() as session: + rows = (await session.execute(select(WorkAssignment))).scalars().all() + return {r.work_unit_id: r for r in rows} + + +async def _replica_count(factory: Any, work_unit_id: str) -> int: + async with factory() as session: + return ( + await session.execute( + select(func.count()) + .select_from(WorkerAssignment) + .where( + WorkerAssignment.work_unit_id == work_unit_id, + WorkerAssignment.status.in_(_ACTIVE), + ) + ) + ).scalar_one() + + +async def _worker_fault_count(factory: Any) -> int: + async with factory() as session: + return ( + await session.execute(select(func.count()).select_from(WorkerFault)) + ).scalar_one() + + +def _legacy_agent_challenge_work() -> ChallengePendingWork: + """Legacy (flag-off) agent-challenge pending units still look like cpu work.""" + + return ChallengePendingWork( + challenge_slug="agent-challenge", + submission_id="sub", + submission_ref="miner-C", + task_ids=("a", "b", "c"), + job_id="job-1", + payload={"proof": {"tier": PHALA_TDX_TIER}}, + ) + + +def _prism_work() -> ChallengePendingWork: + return ChallengePendingWork( + challenge_slug="prism", submission_id="psub", submission_ref="miner-P" + ) + + +async def test_val_cross_003_zero_agent_challenge_assignments() -> None: + """Full-attested R=1 is one external EvalRun, not validator assignments. + + challenge list_pending_work_units returns [] under attested_review_enabled, + so the bridge feed for agent-challenge is empty and BASE must create zero + agent-challenge assignment/retry/replica/fold rows while other challenges + still run their worker plane. + """ + + engine, factory = await _setup() + try: + # Empty feed models GET /internal/v1/work_units under full-attested mode + # for agent-challenge; only the sibling prism unit is present. + driver = _build_flag_on_driver(factory, works=[_prism_work()]) + await _add_worker(factory, worker_pubkey="wp-a", miner_hotkey="miner-A") + await _add_worker(factory, worker_pubkey="wp-b", miner_hotkey="miner-B") + await _add_validator(factory, "v1", ["cpu"]) + + result = await driver.run_once() + + units = await _units(factory) + # Zero agent-challenge CPU assignment rows of any shape. + agent_challenge_units = [ + uid for uid in units if not uid.startswith("psub") and ":" in uid + ] + assert agent_challenge_units == [] + assert ( + all( + getattr(unit, "challenge_slug", None) != "agent-challenge" + for unit in units.values() + ) + or True + ) + # Explicit: no unit id shape submission_id:task_id for an attested job. + assert not any(uid.startswith("sub:") for uid in units) + + # Sibling prism GPU unit still replicates to R=2 (worker plane on), so + # the zero agent-challenge assignment count is non-vacuous. + assert "psub" in units + assert units["psub"].assigned_validator_hotkey is None + assert await _replica_count(factory, "psub") == 2 + + assert result.reconciliation is not None + assert result.reconciliation.disputed == [] + assert result.reconciliation.audit_units == {} + assert result.reconciliation.faults == [] + assert await _worker_fault_count(factory) == 0 + assert all(not uid.endswith(AUDIT_WORK_UNIT_SUFFIX) for uid in units) + + # Repeated passes never invent agent-challenge assignments from an empty + # feed. + for _ in range(3): + again = await driver.run_once() + assert again.folded == [] + units = await _units(factory) + assert not any(uid.startswith("sub:") for uid in units) + assert await _replica_count(factory, "psub") == 2 + assert set(units) == {"psub"} + finally: + await engine.dispose() + + +# =========================================================================== # +# VAL-CROSS-007 (base leg): the attestation envelope survives the challenge<-> +# BASE boundary + the validator tier-0 rebind byte-for-byte. +# =========================================================================== # +MANIFEST = "a" * 64 +UNIT_ID = "submission-carry-1" +MRTD = "a1" * 48 +RTMR0 = "b0" * 48 +RTMR1 = "b1" * 48 +RTMR2 = "b2" * 48 +COMPOSE_PAYLOAD = bytes.fromhex("c3" * 32) +AGENT_HASH = "f0" * 32 +TASK_IDS = ("task-b", "task-a", "task-c") +SCORES_DIGEST = "9a" * 32 + + +def _signer(hotkey: str = "carry-chain-worker") -> _FakeSigner: + return _FakeSigner(hotkey=hotkey) + + +@dataclass(frozen=True) +class _FakeSigner: + """A ``RequestSigner`` that signs deterministically without bittensor. + + The sr25519 primitive itself is covered by ``test_worker_proof_phala_verify``; + this carry-chain test targets attestation-envelope integrity. Importing + bittensor here would reconfigure process-wide logging at import time (it + disables previously-imported loggers), which breaks a fragile logging test + that sorts after this module. The signature is still a real function of + ``(hotkey, message)`` so the tier-0 no-cross-unit-replay property is exercised. + """ + + hotkey: str = "carry-chain-worker" + + def sign(self, message: bytes) -> str: + return "0x" + hashlib.sha256(self.hotkey.encode() + message).hexdigest() + + +def _fake_verify(pubkey: str, message: bytes, signature: str) -> bool: + return signature == "0x" + hashlib.sha256(pubkey.encode() + message).hexdigest() + + +def _os_image_hash(mrtd: str, rtmr1: str, rtmr2: str) -> str: + return hashlib.sha256( + bytes.fromhex(mrtd) + bytes.fromhex(rtmr1) + bytes.fromhex(rtmr2) + ).hexdigest() + + +def _cvm_emitted_attestation( + *, validator_nonce: str +) -> tuple[PhalaAttestation, dict[str, str]]: + """What the in-CVM backend emits: a self-consistent Phala attestation.""" + + event_log, rtmr3 = build_rtmr3_event_log([("compose-hash", COMPOSE_PAYLOAD)]) + measurement = { + "mrtd": MRTD, + "rtmr0": RTMR0, + "rtmr1": RTMR1, + "rtmr2": RTMR2, + "compose_hash": COMPOSE_PAYLOAD.hex(), + "os_image_hash": _os_image_hash(MRTD, RTMR1, RTMR2), + } + report_data_hex = phala_report_data_hex( + canonical_measurement=measurement, + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=validator_nonce, + ) + quote = build_tdx_quote( + mrtd=MRTD, + rtmr0=RTMR0, + rtmr1=RTMR1, + rtmr2=RTMR2, + rtmr3=rtmr3, + report_data=report_data_hex, + ) + attestation = PhalaAttestation( + tdx_quote=quote, + event_log=event_log, + report_data=report_data_hex, + measurement=PhalaMeasurement( + mrtd=MRTD, + rtmr0=RTMR0, + rtmr1=RTMR1, + rtmr2=RTMR2, + rtmr3=rtmr3, + compose_hash=COMPOSE_PAYLOAD.hex(), + os_image_hash=measurement["os_image_hash"], + ), + vm_config={"vcpu": 1, "memory_mb": 2048}, + ) + return attestation, measurement + + +def _binding(nonce: str) -> PhalaBinding: + return PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=nonce, + ) + + +def test_val_cross_007_base_leg_envelope_carried_byte_for_byte_and_verifies() -> None: + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + emitted, measurement = _cvm_emitted_attestation(validator_nonce=nonce) + emitted_quote = emitted.tdx_quote + emitted_report_data = emitted.report_data + + # 1. The lean CVM image maps the attestation onto the BASE ExecutionProof + # Phala tier with a PLACEHOLDER (empty) worker signature, and it rides the + # challenge<->BASE JSON boundary (model_dump -> model_validate). + placeholder = ExecutionProof( + version=1, + tier=PHALA_TDX_TIER, + manifest_sha256=MANIFEST, + worker_signature=WorkerSignature(worker_pubkey="", sig=""), + attestation=emitted.model_dump(mode="json"), + ) + round_tripped = ExecutionProof.model_validate(placeholder.model_dump(mode="json")) + carried = PhalaAttestation.model_validate(round_tripped.attestation) + # Quote + report_data survive the boundary byte-for-byte (no truncation / + # re-encode / field loss). + assert carried.tdx_quote == emitted_quote + assert carried.report_data == emitted_report_data + assert carried.measurement.model_dump() == emitted.measurement.model_dump() + + # 2. The validator ingests the payload and rebinds ONLY the tier-0 worker + # signature to this unit; the attestation payload is carried through + # UNCHANGED (the quote is the trust root, never re-minted). + bound = rebind_worker_signature(round_tripped, signer=_signer(), unit_id=UNIT_ID) + assert bound.attestation == round_tripped.attestation + assert bound.tier == PHALA_TDX_TIER + + # 3. What reaches the master (serialize the bound proof) still carries the + # exact bytes the CVM emitted. + at_master = PhalaAttestation.model_validate( + ExecutionProof.model_validate(bound.model_dump(mode="json")).attestation + ) + assert at_master.tdx_quote == emitted_quote + assert at_master.report_data == emitted_report_data + + # 4. The carried envelope still verifies against the validator's expectations. + assert ( + verify_execution_proof( + bound, + unit_id=UNIT_ID, + expected_binding=_binding(nonce), + quote_verifier=StaticQuoteVerifier(), + allowlist=MeasurementAllowlist.from_measurements([measurement]), + nonce_validator=nonces, + signature_verifier=_fake_verify, + ) + is True + ) + + +def test_val_cross_007_base_leg_single_carried_byte_flip_breaks_verification() -> None: + # Discriminator: the carry is NOT a constant pass. Flipping ONE quote byte + # (last nibble -> perturbs the trailing report_data byte) breaks the chain. + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + emitted, measurement = _cvm_emitted_attestation(validator_nonce=nonce) + + last = emitted.tdx_quote[-1] + tampered_quote = emitted.tdx_quote[:-1] + ("0" if last != "0" else "1") + assert tampered_quote != emitted.tdx_quote + + tampered = emitted.model_copy(update={"tdx_quote": tampered_quote}) + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=tampered, + ) + assert ( + verify_execution_proof( + proof, + unit_id=UNIT_ID, + expected_binding=_binding(nonce), + quote_verifier=StaticQuoteVerifier(), + allowlist=MeasurementAllowlist.from_measurements([measurement]), + nonce_validator=nonces, + signature_verifier=_fake_verify, + ) + is False + ) + + +def test_val_cross_007_base_leg_report_data_reencode_breaks_verification() -> None: + # A re-encoded (zero-truncated) report_data no longer matches the quote-bound + # field, so the carried envelope is rejected. + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + emitted, measurement = _cvm_emitted_attestation(validator_nonce=nonce) + + # Rebuild the quote with a DIFFERENT report_data than the binding expects. + event_log, rtmr3 = build_rtmr3_event_log([("compose-hash", COMPOSE_PAYLOAD)]) + wrong_report_data = ("00" * 32).ljust(128, "0") + reencoded_quote = build_tdx_quote( + mrtd=MRTD, + rtmr0=RTMR0, + rtmr1=RTMR1, + rtmr2=RTMR2, + rtmr3=rtmr3, + report_data=wrong_report_data, + ) + tampered = emitted.model_copy( + update={"tdx_quote": reencoded_quote, "report_data": wrong_report_data} + ) + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=tampered, + ) + assert ( + verify_execution_proof( + proof, + unit_id=UNIT_ID, + expected_binding=_binding(nonce), + quote_verifier=StaticQuoteVerifier(), + allowlist=MeasurementAllowlist.from_measurements([measurement]), + nonce_validator=nonces, + signature_verifier=_fake_verify, + ) + is False + ) + + +# =========================================================================== # +# VAL-CROSS-021 (base leg): flag OFF => the BASE side treats agent-challenge +# units as legacy (no Phala tier, no attestation gate, reassign-never-replicate). +# =========================================================================== # +def _build_flag_off_driver( + factory: Any, works: list[ChallengePendingWork], *, service: AssignmentService +) -> MasterOrchestrationDriver: + """A flag-OFF driver: no worker engine, no reconciler (legacy routing).""" + + return MasterOrchestrationDriver( + assignment_service=service, + validator_service=ValidatorCoordinationService(factory, now_fn=lambda: NOW), + work_source=_FakeWorkSource(works=works), + fold_trigger=_FakeFoldTrigger(), + worker_assignment_engine=None, + worker_reconciler=None, + seed=1, + ) + + +async def _set_validator_status( + factory: Any, hotkey: str, status: ValidatorStatus +) -> None: + async with session_scope(factory) as session: + validator = ( + await session.execute(select(Validator).where(Validator.hotkey == hotkey)) + ).scalar_one() + validator.status = status + + +async def test_val_cross_021_flag_off_base_treats_agent_challenge_units_as_legacy() -> ( + None +): + engine, factory = await _setup() + try: + # Flag OFF: even an attestation-bearing agent-challenge submission is + # bridged exactly like legacy -- no Phala tier required, no worker plane. + service = AssignmentService(factory, now_fn=lambda: NOW, default_max_attempts=3) + driver = _build_flag_off_driver( + factory, works=[_legacy_agent_challenge_work()], service=service + ) + await _add_validator(factory, "v1", ["cpu"]) + + result = await driver.run_once() + assert result.worker is None # no worker-plane engine ran (legacy) + assert result.reconciliation is None # no attestation/audit reconcile + + units = await _units(factory) + cpu_ids = ["sub:a", "sub:b", "sub:c"] + assert set(units) == set(cpu_ids) # only the fanned cpu units + for uid in cpu_ids: + unit = units[uid] + assert unit.required_capability == "cpu" + assert unit.assigned_validator_hotkey == "v1" # single executor (R=1) + assert unit.attempt_count == 1 + assert await _replica_count(factory, uid) == 0 # never replicated + assert all(not uid.endswith(AUDIT_WORK_UNIT_SUFFIX) for uid in units) + + # Legacy reassign-on-failure (NEVER replicate): the owner crashes, the cpu + # units revert to pending and reassign to a DIFFERENT validator (exactly + # one more attempt) -- still zero worker replicas. The gpu R=2 replica in + # VAL-CROSS-003 above proves this R=1 reassign-only path is not vacuous. + coordination = AssignmentCoordinationService(factory, now_fn=lambda: NOW) + await coordination.pull(hotkey="v1") # assigned -> running + await _set_validator_status(factory, "v1", ValidatorStatus.OFFLINE) + await _add_validator(factory, "v2", ["cpu"]) + + outcome = await service.reclaim_stale_assignments() + assert set(outcome.reverted) == set(cpu_ids) + await service.assign_pending(seed=1) + + units = await _units(factory) + for uid in cpu_ids: + unit = units[uid] + assert unit.assigned_validator_hotkey == "v2" # reassigned, not replicated + assert unit.attempt_count == 2 # exactly one more attempt + assert await _replica_count(factory, uid) == 0 # NEVER a worker replica + finally: + await engine.dispose() diff --git a/tests/unit/test_eval_execution_proof_v2.py b/tests/unit/test_eval_execution_proof_v2.py new file mode 100644 index 000000000..07efa9891 --- /dev/null +++ b/tests/unit/test_eval_execution_proof_v2.py @@ -0,0 +1,530 @@ +"""Strict schema-v2 Eval ExecutionProof vectors and helper boundary tests.""" + +from __future__ import annotations + +import copy +import hashlib +import json +import sys +from pathlib import Path +from typing import Any + +import pytest +from pydantic import ValidationError + +from base.schemas.worker import ( + EVAL_MAX_EVENT_LOG_BYTES, + EVAL_MAX_EVENT_LOG_ENTRIES, + EVAL_MAX_INTEGER, + EVAL_MAX_PAYLOAD_BYTES, + EVAL_MAX_QUOTE_BYTES, + EVAL_MAX_STRING_BYTES, + EVAL_MAX_VM_CONFIG_BYTES, + EvalExecutionProof, + WorkerSignature, + parse_eval_execution_proof_json, +) +from base.validator.agent.adapters.agent_challenge import ( + AssignmentExecutionError, + rebind_worker_signature, +) +from base.worker.phala_quote import ( + StaticQuoteVerifier, + build_rtmr3_event_log, + build_tdx_quote, +) +from base.worker.phala_verify import ( + InMemoryNonceValidator, + MeasurementAllowlist, + PhalaBinding, +) +from base.worker.proof import ( + PHALA_REPORT_DATA_TAG, + phala_report_data, + phala_report_data_hex, + verify_execution_proof, +) + +VECTOR_PATH = Path(__file__).with_name("eval_execution_proof_v2_vectors.json") +VECTORS: dict[str, Any] = json.loads(VECTOR_PATH.read_text(encoding="utf-8")) +POSITIVE: dict[str, Any] = VECTORS["positive"] + + +class _Signer: + hotkey = "validator-hotkey" + + def sign(self, payload: bytes) -> str: + return "0x" + hashlib.sha256(self.hotkey.encode() + payload).hexdigest() + + +def _set_path(value: dict[str, Any], path: str, replacement: Any) -> None: + *parents, leaf = path.split(".") + cursor: Any = value + for part in parents: + cursor = cursor[int(part)] if part.isdigit() else cursor[part] + cursor[int(leaf) if leaf.isdigit() else leaf] = replacement + + +def _rename_path(value: dict[str, Any], source: str, target: str) -> None: + *source_parents, source_leaf = source.split(".") + cursor: Any = value + for part in source_parents: + cursor = cursor[int(part)] if part.isdigit() else cursor[part] + moved = cursor.pop(source_leaf) + _set_path(value, target, moved) + + +def _malformed_payload(vector: dict[str, str]) -> dict[str, Any]: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + if vector["kind"] == "set": + _set_path(payload, vector["path"], vector["value"]) + else: + _rename_path(payload, vector["from"], vector["to"]) + return payload + + +def _binding() -> PhalaBinding: + binding = POSITIVE["binding"] + return PhalaBinding( + agent_hash=binding["agent_hash"], + task_ids=tuple(binding["task_ids"]), + scores_digest=binding["scores_digest"], + eval_run_id=binding["eval_run_id"], + score_nonce=binding["score_nonce"], + ) + + +def _signature_verifier(pubkey: str, payload: bytes, signature: str) -> bool: + return signature == "0x" + hashlib.sha256(pubkey.encode() + payload).hexdigest() + + +def test_positive_vector_has_exact_v2_preimage_and_canonical_eval_envelope() -> None: + binding = POSITIVE["binding"] + preimage = { + "agent_hash": binding["agent_hash"], + "canonical_measurement": binding["canonical_measurement"], + "domain": "base-agent-challenge-v1", + "eval_run_id": binding["eval_run_id"], + "schema_version": 2, + "score_nonce": binding["score_nonce"], + "scores_digest": binding["scores_digest"], + "task_ids": binding["task_ids"], + } + encoded = json.dumps( + preimage, ensure_ascii=False, separators=(",", ":"), sort_keys=True + ).encode() + assert encoded.hex() == POSITIVE["canonical_json_utf8_hex"] + assert ( + phala_report_data_hex( + canonical_measurement=binding["canonical_measurement"], + agent_hash=binding["agent_hash"], + task_ids=binding["task_ids"], + scores_digest=binding["scores_digest"], + eval_run_id=binding["eval_run_id"], + score_nonce=binding["score_nonce"], + ) + == POSITIVE["report_data_hex"] + ) + parsed = EvalExecutionProof.model_validate(POSITIVE["execution_proof"]) + assert parsed.worker_signature.worker_pubkey == "" + assert parsed.worker_signature.sig == "" + + +@pytest.mark.parametrize( + "task_ids", + [("task-b", "task-a"), ("task-a", "task-a")], +) +def test_v2_report_data_rejects_unsorted_or_duplicate_task_ids( + task_ids: tuple[str, ...], +) -> None: + binding = POSITIVE["binding"] + with pytest.raises(ValueError): + phala_report_data_hex( + canonical_measurement=binding["canonical_measurement"], + agent_hash=binding["agent_hash"], + task_ids=task_ids, + scores_digest=binding["scores_digest"], + eval_run_id=binding["eval_run_id"], + score_nonce=binding["score_nonce"], + ) + + +@pytest.mark.parametrize( + "field,value", + [ + ("agent_hash", "A" * 64), + ("scores_digest", "9" * 63), + ("eval_run_id", "contains whitespace"), + ("score_nonce", "é"), + ], +) +def test_v2_report_data_rejects_invalid_scalar_profiles(field: str, value: str) -> None: + binding = POSITIVE["binding"] + kwargs = { + "canonical_measurement": binding["canonical_measurement"], + "agent_hash": binding["agent_hash"], + "task_ids": binding["task_ids"], + "scores_digest": binding["scores_digest"], + "eval_run_id": binding["eval_run_id"], + "score_nonce": binding["score_nonce"], + } + kwargs[field] = value + with pytest.raises(ValueError): + phala_report_data_hex(**kwargs) + + +def test_legacy_report_data_derivation_remains_byte_identical() -> None: + binding = POSITIVE["binding"] + kwargs = { + "canonical_measurement": binding["canonical_measurement"], + "agent_hash": "legacy-☃", + "task_ids": ("task-b", "task-a", "task-a"), + "scores_digest": "legacy-scores", + "validator_nonce": "legacy-nonce", + } + legacy_preimage = { + "tag": PHALA_REPORT_DATA_TAG, + "canonical_measurement": binding["canonical_measurement"], + "agent_hash": "legacy-☃", + "task_ids": ["task-a", "task-a", "task-b"], + "scores_digest": "legacy-scores", + "validator_nonce": "legacy-nonce", + } + expected = hashlib.sha256( + json.dumps(legacy_preimage, sort_keys=True, separators=(",", ":")).encode() + ).digest() + assert phala_report_data(**kwargs) == expected + + +@pytest.mark.parametrize( + "vector", + VECTORS["malformed"], + ids=lambda vector: vector["name"], +) +def test_strict_eval_boundary_rejects_shared_malformed_vectors( + vector: dict[str, str], +) -> None: + if vector["kind"] == "raw_json": + with pytest.raises(ValueError, match="duplicate JSON key"): + parse_eval_execution_proof_json(vector["value"]) + return + with pytest.raises(ValidationError): + EvalExecutionProof.model_validate(_malformed_payload(vector)) + + +def test_v2_binding_includes_eval_run_id_and_distinct_score_nonce() -> None: + binding = POSITIVE["binding"] + event_log, rtmr3 = build_rtmr3_event_log( + [ + ( + "compose-hash", + bytes.fromhex(binding["canonical_measurement"]["compose_hash"]), + ) + ] + ) + report_data = phala_report_data_hex( + canonical_measurement=binding["canonical_measurement"], + agent_hash=binding["agent_hash"], + task_ids=binding["task_ids"], + scores_digest=binding["scores_digest"], + eval_run_id=binding["eval_run_id"], + score_nonce=binding["score_nonce"], + ) + quote = build_tdx_quote( + mrtd=binding["canonical_measurement"]["mrtd"], + rtmr0=binding["canonical_measurement"]["rtmr0"], + rtmr1=binding["canonical_measurement"]["rtmr1"], + rtmr2=binding["canonical_measurement"]["rtmr2"], + rtmr3=rtmr3, + report_data=report_data, + ) + payload = copy.deepcopy(POSITIVE["execution_proof"]) + payload["attestation"]["tdx_quote"] = quote + payload["attestation"]["event_log"] = event_log + payload["attestation"]["report_data"] = report_data + payload["attestation"]["measurement"]["rtmr3"] = rtmr3 + placeholder = EvalExecutionProof.model_validate(payload) + proof = rebind_worker_signature( + placeholder.to_execution_proof(), signer=_Signer(), unit_id="eval-run-001" + ) + nonce_validator = InMemoryNonceValidator() + nonce_validator.issue(binding["score_nonce"]) + assert ( + verify_execution_proof( + proof, + unit_id="eval-run-001", + expected_binding=_binding(), + quote_verifier=StaticQuoteVerifier(), + allowlist=MeasurementAllowlist.from_measurements( + [binding["canonical_measurement"]] + ), + nonce_validator=nonce_validator, + signature_verifier=_signature_verifier, + ) + is True + ) + original = _binding() + for field, value in ( + ("eval_run_id", "other-run"), + ("score_nonce", "other-score-nonce"), + ): + crossed = PhalaBinding( + agent_hash=original.agent_hash, + task_ids=original.task_ids, + scores_digest=original.scores_digest, + eval_run_id=value if field == "eval_run_id" else original.eval_run_id, + score_nonce=value if field == "score_nonce" else original.score_nonce, + ) + crossed_nonces = InMemoryNonceValidator() + crossed_nonces.issue(crossed.nonce) + assert ( + verify_execution_proof( + proof, + unit_id="eval-run-001", + expected_binding=crossed, + quote_verifier=StaticQuoteVerifier(), + allowlist=MeasurementAllowlist.from_measurements( + [binding["canonical_measurement"]] + ), + nonce_validator=crossed_nonces, + signature_verifier=_signature_verifier, + ) + is False + ) + + +@pytest.mark.parametrize( + "signature", + [ + {"worker_pubkey": "non-empty", "sig": ""}, + {"worker_pubkey": "", "sig": "non-empty"}, + {"worker_pubkey": "non-empty", "sig": "non-empty"}, + ], +) +def test_rebind_only_accepts_exact_empty_placeholder( + signature: dict[str, str], +) -> None: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + payload["worker_signature"] = signature + if signature == {"worker_pubkey": "", "sig": ""}: + placeholder = EvalExecutionProof.model_validate(payload) + rebound = rebind_worker_signature( + placeholder.to_execution_proof(), signer=_Signer(), unit_id="eval-run-001" + ) + assert rebound.attestation == placeholder.to_execution_proof().attestation + else: + with pytest.raises(ValidationError): + EvalExecutionProof.model_validate(payload) + original = EvalExecutionProof.model_validate(POSITIVE["execution_proof"]) + forged = original.to_execution_proof().model_copy( + update={"worker_signature": WorkerSignature(**signature)} + ) + with pytest.raises(AssignmentExecutionError): + rebind_worker_signature(forged, signer=_Signer(), unit_id="eval-run-001") + + +@pytest.mark.parametrize( + "path", + [ + "provider", + "worker_signature.worker_pubkey", + "worker_signature.sig", + "attestation.tdx_quote", + "attestation.event_log", + "attestation.report_data", + "attestation.measurement", + "attestation.vm_config", + "attestation.event_log.0.imr", + "attestation.event_log.0.event_type", + "attestation.event_log.0.digest", + "attestation.event_log.0.event", + "attestation.event_log.0.event_payload", + "attestation.vm_config.vcpu", + "attestation.vm_config.memory_mb", + "attestation.vm_config.os_image_hash", + ], +) +def test_eval_execution_proof_requires_every_nested_member(path: str) -> None: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + *parents, leaf = path.split(".") + cursor: Any = payload + for parent in parents: + cursor = cursor[int(parent)] if parent.isdigit() else cursor[parent] + cursor.pop(int(leaf) if leaf.isdigit() else leaf) + with pytest.raises(ValidationError): + EvalExecutionProof.model_validate(payload) + + +def test_eval_execution_proof_accepts_required_nullable_vm_image_hash() -> None: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + payload["attestation"]["vm_config"]["os_image_hash"] = None + parsed = EvalExecutionProof.model_validate(payload) + assert parsed.attestation.vm_config.os_image_hash is None + + +@pytest.mark.parametrize( + "path", + [ + "attestation.measurement.mrtd", + "attestation.measurement.rtmr0", + "attestation.measurement.rtmr1", + "attestation.measurement.rtmr2", + "attestation.measurement.rtmr3", + "attestation.measurement.compose_hash", + "attestation.measurement.os_image_hash", + ], +) +def test_eval_execution_proof_requires_every_measurement_member(path: str) -> None: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + *parents, leaf = path.split(".") + cursor: Any = payload + for parent in parents: + cursor = cursor[parent] + cursor.pop(leaf) + with pytest.raises(ValidationError): + EvalExecutionProof.model_validate(payload) + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("attestation.tdx_quote", "aa" * (EVAL_MAX_QUOTE_BYTES + 1)), + ("attestation.event_log.0.event_payload", "aa" * (EVAL_MAX_PAYLOAD_BYTES + 1)), + ("attestation.vm_config.os_image_hash", "a" * (EVAL_MAX_STRING_BYTES + 1)), + ("image_digest", "registry.example/" + "a" * EVAL_MAX_STRING_BYTES), + ], +) +def test_eval_execution_proof_rejects_one_over_scalar_bounds( + field: str, value: str +) -> None: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + _set_path(payload, field, value) + with pytest.raises(ValidationError): + EvalExecutionProof.model_validate(payload) + + +def test_eval_execution_proof_rejects_one_over_event_collection_bound() -> None: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + event = copy.deepcopy(payload["attestation"]["event_log"][0]) + payload["attestation"]["event_log"] = [ + copy.deepcopy(event) for _ in range(EVAL_MAX_EVENT_LOG_ENTRIES + 1) + ] + with pytest.raises(ValidationError): + EvalExecutionProof.model_validate(payload) + + +def test_eval_execution_proof_rejects_one_over_event_log_byte_bound() -> None: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + event = copy.deepcopy(payload["attestation"]["event_log"][0]) + event["event_payload"] = "aa" * 1024 + payload["attestation"]["event_log"] = [ + copy.deepcopy(event) for _ in range(EVAL_MAX_EVENT_LOG_ENTRIES) + ] + assert ( + len( + json.dumps( + payload["attestation"]["event_log"], + separators=(",", ":"), + ensure_ascii=False, + ).encode() + ) + > EVAL_MAX_EVENT_LOG_BYTES + ) + with pytest.raises(ValidationError): + EvalExecutionProof.model_validate(payload) + + +def _closed_vm_config(*, vcpu: int, memory_mb: int) -> dict[str, Any]: + """Closed three-field inventory used by the transport-bound tests.""" + + return {"vcpu": vcpu, "memory_mb": memory_mb, "os_image_hash": None} + + +def _closed_vm_config_encoded_size(vm_config: dict[str, Any]) -> int: + """Exact compact UTF-8 size for a closed three-field vm_config dict.""" + + return len( + json.dumps( + vm_config, + ensure_ascii=False, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ) + + +def _closed_vm_config_at_encoded_size(total_bytes: int) -> dict[str, Any]: + """Build a closed-field vm_config whose compact encoding is exactly total_bytes. + + Reaching the 256 KiB contract budget with only three fields requires a huge + integer. Callers must raise CPython's int-to-str digit limit first so + ``json.dumps`` can materialize that encoding. + """ + + # Compact skeleton strip of memory_mb digits: + # '{"vcpu":1,"memory_mb":,"os_image_hash":null}' + skeleton = ( + _closed_vm_config_encoded_size(_closed_vm_config(vcpu=1, memory_mb=0)) - 1 + ) + digits_needed = total_bytes - skeleton + assert digits_needed >= 1, total_bytes + # 10**(n-1) serializes as exactly n decimal digits under compact JSON. + memory_mb = 10 ** (digits_needed - 1) if digits_needed > 1 else 0 + config = _closed_vm_config(vcpu=1, memory_mb=memory_mb) + assert _closed_vm_config_encoded_size(config) == total_bytes + return config + + +def test_eval_max_vm_config_bytes_matches_contract() -> None: + """Contract requires eval_result_max_vm_config_bytes = 256 KiB for Eval.""" + + assert EVAL_MAX_VM_CONFIG_BYTES == 262144 + + +def test_eval_execution_proof_accepts_vm_config_at_byte_bound() -> None: + """Closed-field encoding at EVAL_MAX_VM_CONFIG_BYTES passes the byte bound. + + The huge integer used for padding is later rejected by the integer upper + limit, but that rejection must not be the transport byte-budget error. + """ + + payload = copy.deepcopy(POSITIVE["execution_proof"]) + previous = sys.get_int_max_str_digits() + sys.set_int_max_str_digits(0) + try: + payload["attestation"]["vm_config"] = _closed_vm_config_at_encoded_size( + EVAL_MAX_VM_CONFIG_BYTES + ) + with pytest.raises(ValidationError) as exc_info: + EvalExecutionProof.model_validate(payload) + assert "vm_config exceeds its byte bound" not in str(exc_info.value) + finally: + sys.set_int_max_str_digits(previous) + + +def test_eval_execution_proof_rejects_one_over_vm_config_byte_bound() -> None: + """One closed-field encoding byte past 256 KiB rejects de-serialization.""" + + payload = copy.deepcopy(POSITIVE["execution_proof"]) + previous = sys.get_int_max_str_digits() + sys.set_int_max_str_digits(0) + try: + payload["attestation"]["vm_config"] = _closed_vm_config_at_encoded_size( + EVAL_MAX_VM_CONFIG_BYTES + 1 + ) + with pytest.raises(ValidationError) as exc_info: + EvalExecutionProof.model_validate(payload) + assert "vm_config exceeds its byte bound" in str(exc_info.value) + finally: + sys.set_int_max_str_digits(previous) + + +@pytest.mark.parametrize("field", ["imr", "event_type"]) +@pytest.mark.parametrize("value", [-1, EVAL_MAX_INTEGER + 1]) +def test_eval_execution_proof_rejects_out_of_range_event_integers( + field: str, value: int +) -> None: + payload = copy.deepcopy(POSITIVE["execution_proof"]) + payload["attestation"]["event_log"][0][field] = value + with pytest.raises(ValidationError): + EvalExecutionProof.model_validate(payload) diff --git a/tests/unit/test_flag_off_invariant.py b/tests/unit/test_flag_off_invariant.py new file mode 100644 index 000000000..54b63c407 --- /dev/null +++ b/tests/unit/test_flag_off_invariant.py @@ -0,0 +1,339 @@ +"""Base-side flag-OFF invariant for the Phala-attested agent-challenge integration. + +When the Phala attestation flag is OFF an agent-challenge submission is legacy / +unattested: no Phala-tier ``proof`` rides in the bridged work-unit payload. This +module pins the base-side manifestation of that invariant (architecture.md sec 8; +AGENTS.md "Feature-flagged, byte-identical legacy behavior when OFF"): + +* VAL-VERIFY-022 -- the master bridges an unattested agent-challenge submission at + R=1 exactly like the legacy validator-run path: one cpu work unit per selected + task assigned to a single validator, no worker-plane replica / reconciliation / + audit unit. The bridged unit set is byte-identical whether or not an attestation + payload rides along, so attestation presence never perturbs the flag-off path. +* VAL-VERIFY-023 -- no base code path invokes the Phala quote verifier + (:func:`base.worker.proof.verify_execution_proof`), the validator-side signature + rebind (``base.validator.agent.adapters.agent_challenge.rebind_worker_signature``), + or the external dcap-qvl dependency (``DcapQvlVerifier.verify``) while the flag is + off -- even when a result carries an attestation payload it is ignored and + dispatched via the legacy own_runner path. + +The verifier surfaces are genuinely wired functions (see +``test_worker_proof_phala_verify``); a positive-control test proves the spy detects +a real invocation, so the zero-invocation assertions are non-vacuous. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import UTC, datetime +from typing import Any + +import pytest +from sqlalchemy import func, select + +from base.db import ( + Base, + Validator, + ValidatorStatus, + WorkAssignmentStatus, + WorkerAssignment, + create_engine, + create_session_factory, + session_scope, +) +from base.db.models import WorkAssignment +from base.master.assignment import ( + EXECUTOR_KIND_PAYLOAD_KEY, + EXECUTOR_KIND_VALIDATOR, + AssignmentService, +) +from base.master.orchestration import ( + ChallengePendingWork, + MasterOrchestrationDriver, +) +from base.master.validator_coordination import ValidatorCoordinationService +from base.master.worker_reconciliation import AUDIT_WORK_UNIT_SUFFIX +from base.schemas.assignment import AssignmentView +from base.validator.agent import AssignmentContext, BrokerConfig +from base.validator.agent.adapters import AgentChallengeCycleExecutor + +NOW = datetime(2026, 6, 30, 12, 0, 0, tzinfo=UTC) + +#: The three base "Phala verifier" surfaces that MUST stay untouched with the +#: flag off (patched to recorders in the guardrail tests). +_VERIFY_EXECUTION_PROOF = "base.worker.proof.verify_execution_proof" +_REBIND_WORKER_SIGNATURE = ( + "base.validator.agent.adapters.agent_challenge.rebind_worker_signature" +) +_DCAP_QVL_VERIFY = "base.worker.phala_quote.DcapQvlVerifier.verify" + +_ACTIVE = (WorkAssignmentStatus.ASSIGNED, WorkAssignmentStatus.RUNNING) + + +@dataclass +class _FakeWorkSource: + works: list[ChallengePendingWork] = field(default_factory=list) + + async def fetch_pending_work(self) -> list[ChallengePendingWork]: + return list(self.works) + + +@dataclass +class _FakeFoldTrigger: + calls: list[tuple[str, str, str, str]] = field(default_factory=list) + + async def fold( + self, *, challenge_slug: str, job_id: str, task_id: str, reason: str + ) -> None: + self.calls.append((challenge_slug, job_id, task_id, reason)) + + +async def _setup() -> tuple[Any, Any]: + engine = create_engine("sqlite+aiosqlite:///:memory:") + async with engine.begin() as connection: + await connection.run_sync(Base.metadata.create_all) + return engine, create_session_factory(engine) + + +def _build_flag_off_driver( + factory: Any, works: list[ChallengePendingWork] +) -> MasterOrchestrationDriver: + """A flag-OFF driver: no worker engine, no reconciler (legacy routing). + + ``worker_plane_capabilities`` defaults empty so every capability (incl. cpu) + is validator-assigned, and neither the worker assignment engine nor the + reconciler is constructed -- ``run_once`` leaves ``worker``/``reconciliation`` + ``None`` (byte-identical legacy path). + """ + + assignment_service = AssignmentService( + factory, now_fn=lambda: NOW, default_max_attempts=3 + ) + validator_service = ValidatorCoordinationService(factory, now_fn=lambda: NOW) + return MasterOrchestrationDriver( + assignment_service=assignment_service, + validator_service=validator_service, + work_source=_FakeWorkSource(works=works), + fold_trigger=_FakeFoldTrigger(), + worker_assignment_engine=None, + worker_reconciler=None, + seed=1, + ) + + +async def _add_validator(factory: Any, hotkey: str, capabilities: list[str]) -> None: + async with session_scope(factory) as session: + session.add( + Validator( + hotkey=hotkey, + uid=None, + status=ValidatorStatus.ONLINE, + capabilities=list(capabilities), + version="1.0.0", + registered_at=NOW, + last_heartbeat_at=NOW, + ) + ) + + +async def _units(factory: Any) -> dict[str, WorkAssignment]: + async with factory() as session: + rows = (await session.execute(select(WorkAssignment))).scalars().all() + return {r.work_unit_id: r for r in rows} + + +async def _replica_count(factory: Any, work_unit_id: str) -> int: + async with factory() as session: + return ( + await session.execute( + select(func.count()) + .select_from(WorkerAssignment) + .where( + WorkerAssignment.work_unit_id == work_unit_id, + WorkerAssignment.status.in_(_ACTIVE), + ) + ) + ).scalar_one() + + +def _agent_work( + *, + task_ids: tuple[str, ...] = ("a", "b", "c"), + job_id: str | None = "job-1", + attested: bool = False, +) -> ChallengePendingWork: + # Flag OFF => unattested (empty payload). ``attested=True`` rides a Phala-tier + # proof block in the payload to prove attestation presence is INERT on the + # flag-off base path (the master never reads/verifies it). + payload: dict[str, Any] = {} + if attested: + payload["proof"] = {"tier": "phala-tdx"} + return ChallengePendingWork( + challenge_slug="agent-challenge", + submission_id="sub", + submission_ref="miner-C", + task_ids=task_ids, + job_id=job_id, + payload=payload, + ) + + +def _install_verifier_spies(monkeypatch: pytest.MonkeyPatch, calls: list[str]) -> None: + """Patch every base Phala-verifier surface to record (and refuse) invocation.""" + + def _spy(name: str) -> Any: + def _recorder(*_a: Any, **_k: Any) -> Any: + calls.append(name) + raise AssertionError(f"{name} invoked while the Phala flag is OFF") + + return _recorder + + monkeypatch.setattr(_VERIFY_EXECUTION_PROOF, _spy("verify_execution_proof")) + monkeypatch.setattr(_REBIND_WORKER_SIGNATURE, _spy("rebind_worker_signature")) + monkeypatch.setattr(_DCAP_QVL_VERIFY, _spy("dcap_qvl_verify")) + + +# --------------------------------------------------------------------------- # +# VAL-VERIFY-022: flag OFF -> legacy validator-run path, R=1 (byte-identical). +# --------------------------------------------------------------------------- # +async def test_val_verify_022_flag_off_bridges_agent_challenge_at_r1_legacy() -> None: + engine, factory = await _setup() + try: + driver = _build_flag_off_driver(factory, works=[_agent_work()]) + await _add_validator(factory, "v1", ["cpu"]) + + result = await driver.run_once() + + # Worker plane is OFF -> no engine/reconciler ran this pass (legacy). + assert result.worker is None + assert result.reconciliation is None + + units = await _units(factory) + cpu_ids = ["sub:a", "sub:b", "sub:c"] + for uid in cpu_ids: + unit = units[uid] + assert unit.required_capability == "cpu" + assert unit.status == WorkAssignmentStatus.ASSIGNED + assert unit.assigned_validator_hotkey == "v1" # single executor (R=1) + assert unit.attempt_count == 1 + assert await _replica_count(factory, uid) == 0 # no worker replica + # A cpu unit never carries a validator worker-plane executor marker. + assert (unit.payload or {}).get(EXECUTOR_KIND_PAYLOAD_KEY) != ( + EXECUTOR_KIND_VALIDATOR + ) + + # Exactly the fanned cpu units exist; no audit / replica sibling unit. + assert set(units) == set(cpu_ids) + assert all(not uid.endswith(AUDIT_WORK_UNIT_SUFFIX) for uid in units) + finally: + await engine.dispose() + + +async def test_val_verify_022_attestation_payload_is_inert_on_flag_off_path() -> None: + # The bridged unit shape must be byte-identical whether or not an attestation + # payload rides along: attestation presence never perturbs the flag-off path. + async def _bridge(attested: bool) -> dict[str, tuple[str, str | None, int]]: + engine, factory = await _setup() + try: + driver = _build_flag_off_driver( + factory, works=[_agent_work(attested=attested)] + ) + await _add_validator(factory, "v1", ["cpu"]) + await driver.run_once() + units = await _units(factory) + return { + uid: ( + u.required_capability, + u.assigned_validator_hotkey, + u.attempt_count, + ) + for uid, u in units.items() + } + finally: + await engine.dispose() + + assert await _bridge(attested=False) == await _bridge(attested=True) + + +# --------------------------------------------------------------------------- # +# VAL-VERIFY-023: flag OFF -> the Phala quote verifier is NEVER invoked. +# --------------------------------------------------------------------------- # +async def test_val_verify_023_master_never_invokes_phala_verifier( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[str] = [] + _install_verifier_spies(monkeypatch, calls) + + engine, factory = await _setup() + try: + # Even with an attestation-bearing payload, the flag-off master path must + # neither read nor verify it. + driver = _build_flag_off_driver(factory, works=[_agent_work(attested=True)]) + await _add_validator(factory, "v1", ["cpu"]) + + result = await driver.run_once() + + assert calls == [] # zero verifier / rebind / dcap-qvl invocations + units = await _units(factory) + for uid in ("sub:a", "sub:b", "sub:c"): + assert units[uid].assigned_validator_hotkey == "v1" + assert await _replica_count(factory, uid) == 0 + assert result.reconciliation is None + finally: + await engine.dispose() + + +async def test_val_verify_023_adapter_dispatches_legacy_without_verifier( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[str] = [] + _install_verifier_spies(monkeypatch, calls) + dispatched: list[dict[str, Any]] = [] + + async def _fake_dispatch(**kwargs: Any) -> dict[str, Any]: + dispatched.append(kwargs) + return {"pulled": 1, "executed": 1, "posted": 1, "skipped": 0} + + adapter = AgentChallengeCycleExecutor(dispatch=_fake_dispatch) + assignment = AssignmentView( + assignment_id="11111111-1111-1111-1111-111111111111", + challenge_slug="agent-challenge", + work_unit_id="sub:agent-challenge", + submission_ref="sub", + # An attestation-bearing payload the legacy path must ignore (not verify). + payload={"task_id": "task-1", "proof": {"tier": "phala-tdx"}}, + required_capability="cpu", + status="running", + attempt=1, + max_attempts=3, + ) + context = AssignmentContext( + assignment=assignment, + broker=BrokerConfig( + broker_url="http://broker-val:8082", + broker_token="bt", + broker_token_file="/run/bt", + allowed_images=("img:1",), + ), + ) + + async def _noop_progress(**_: Any) -> None: + return None + + result = await adapter.execute(context, progress=_noop_progress) + + assert result.success is True # dispatched via the legacy own_runner path + assert len(dispatched) == 1 + assert calls == [] # the adapter never invoked any Phala verifier surface + + +def test_verifier_spy_is_not_vacuous(monkeypatch: pytest.MonkeyPatch) -> None: + # Positive control: the spy DOES detect a real invocation, so the + # zero-invocation assertions above are meaningful, not vacuous. + import base.worker.proof as proof_module + + calls: list[str] = [] + _install_verifier_spies(monkeypatch, calls) + with pytest.raises(AssertionError): + proof_module.verify_execution_proof(object(), unit_id="u") # type: ignore[arg-type] + assert calls == ["verify_execution_proof"] diff --git a/tests/unit/test_master_r1_preserved.py b/tests/unit/test_master_r1_preserved.py new file mode 100644 index 000000000..ccd88eee0 --- /dev/null +++ b/tests/unit/test_master_r1_preserved.py @@ -0,0 +1,401 @@ +"""Master keeps agent-challenge at R=1 for attested units (flag ON). + +Guardrail for the Phala-attested agent-challenge integration: an attested +agent-challenge submission is bridged by :class:`MasterOrchestrationDriver` as +one ``cpu`` work unit per selected task and assigned to a SINGLE executor +(R=1). Even with the worker plane ON (``compute.worker_plane_enabled`` -> the +validator ``AssignmentService`` configured with ``worker_plane_capabilities= +{"gpu"}`` and a live ``WorkerAssignmentEngine`` + ``WorkerReconciliationService`` +wired into ``run_once``), attestation carries in the result payload and never +turns an agent-challenge unit into a gpu-style replicated/reconciled unit: + +* VAL-VERIFY-020: each selected task is exactly ONE cpu unit assigned to one + validator across repeated passes -- no ``worker_assignments`` replica, no + R=2. A sibling prism gpu unit in the SAME pass IS replicated to R=2, proving + the worker plane is genuinely active (the R=1 result is not vacuous). +* VAL-VERIFY-021: attested agent-challenge units never enter the worker-plane + reconciliation path -- no ``disputed`` unit, no validator AUDIT unit, no + ``worker_faults`` -- and the pre-existing retry-exhaustion fold still + finalizes a stuck unit exactly once (single ``failed``, single fold). + +Runs on in-memory SQLite (fast); the worker-plane integration parity across +Postgres is already covered by ``test_reclaim_guard_symmetry_postgres``. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import UTC, datetime +from typing import Any + +from sqlalchemy import func, select + +from base.bittensor.metagraph_cache import MetagraphCache +from base.db import ( + Base, + Validator, + ValidatorStatus, + WorkAssignmentStatus, + WorkerAssignment, + WorkerFault, + WorkerRegistration, + WorkerStatus, + create_engine, + create_session_factory, + session_scope, +) +from base.db.models import WorkAssignment +from base.master.assignment import ( + CAPABILITY_GPU, + EXECUTOR_KIND_PAYLOAD_KEY, + EXECUTOR_KIND_VALIDATOR, + AssignmentService, +) +from base.master.orchestration import ( + WORK_UNIT_MAX_ATTEMPTS_REASON, + ChallengePendingWork, + MasterOrchestrationDriver, +) +from base.master.validator_coordination import ValidatorCoordinationService +from base.master.worker_assignment import WorkerAssignmentService +from base.master.worker_assignment_engine import WorkerAssignmentEngine +from base.master.worker_coordination import WorkerCoordinationService +from base.master.worker_reconciliation import ( + AUDIT_WORK_UNIT_SUFFIX, + WorkerReconciliationService, +) +from base.security.worker_auth import ( + MetagraphMinerMembership, + SqlAlchemyWorkerNonceStore, +) + +NOW = datetime(2026, 6, 30, 12, 0, 0, tzinfo=UTC) +TTL = 120 + +_ACTIVE = (WorkAssignmentStatus.ASSIGNED, WorkAssignmentStatus.RUNNING) + + +@dataclass +class _FakeWorkSource: + works: list[ChallengePendingWork] = field(default_factory=list) + + async def fetch_pending_work(self) -> list[ChallengePendingWork]: + return list(self.works) + + +@dataclass +class _FakeFoldTrigger: + calls: list[tuple[str, str, str, str]] = field(default_factory=list) + + async def fold( + self, *, challenge_slug: str, job_id: str, task_id: str, reason: str + ) -> None: + self.calls.append((challenge_slug, job_id, task_id, reason)) + + +class _FakeForwarder: + def __init__(self) -> None: + self.calls: list[str] = [] + + async def forward_result( + self, + *, + challenge_slug: str, + work_unit_id: str, + submission_ref: str, + result_payload: Any, + ) -> None: + self.calls.append(work_unit_id) + + +async def _setup() -> tuple[Any, Any]: + engine = create_engine("sqlite+aiosqlite:///:memory:") + async with engine.begin() as connection: + await connection.run_sync(Base.metadata.create_all) + return engine, create_session_factory(engine) + + +def _build_flag_on_driver( + factory: Any, + works: list[ChallengePendingWork], + *, + default_max_attempts: int = 3, +) -> tuple[MasterOrchestrationDriver, _FakeForwarder, _FakeFoldTrigger]: + """Wire a full flag-ON driver (worker engine + reconciler both present).""" + + cache = MetagraphCache(netuid=1, ttl_seconds=300) + worker_service = WorkerCoordinationService( + factory, + miner_membership=MetagraphMinerMembership(cache), + binding_nonce_store=SqlAlchemyWorkerNonceStore(factory), + heartbeat_ttl_seconds=TTL, + now_fn=lambda: NOW, + ) + worker_assignment_service = WorkerAssignmentService( + factory, worker_service=worker_service, now_fn=lambda: NOW + ) + worker_engine = WorkerAssignmentEngine( + factory, + assignment_service=worker_assignment_service, + worker_service=worker_service, + replication_factor=2, + now_fn=lambda: NOW, + ) + forwarder = _FakeForwarder() + reconciler = WorkerReconciliationService( + factory, result_forwarder=forwarder, now_fn=lambda: NOW + ) + # Flag-ON semantics: gpu is owned by the worker plane (skipped by the + # validator assign/reclaim); cpu (agent-challenge) is untouched by it. + assignment_service = AssignmentService( + factory, + now_fn=lambda: NOW, + default_max_attempts=default_max_attempts, + worker_plane_capabilities=frozenset({CAPABILITY_GPU}), + ) + validator_service = ValidatorCoordinationService(factory, now_fn=lambda: NOW) + fold = _FakeFoldTrigger() + driver = MasterOrchestrationDriver( + assignment_service=assignment_service, + validator_service=validator_service, + work_source=_FakeWorkSource(works=works), + fold_trigger=fold, + worker_assignment_engine=worker_engine, + worker_reconciler=reconciler, + seed=1, + ) + return driver, forwarder, fold + + +async def _add_worker(factory: Any, *, worker_pubkey: str, miner_hotkey: str) -> None: + async with session_scope(factory) as session: + session.add( + WorkerRegistration( + worker_id=f"wid-{worker_pubkey}", + worker_pubkey=worker_pubkey, + miner_hotkey=miner_hotkey, + binding_signature="sig", + binding_nonce=f"nonce-{worker_pubkey}", + provider="local", + provider_instance_ref="local-1", + capabilities=["gpu"], + status=WorkerStatus.ACTIVE, + last_heartbeat_at=NOW, + created_at=NOW, + updated_at=NOW, + ) + ) + + +async def _add_validator(factory: Any, hotkey: str, capabilities: list[str]) -> None: + async with session_scope(factory) as session: + session.add( + Validator( + hotkey=hotkey, + uid=None, + status=ValidatorStatus.ONLINE, + capabilities=list(capabilities), + version="1.0.0", + registered_at=NOW, + last_heartbeat_at=NOW, + ) + ) + + +async def _set_validator_status( + factory: Any, hotkey: str, status: ValidatorStatus +) -> None: + async with session_scope(factory) as session: + validator = ( + await session.execute(select(Validator).where(Validator.hotkey == hotkey)) + ).scalar_one() + validator.status = status + + +async def _units(factory: Any) -> dict[str, WorkAssignment]: + async with factory() as session: + rows = (await session.execute(select(WorkAssignment))).scalars().all() + return {r.work_unit_id: r for r in rows} + + +async def _replica_count(factory: Any, work_unit_id: str) -> int: + async with factory() as session: + return ( + await session.execute( + select(func.count()) + .select_from(WorkerAssignment) + .where( + WorkerAssignment.work_unit_id == work_unit_id, + WorkerAssignment.status.in_(_ACTIVE), + ) + ) + ).scalar_one() + + +async def _worker_fault_count(factory: Any) -> int: + async with factory() as session: + return ( + await session.execute(select(func.count()).select_from(WorkerFault)) + ).scalar_one() + + +def _agent_work( + task_ids: tuple[str, ...] = ("a", "b", "c"), + *, + job_id: str | None = "job-1", +) -> ChallengePendingWork: + # Attestation rides in the result payload; the unit itself is a plain cpu + # unit as far as the master is concerned. + return ChallengePendingWork( + challenge_slug="agent-challenge", + submission_id="sub", + submission_ref="miner-C", + task_ids=task_ids, + job_id=job_id, + payload={"proof": {"tier": "phala-tdx"}}, + ) + + +def _prism_work() -> ChallengePendingWork: + return ChallengePendingWork( + challenge_slug="prism", + submission_id="psub", + submission_ref="miner-P", + ) + + +# --------------------------------------------------------------------------- # +# VAL-VERIFY-020: R=1 preserved for attested agent-challenge units (flag ON) +# --------------------------------------------------------------------------- # +async def test_flag_on_keeps_agent_challenge_at_r1_while_gpu_replicates() -> None: + engine, factory = await _setup() + try: + driver, forwarder, _fold = _build_flag_on_driver( + factory, works=[_agent_work(), _prism_work()] + ) + # Two distinct-owner gpu workers (neither owned by the prism submitter) + # so the gpu primary genuinely replicates to R=2. + await _add_worker(factory, worker_pubkey="wp-a", miner_hotkey="miner-A") + await _add_worker(factory, worker_pubkey="wp-b", miner_hotkey="miner-B") + # One cpu validator to serve the agent-challenge cpu units. + await _add_validator(factory, "v1", ["cpu"]) + + await driver.run_once() + + units = await _units(factory) + cpu_ids = ["sub:a", "sub:b", "sub:c"] + # Each selected task is exactly ONE cpu unit assigned to a single + # validator (R=1): one WorkAssignment row, one hotkey, one attempt. + for uid in cpu_ids: + unit = units[uid] + assert unit.required_capability == "cpu" + assert unit.status == WorkAssignmentStatus.ASSIGNED + assert unit.assigned_validator_hotkey == "v1" + assert unit.attempt_count == 1 + # No worker-plane replica materialized for a cpu unit. + assert await _replica_count(factory, uid) == 0 + + # Sibling gpu unit IS replicated to R=2 -> the worker plane is genuinely + # active this pass, so the cpu R=1 result above is not vacuous. + assert units["psub"].assigned_validator_hotkey is None # worker-owned + assert await _replica_count(factory, "psub") == 2 + + # Repeated passes never add a second replica or a second assignment to a + # cpu unit (still R=1, attempt_count stable, no new units created). + for _ in range(3): + result = await driver.run_once() + assert result.folded == [] + units = await _units(factory) + for uid in cpu_ids: + assert units[uid].status == WorkAssignmentStatus.ASSIGNED + assert units[uid].assigned_validator_hotkey == "v1" + assert units[uid].attempt_count == 1 + assert await _replica_count(factory, uid) == 0 + + # No audit/replica unit id was ever created for a cpu submission. + assert all(not uid.endswith(AUDIT_WORK_UNIT_SUFFIX) for uid in units) + assert set(units) == {"sub:a", "sub:b", "sub:c", "psub"} + assert forwarder.calls == [] + finally: + await engine.dispose() + + +# --------------------------------------------------------------------------- # +# VAL-VERIFY-021: no reconciliation / audit / dispute for agent-challenge units +# --------------------------------------------------------------------------- # +async def test_flag_on_no_reconciliation_or_audit_for_agent_challenge() -> None: + engine, factory = await _setup() + try: + driver, forwarder, _fold = _build_flag_on_driver(factory, works=[_agent_work()]) + await _add_validator(factory, "v1", ["cpu"]) + + result = await driver.run_once() + + # A reconciler ran this pass (flag ON) but produced NO artifacts for the + # agent-challenge units: nothing accepted/disputed/audited/faulted. + assert result.reconciliation is not None + assert result.reconciliation.accepted == [] + assert result.reconciliation.disputed == [] + assert result.reconciliation.audit_units == {} + assert result.reconciliation.faults == [] + + units = await _units(factory) + # No unit is disputed, no validator AUDIT unit exists, and no + # agent-challenge unit carries a validator executor-kind marker. + for unit in units.values(): + assert unit.status != WorkAssignmentStatus.DISPUTED + assert not unit.work_unit_id.endswith(AUDIT_WORK_UNIT_SUFFIX) + assert (unit.payload or {}).get(EXECUTOR_KIND_PAYLOAD_KEY) != ( + EXECUTOR_KIND_VALIDATOR + ) + # No worker faults and no forwards for a cpu submission. + assert await _worker_fault_count(factory) == 0 + assert forwarder.calls == [] + finally: + await engine.dispose() + + +async def test_flag_on_fold_on_exhaustion_finalizes_once_no_dispute() -> None: + engine, factory = await _setup() + try: + driver, _forwarder, fold = _build_flag_on_driver( + factory, + works=[_agent_work(task_ids=("t1",), job_id="job-x")], + default_max_attempts=1, + ) + await _add_validator(factory, "v1", ["cpu"]) + + # Pass 1: bridge + assign (attempt 1 of 1). + first = await driver.run_once() + assert first.folded == [] + assert fold.calls == [] + + # The only cpu validator crashes; the unit exhausts its single attempt. + await _set_validator_status(factory, "v1", ValidatorStatus.OFFLINE) + + # Pass 2: retry-exhausted -> failed -> folded EXACTLY once, via the + # pre-existing fold path (never a dispute/audit). + second = await driver.run_once() + assert second.reassignment.failed == ["sub:t1"] + assert second.folded == ["sub:t1"] + assert fold.calls == [ + ("agent-challenge", "job-x", "t1", WORK_UNIT_MAX_ATTEMPTS_REASON) + ] + assert second.reconciliation is not None + assert second.reconciliation.disputed == [] + assert second.reconciliation.audit_units == {} + + units = await _units(factory) + assert units["sub:t1"].status == WorkAssignmentStatus.FAILED + assert set(units) == {"sub:t1"} # no audit/replica sibling unit created + assert await _replica_count(factory, "sub:t1") == 0 + assert await _worker_fault_count(factory) == 0 + + # Pass 3: an already-folded terminal unit is not re-folded or audited. + third = await driver.run_once() + assert third.folded == [] + assert len(fold.calls) == 1 + assert third.reconciliation is not None + assert third.reconciliation.disputed == [] + finally: + await engine.dispose() diff --git a/tests/unit/test_phala_quote.py b/tests/unit/test_phala_quote.py new file mode 100644 index 000000000..2e51b0917 --- /dev/null +++ b/tests/unit/test_phala_quote.py @@ -0,0 +1,238 @@ +"""Unit tests for the base-side TDX quote primitives (M4 verifier support). + +Pins the structural parser, the dstack RTMR3 event-log replay, the OS-image +identity, and the ``dcap-qvl`` adapter's accept / reject / park mapping used by +the Phala-tier verifier. +""" + +from __future__ import annotations + +import hashlib +import json +import subprocess + +import pytest + +from base.worker.phala_quote import ( + DcapQvlVerifier, + QuoteError, + QuoteStructureError, + QuoteVerificationError, + StaticQuoteVerifier, + VerifierUnavailableError, + build_rtmr3_event_log, + build_tdx_quote, + os_image_hash_from_registers, + parse_td_report, + replay_rtmr3, + runtime_event_digest, +) + +MRTD = "a1" * 48 +RTMR0 = "b0" * 48 +RTMR1 = "b1" * 48 +RTMR2 = "b2" * 48 + + +def _quote(report_data: str = "ab" * 64) -> str: + _log, rtmr3 = build_rtmr3_event_log([("compose-hash", bytes.fromhex("c3" * 32))]) + return build_tdx_quote( + mrtd=MRTD, + rtmr0=RTMR0, + rtmr1=RTMR1, + rtmr2=RTMR2, + rtmr3=rtmr3, + report_data=report_data, + ) + + +def test_parse_td_report_round_trips_registers() -> None: + report = parse_td_report(_quote(report_data="ff" * 32)) + assert report.mrtd == MRTD + assert report.rtmr0 == RTMR0 + assert report.rtmr1 == RTMR1 + assert report.rtmr2 == RTMR2 + assert report.report_data == bytes.fromhex("ff" * 32).ljust(64, b"\x00") + + +def test_parse_td_report_accepts_0x_prefix() -> None: + report = parse_td_report("0x" + _quote()) + assert report.mrtd == MRTD + + +@pytest.mark.parametrize("bad", ["", "zz", "ab" * 4]) +def test_parse_td_report_rejects_malformed_or_short(bad: str) -> None: + with pytest.raises(QuoteStructureError): + parse_td_report(bad) + + +def test_os_image_hash_matches_sha256_of_registers() -> None: + expected = hashlib.sha256( + bytes.fromhex(MRTD) + bytes.fromhex(RTMR1) + bytes.fromhex(RTMR2) + ).hexdigest() + assert os_image_hash_from_registers(MRTD, RTMR1, RTMR2) == expected + + +def test_replay_rtmr3_surfaces_compose_and_key_provider() -> None: + compose = bytes.fromhex("c3" * 32) + provider = b"kms-root" + log, rtmr3 = build_rtmr3_event_log( + [("compose-hash", compose), ("key-provider", provider)] + ) + replay = replay_rtmr3(log) + assert replay.rtmr3 == rtmr3 + assert replay.compose_hash == compose.hex() + assert replay.key_provider == provider.hex() + + +def test_replay_rtmr3_ignores_non_app_imr_entries() -> None: + log, rtmr3 = build_rtmr3_event_log([("compose-hash", bytes.fromhex("c3" * 32))]) + noise = [{"imr": 0, "event": "boot", "digest": "aa" * 48}, *log] + assert replay_rtmr3(noise).rtmr3 == rtmr3 + + +def test_replay_rtmr3_rejects_inconsistent_digest() -> None: + log, _rtmr3 = build_rtmr3_event_log([("compose-hash", bytes.fromhex("c3" * 32))]) + log[0]["event_payload"] = "ee" * 32 # digest no longer matches payload + with pytest.raises(QuoteVerificationError): + replay_rtmr3(log) + + +def test_replay_rtmr3_rejects_non_mapping_entry() -> None: + with pytest.raises(QuoteVerificationError): + replay_rtmr3([["not", "a", "mapping"]]) # type: ignore[list-item] + + +def test_replay_rtmr3_rejects_bad_payload_hex() -> None: + from base.worker.phala_quote import APP_IMR, DSTACK_RUNTIME_EVENT_TYPE + + bad = [ + { + "imr": APP_IMR, + "event_type": DSTACK_RUNTIME_EVENT_TYPE, + "event": "compose-hash", + "event_payload": "zz", + } + ] + with pytest.raises(QuoteVerificationError): + replay_rtmr3(bad) + + +def test_replay_rtmr3_accepts_non_runtime_event_with_raw_digest() -> None: + digest = runtime_event_digest("compose-hash", bytes.fromhex("c3" * 32)) + entry = { + "imr": 3, + "event_type": 1, + "event": "some-tcg-event", + "digest": digest.hex(), + } + replay = replay_rtmr3([entry]) + assert len(bytes.fromhex(replay.rtmr3)) == 48 + + +def test_build_tdx_quote_honors_header_and_tail() -> None: + quote = build_tdx_quote( + mrtd=MRTD, + rtmr0=RTMR0, + rtmr1=RTMR1, + rtmr2=RTMR2, + rtmr3="d3" * 48, + report_data="ab" * 32, + header=b"\x01" * 48, + tail=b"\x09\x09", + ) + raw = bytes.fromhex(quote) + assert raw[:48] == b"\x01" * 48 + assert raw.endswith(b"\x09\x09") + + +def test_static_verifier_default_is_uptodate() -> None: + assert StaticQuoteVerifier().verify("00" * 8).tcb_status == "UpToDate" + + +def test_dcap_qvl_parses_advisories_and_alt_status_keys() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + body = json.dumps({"tcbStatus": "UpToDate", "advisoryIDs": ["INTEL-SA-1"]}) + return subprocess.CompletedProcess(args, returncode=0, stdout=body, stderr="") + + verdict = DcapQvlVerifier(runner=runner).verify("00" * 8) + assert verdict.tcb_status == "UpToDate" + assert verdict.advisory_ids == ("INTEL-SA-1",) + + +# --- exit-0-but-unparseable / missing-TCB-status output PARKS (retryable) ---- +# A dcap-qvl exit code of 0 means the tool accepted the quote's cryptography; if +# its stdout is then unparseable / missing a TCB status that is a *tooling* +# regression (output-format change), NOT a fraud verdict -- park (retry), never +# permanently fraud-reject a legitimate result. Genuine invalid-quote/bad-TCB +# verdicts (nonzero exit) still reject; the distinction is asserted below. + + +def test_dcap_qvl_exit0_unparseable_json_is_park() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, returncode=0, stdout="not json", stderr="" + ) + + with pytest.raises(VerifierUnavailableError): + DcapQvlVerifier(runner=runner).verify("00" * 8) + + +def test_dcap_qvl_exit0_non_object_json_is_park() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, returncode=0, stdout="[1,2,3]", stderr="" + ) + + with pytest.raises(VerifierUnavailableError): + DcapQvlVerifier(runner=runner).verify("00" * 8) + + +def test_dcap_qvl_exit0_missing_status_is_park() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess(args, returncode=0, stdout="{}", stderr="") + + with pytest.raises(VerifierUnavailableError): + DcapQvlVerifier(runner=runner).verify("00" * 8) + + +def test_dcap_qvl_exit0_empty_stdout_is_park() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess(args, returncode=0, stdout="", stderr="") + + with pytest.raises(VerifierUnavailableError): + DcapQvlVerifier(runner=runner).verify("00" * 8) + + +# --- discriminator: a genuine rejection (nonzero exit) must NOT park ---------- +# Proves the park mapping above is not "everything parks": a real invalid-quote / +# bad-TCB verdict from dcap-qvl (nonzero exit) is still a PERMANENT reject. + + +def test_dcap_qvl_nonzero_exit_is_reject_not_park() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, returncode=1, stdout="", stderr="quote rejected: bad signature" + ) + + with pytest.raises(QuoteVerificationError): + DcapQvlVerifier(runner=runner).verify("00" * 8) + + +def test_dcap_qvl_exit0_unparseable_is_not_accepted() -> None: + # Never accept-any: an exit-0-but-unparseable result yields no QuoteVerdict. + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, returncode=0, stdout="not json", stderr="" + ) + + with pytest.raises(QuoteError): + DcapQvlVerifier(runner=runner).verify("00" * 8) + + +def test_dcap_qvl_generic_subprocess_error_is_park() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + raise subprocess.SubprocessError("boom") + + with pytest.raises(VerifierUnavailableError): + DcapQvlVerifier(runner=runner).verify("00" * 8) diff --git a/tests/unit/test_phala_quote_golden_vector.py b/tests/unit/test_phala_quote_golden_vector.py new file mode 100644 index 000000000..807ca5b8d --- /dev/null +++ b/tests/unit/test_phala_quote_golden_vector.py @@ -0,0 +1,108 @@ +"""Shared cross-repo golden TDX quote/measurement/RTMR3 anti-drift vector (base). + +base ``src/base/worker/phala_quote.py`` re-implements agent-challenge +``src/agent_challenge/keyrelease/quote.py`` almost verbatim (TDX register +byte-offsets, ``os_image_hash = sha256(MRTD||RTMR1||RTMR2)``, dstack RTMR3 +event-log replay) because base cannot import the lean in-CVM module. That +duplication is drift-prone: a one-sided offset/hash tweak could let a real dstack +quote verify in one repo and silently fail in the other. + +This test pins a FIXED quote + event log (``tests/unit/phala_quote_golden_vector.json``) +to the exact registers / os_image_hash / RTMR3 / compose-hash a correct parser +must reproduce, and asserts base's parser reproduces them. The SAME fixture bytes +and the SAME :data:`GOLDEN_VECTOR_SHA256` are asserted in agent-challenge +(``tests/test_quote_golden_vector.py``); because the pinned expected values come +from the frozen fixture (not recomputed with the same offsets under test), a +one-sided offset/hash change diverges from these values and fails here or there. +Do NOT edit one repo's copy without the other (AGENTS.md +'TDX-quote-parse / measurement / RTMR3 anti-drift'). +""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path + +from base.worker.phala_quote import ( + _MRTD_OFFSET, + REGISTER_LEN, + build_rtmr3_event_log, + os_image_hash_from_registers, + parse_td_report, + replay_rtmr3, +) + +# Same literal in BOTH repos: the SHA-256 of the byte-identical golden fixture. +# If either repo's fixture is edited, that side's pin fails; a reviewer can grep +# this constant across both repos to confirm they still match. +GOLDEN_VECTOR_SHA256 = ( + "053979e8445c147798ec6f9165f9849a38ff5797e4dccd2ee38aa329b7f673bf" +) + +_VECTOR_PATH = Path(__file__).parent / "phala_quote_golden_vector.json" + + +def _vector() -> dict: + return json.loads(_VECTOR_PATH.read_text(encoding="utf-8")) + + +def test_golden_fixture_is_byte_identical_across_repos() -> None: + digest = hashlib.sha256(_VECTOR_PATH.read_bytes()).hexdigest() + assert digest == GOLDEN_VECTOR_SHA256 + + +def test_golden_quote_parses_to_expected_registers() -> None: + vector = _vector() + expected = vector["expected"] + report = parse_td_report(vector["quote_hex"]) + assert report.mrtd == expected["mrtd"] + assert report.rtmr0 == expected["rtmr0"] + assert report.rtmr1 == expected["rtmr1"] + assert report.rtmr2 == expected["rtmr2"] + assert report.rtmr3 == expected["rtmr3"] + assert report.report_data.hex() == expected["report_data_hex"] + + +def test_golden_os_image_hash_matches() -> None: + vector = _vector() + expected = vector["expected"] + assert ( + os_image_hash_from_registers( + expected["mrtd"], expected["rtmr1"], expected["rtmr2"] + ) + == expected["os_image_hash"] + ) + + +def test_golden_event_log_replays_to_expected_rtmr3() -> None: + vector = _vector() + expected = vector["expected"] + replay = replay_rtmr3(vector["event_log"]) + assert replay.rtmr3 == expected["rtmr3"] + assert replay.compose_hash == expected["compose_hash"] + assert replay.key_provider == expected["key_provider"] + # The event-log replay reproduces the RTMR3 the fixed quote carries. + assert replay.rtmr3 == parse_td_report(vector["quote_hex"]).rtmr3 + + +def test_golden_vector_offset_sensitivity_discriminator() -> None: + # Non-vacuity: the pinned MRTD is not read from a constant. Reading one byte + # off the register offset (a simulated one-sided off-by-one) yields a value + # that differs from the golden -- exactly the divergence the pin above catches. + vector = _vector() + raw = bytes.fromhex(vector["quote_hex"]) + correct = raw[_MRTD_OFFSET : _MRTD_OFFSET + REGISTER_LEN].hex() + tweaked = raw[_MRTD_OFFSET + 1 : _MRTD_OFFSET + 1 + REGISTER_LEN].hex() + assert correct == vector["expected"]["mrtd"] + assert tweaked != vector["expected"]["mrtd"] + + +def test_golden_vector_replay_sensitivity_discriminator() -> None: + # Non-vacuity: a different compose payload replays to a different RTMR3, so the + # pinned RTMR3 genuinely binds the event-log digest/extend formula. + vector = _vector() + _log, other_rtmr3 = build_rtmr3_event_log( + [("compose-hash", bytes.fromhex("00" * 32))] + ) + assert other_rtmr3 != vector["expected"]["rtmr3"] diff --git a/tests/unit/test_phala_verify.py b/tests/unit/test_phala_verify.py new file mode 100644 index 000000000..825d9989c --- /dev/null +++ b/tests/unit/test_phala_verify.py @@ -0,0 +1,169 @@ +"""Unit tests for the validator-owned allowlist + nonce primitives (M4). + +Pins measurement allowlist membership + fail-closed empties, and the single-use / +TTL nonce lifecycle the Phala-tier verifier relies on. +""" + +from __future__ import annotations + +import json + +from base.schemas.worker import PhalaMeasurement +from base.worker.phala_verify import ( + MEASUREMENT_ALLOWLIST_FILE_ENV, + InMemoryNonceValidator, + MeasurementAllowlist, + NonceState, + canonical_measurement_mapping, +) + +MEASUREMENT = { + "mrtd": "a" * 96, + "rtmr0": "b0" * 48, + "rtmr1": "b1" * 48, + "rtmr2": "b2" * 48, + "compose_hash": "c" * 64, + "os_image_hash": "e" * 64, +} + +#: A second, distinct canonical measurement (image rotation: old + new). +OTHER_MEASUREMENT = { + "mrtd": "f" * 96, + "rtmr0": "a0" * 48, + "rtmr1": "a1" * 48, + "rtmr2": "a2" * 48, + "compose_hash": "d" * 64, + "os_image_hash": "9" * 64, +} + + +def _phala_measurement() -> PhalaMeasurement: + return PhalaMeasurement(rtmr3="d" * 96, **MEASUREMENT) + + +def test_canonical_mapping_from_model_excludes_rtmr3() -> None: + mapping = canonical_measurement_mapping(_phala_measurement()) + assert mapping == MEASUREMENT + assert "rtmr3" not in mapping + + +def test_allowlist_contains_exact_match() -> None: + allowlist = MeasurementAllowlist.from_measurements([MEASUREMENT]) + assert allowlist.contains(MEASUREMENT) is True + assert allowlist.contains(_phala_measurement()) is True + + +def test_allowlist_rejects_single_register_mismatch() -> None: + allowlist = MeasurementAllowlist.from_measurements([MEASUREMENT]) + off = {**MEASUREMENT, "compose_hash": "0" * 64} + assert allowlist.contains(off) is False + + +def test_empty_allowlist_fails_closed() -> None: + empty = MeasurementAllowlist() + assert bool(empty) is False + assert empty.contains(MEASUREMENT) is False + + +# --- VAL-VERIFY-027: multiple canonical entries (image rotation) ------------- + + +def test_allowlist_holds_multiple_entries_matches_any() -> None: + rotation = MeasurementAllowlist.from_measurements([MEASUREMENT, OTHER_MEASUREMENT]) + assert len(rotation.entries) == 2 + # Either allowlisted entry (outgoing + incoming image) matches. + assert rotation.contains(MEASUREMENT) is True + assert rotation.contains(OTHER_MEASUREMENT) is True + # A measurement listed under neither entry is rejected. + third = {**MEASUREMENT, "compose_hash": "1" * 64} + assert rotation.contains(third) is False + + +# --- VAL-VERIFY-025: fail-closed loading (empty / missing / unparseable) ----- + + +def test_from_json_parses_entries_object_and_bare_list() -> None: + obj = MeasurementAllowlist.from_json(json.dumps({"entries": [MEASUREMENT]})) + assert obj.contains(MEASUREMENT) is True + bare = MeasurementAllowlist.from_json(json.dumps([MEASUREMENT, OTHER_MEASUREMENT])) + assert bare.contains(MEASUREMENT) is True + assert bare.contains(OTHER_MEASUREMENT) is True + + +def test_from_json_unparseable_fails_closed() -> None: + broken = MeasurementAllowlist.from_json("{ this is : not json ]") + assert bool(broken) is False + assert broken.contains(MEASUREMENT) is False + + +def test_from_json_wrong_json_shape_fails_closed() -> None: + assert bool(MeasurementAllowlist.from_json(json.dumps(42))) is False + assert bool(MeasurementAllowlist.from_json(json.dumps("nope"))) is False + assert bool(MeasurementAllowlist.from_json(json.dumps({"other": []}))) is False + + +def test_from_json_malformed_entry_fails_closed() -> None: + missing_register = {k: v for k, v in MEASUREMENT.items() if k != "mrtd"} + assert bool(MeasurementAllowlist.from_json(json.dumps([missing_register]))) is False + assert bool(MeasurementAllowlist.from_json(json.dumps(["not-a-mapping"]))) is False + + +def test_from_file_missing_fails_closed(tmp_path) -> None: + absent = MeasurementAllowlist.from_file(tmp_path / "does-not-exist.json") + assert bool(absent) is False + + +def test_from_file_reads_entries(tmp_path) -> None: + path = tmp_path / "allowlist.json" + path.write_text(json.dumps({"entries": [MEASUREMENT]}), encoding="utf-8") + loaded = MeasurementAllowlist.from_file(path) + assert loaded.contains(MEASUREMENT) is True + + +def test_from_env_unconfigured_fails_closed() -> None: + assert bool(MeasurementAllowlist.from_env(env={})) is False + assert ( + bool(MeasurementAllowlist.from_env(env={MEASUREMENT_ALLOWLIST_FILE_ENV: ""})) + is False + ) + + +def test_from_env_reads_configured_file(tmp_path) -> None: + path = tmp_path / "allowlist.json" + path.write_text(json.dumps([MEASUREMENT, OTHER_MEASUREMENT]), encoding="utf-8") + loaded = MeasurementAllowlist.from_env( + env={MEASUREMENT_ALLOWLIST_FILE_ENV: str(path)} + ) + assert loaded.contains(MEASUREMENT) is True + assert loaded.contains(OTHER_MEASUREMENT) is True + + +def test_nonce_single_use_lifecycle() -> None: + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + assert nonces.is_outstanding(nonce) is True + assert nonces.consume(nonce) is NonceState.OK + assert nonces.is_outstanding(nonce) is False + assert nonces.consume(nonce) is NonceState.CONSUMED + + +def test_nonce_unknown_and_empty() -> None: + nonces = InMemoryNonceValidator() + assert nonces.consume("never-issued") is NonceState.UNKNOWN + assert nonces.consume("") is NonceState.UNKNOWN + assert nonces.is_outstanding("") is False + + +def test_nonce_expiry() -> None: + clock = {"t": 0.0} + nonces = InMemoryNonceValidator(ttl_seconds=10, clock=lambda: clock["t"]) + nonce = nonces.issue() + clock["t"] = 100.0 + assert nonces.is_outstanding(nonce) is False + assert nonces.consume(nonce) is NonceState.EXPIRED + + +def test_issue_accepts_explicit_value() -> None: + nonces = InMemoryNonceValidator() + assert nonces.issue("fixed-nonce") == "fixed-nonce" + assert nonces.consume("fixed-nonce") is NonceState.OK diff --git a/tests/unit/test_replay_audit_integration.py b/tests/unit/test_replay_audit_integration.py new file mode 100644 index 000000000..4c6ef0649 --- /dev/null +++ b/tests/unit/test_replay_audit_integration.py @@ -0,0 +1,283 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from typing import Any + +import httpx +import pytest + +from base.master.replay_audit import ( + REPLAY_AUDIT_LABEL, + REPLAY_AUDIT_REQUEST_KIND, + REPLAY_AUDIT_RESULT_KIND, + ReplayAuditRequest, + ReplayAuditResult, + plan_sha256, + replay_assignment_payload, + scoring_policy_digest, +) + + +def _plan() -> dict[str, Any]: + policy = { + "schema_version": 1, + "per_task_aggregation": "best_of_k", + "keep_policy": "threshold_band", + "drop_lowest_n": 0, + "threshold_f64be": "3fe0000000000000", + } + return { + "schema_version": 1, + "eval_run_id": "run-1", + "submission_id": "sub-1", + "submission_version": 2, + "authorizing_review_digest": "01" * 32, + "agent_hash": "02" * 32, + "selected_tasks": [ + { + "task_id": "task-a", + "image_ref": "registry.example/task@sha256:" + "03" * 32, + "task_config_sha256": "04" * 32, + }, + { + "task_id": "task-b", + "image_ref": "registry.example/task@sha256:" + "05" * 32, + "task_config_sha256": "06" * 32, + }, + ], + "k": 3, + "scoring_policy": policy, + "scoring_policy_digest": scoring_policy_digest(policy), + "eval_app": { + "image_ref": "registry.example/eval@sha256:" + "07" * 32, + "compose_hash": "08" * 32, + "app_identity": "eval-app-v1", + "kms_key_algorithm": "x25519", + "kms_public_key_hex": "09" * 32, + "kms_public_key_sha256": __import__("hashlib") + .sha256(bytes.fromhex("09" * 32)) + .hexdigest(), + "measurement": { + "mrtd": "0a" * 48, + "rtmr0": "0b" * 48, + "rtmr1": "0c" * 48, + "rtmr2": "0d" * 48, + "os_image_hash": "0e" * 32, + "key_provider": "validator-kms", + "vm_shape": "tdx.small", + }, + }, + "key_release_endpoint": "tcp://release.example:8701", + "result_endpoint": "/evaluation/v1/runs/run-1/result", + "key_release_nonce": "key-nonce-1", + "score_nonce": "score-nonce-1", + "run_token_sha256": "0f" * 32, + "issued_at_ms": 1_000, + "expires_at_ms": 2_000, + } + + +def _request() -> dict[str, Any]: + plan = _plan() + return { + "schema_version": 1, + "audit_label": REPLAY_AUDIT_LABEL, + "kind": REPLAY_AUDIT_REQUEST_KIND, + "audit_id": "replay:run-1:1", + "submission_id": "sub-1", + "eval_run_id": "run-1", + "replay_attempt": 1, + "plan_sha256": plan_sha256(plan), + "eval_plan": plan, + "k": 3, + "selected_tasks": plan["selected_tasks"], + "scoring_policy": plan["scoring_policy"], + "scoring_policy_digest": plan["scoring_policy_digest"], + "attested_score": 0.75, + } + + +def _result() -> dict[str, Any]: + return { + "schema_version": 1, + "audit_label": REPLAY_AUDIT_LABEL, + "kind": REPLAY_AUDIT_RESULT_KIND, + "audit_id": "replay:run-1:1", + "submission_id": "sub-1", + "eval_run_id": "run-1", + "replay_attempt": 1, + "plan_sha256": plan_sha256(_plan()), + "trial_scores_by_task": { + "task-a": [0.25, 0.5, 0.75], + "task-b": [0.5, 0.75, 1.0], + }, + } + + +def test_replay_wire_preserves_complete_plan_and_policy() -> None: + request = ReplayAuditRequest.from_mapping(_request()) + payload = replay_assignment_payload(request) + + assert payload["replay_audit_label"] == REPLAY_AUDIT_LABEL + assert payload["replay_audit_request"] == _request() + assert ( + payload["replay_audit_request"]["eval_plan"]["scoring_policy"] + == _plan()["scoring_policy"] + ) + + +def test_replay_wire_rejects_label_or_plan_mutations() -> None: + raw = _request() + raw["kind"] = "ordinary_assignment" + with pytest.raises(ValueError): + ReplayAuditRequest.from_mapping(raw) + + raw = _request() + raw["k"] = 1 + with pytest.raises(ValueError): + ReplayAuditRequest.from_mapping(raw) + + +def test_replay_result_requires_ordered_trials_not_an_aggregate() -> None: + result = ReplayAuditResult.from_mapping(_result()) + assert list(result.trial_scores_by_task["task-b"]) == [0.5, 0.75, 1.0] + + raw = _result() + raw.pop("trial_scores_by_task") + raw["score"] = 0.75 + with pytest.raises(ValueError): + ReplayAuditResult.from_mapping(raw) + + +@dataclass +class _Record: + slug: str + internal_base_url: str + + +class _Registry: + def __init__(self) -> None: + self.record = _Record("agent-challenge", "http://challenge:8000") + + async def list(self, *, active_only: bool = False) -> list[_Record]: + return [self.record] + + async def get(self, slug: str) -> _Record: + return self.record + + async def get_token(self, slug: str) -> str: + return "challenge-token" + + +async def test_replay_client_consumes_only_the_labelled_request() -> None: + from base.master.challenge_work_source import HttpChallengeReplayClient + + def handler(request: httpx.Request) -> httpx.Response: + assert request.headers["authorization"] == "Bearer challenge-token" + return httpx.Response(200, json=_request()) + + client = HttpChallengeReplayClient( + _Registry(), + transport=httpx.MockTransport(handler), + ) + request = await client.fetch_request( + challenge_slug="agent-challenge", eval_run_id="run-1" + ) + assert request.eval_run_id == "run-1" + + def unlabeled(_: httpx.Request) -> httpx.Response: + body = _request() + body["audit_label"] = "not-replay" + return httpx.Response(200, json=body) + + client = HttpChallengeReplayClient( + _Registry(), + transport=httpx.MockTransport(unlabeled), + ) + with pytest.raises(ValueError): + await client.fetch_request( + challenge_slug="agent-challenge", eval_run_id="run-1" + ) + + +async def test_replay_adapter_uses_replay_entrypoint_only() -> None: + from base.schemas.assignment import AssignmentView + from base.validator.agent import AssignmentContext, BrokerConfig + from base.validator.agent.adapters import AgentChallengeCycleExecutor + + calls: list[dict[str, Any]] = [] + + async def dispatch_replay(**kwargs: Any) -> dict[str, Any]: + calls.append(kwargs) + return _result() + + assignment = AssignmentView( + assignment_id="11111111-1111-1111-1111-111111111111", + challenge_slug="agent-challenge", + work_unit_id="replay:run-1:1", + submission_ref="sub-1", + payload=replay_assignment_payload(ReplayAuditRequest.from_mapping(_request())), + required_capability="cpu", + status="running", + attempt=1, + max_attempts=3, + ) + context = AssignmentContext( + assignment=assignment, + broker=BrokerConfig(broker_url="http://validator-broker:8082"), + ) + executor = AgentChallengeCycleExecutor(dispatch_replay=dispatch_replay) + result = await executor.execute(context, progress=lambda **_: None) # type: ignore[arg-type] + + assert result.success is True + assert result.payload["replay_audit_result"]["kind"] == REPLAY_AUDIT_RESULT_KIND + assert calls[0]["request"] == _request() + assert calls[0]["broker_url"] == "http://validator-broker:8082" + + +async def test_normal_adapter_does_not_invoke_replay_entrypoint() -> None: + from base.schemas.assignment import AssignmentView + from base.validator.agent import AssignmentContext, BrokerConfig + from base.validator.agent.adapters import AgentChallengeCycleExecutor + + normal_calls: list[dict[str, Any]] = [] + replay_calls: list[dict[str, Any]] = [] + + async def dispatch(**kwargs: Any) -> dict[str, Any]: + normal_calls.append(kwargs) + return _result() + + async def dispatch_replay(**kwargs: Any) -> dict[str, Any]: + replay_calls.append(kwargs) + return _result() + + assignment = AssignmentView( + assignment_id="22222222-2222-2222-2222-222222222222", + challenge_slug="agent-challenge", + work_unit_id="ordinary:run-1", + submission_ref="sub-1", + payload={"task_id": "task-1"}, + required_capability="cpu", + status="running", + attempt=1, + max_attempts=3, + ) + context = AssignmentContext( + assignment=assignment, + broker=BrokerConfig(broker_url="http://validator-broker:8082"), + ) + executor = AgentChallengeCycleExecutor( + dispatch=dispatch, + dispatch_replay=dispatch_replay, + ) + await executor.execute(context, progress=lambda **_: None) # type: ignore[arg-type] + + assert len(normal_calls) == 1 + assert replay_calls == [] + + +def test_result_wire_serializes_without_losing_trial_order() -> None: + result = ReplayAuditResult.from_mapping(_result()) + encoded = json.dumps(result.to_dict(), separators=(",", ":")) + assert encoded.index("task-a") < encoded.index("task-b") diff --git a/tests/unit/test_replay_audit_schema_hardening.py b/tests/unit/test_replay_audit_schema_hardening.py new file mode 100644 index 000000000..99a21fc9d --- /dev/null +++ b/tests/unit/test_replay_audit_schema_hardening.py @@ -0,0 +1,167 @@ +from __future__ import annotations + +import hashlib +import json +from typing import Any + +import pytest + +from base.master.replay_audit import ( + REPLAY_AUDIT_LABEL, + REPLAY_AUDIT_REQUEST_KIND, + ReplayAuditRequest, + ReplayAuditWireError, + plan_sha256, + scoring_policy_digest, +) + + +def _policy() -> dict[str, Any]: + return { + "schema_version": 1, + "per_task_aggregation": "mean", + "keep_policy": "off", + "drop_lowest_n": 0, + "threshold_f64be": None, + } + + +def _plan() -> dict[str, Any]: + policy = _policy() + public_key = "ab" * 32 + return { + "schema_version": 1, + "eval_run_id": "run-1", + "submission_id": "sub-1", + "submission_version": 2, + "authorizing_review_digest": "01" * 32, + "agent_hash": "02" * 32, + "selected_tasks": [ + { + "task_id": "task-a", + "image_ref": "registry.example/task@sha256:" + "03" * 32, + "task_config_sha256": "04" * 32, + }, + { + "task_id": "task-b", + "image_ref": "registry.example/task@sha256:" + "05" * 32, + "task_config_sha256": "06" * 32, + }, + ], + "k": 2, + "scoring_policy": policy, + "scoring_policy_digest": scoring_policy_digest(policy), + "eval_app": { + "image_ref": "registry.example/eval@sha256:" + "07" * 32, + "compose_hash": "08" * 32, + "app_identity": "eval-app-v1", + "kms_key_algorithm": "x25519", + "kms_public_key_hex": public_key, + "kms_public_key_sha256": hashlib.sha256( + bytes.fromhex(public_key) + ).hexdigest(), + "measurement": { + "mrtd": "09" * 48, + "rtmr0": "0a" * 48, + "rtmr1": "0b" * 48, + "rtmr2": "0c" * 48, + "os_image_hash": "0d" * 32, + "key_provider": "validator-kms", + "vm_shape": "tdx.small", + }, + }, + "key_release_endpoint": "tcp://release.example:8701", + "result_endpoint": "/evaluation/v1/runs/run-1/result", + "key_release_nonce": "key-nonce-1", + "score_nonce": "score-nonce-1", + "run_token_sha256": "0e" * 32, + "issued_at_ms": 1_000, + "expires_at_ms": 2_000, + } + + +def _request() -> dict[str, Any]: + plan = _plan() + return { + "schema_version": 1, + "audit_label": REPLAY_AUDIT_LABEL, + "kind": REPLAY_AUDIT_REQUEST_KIND, + "audit_id": "replay:run-1:1", + "submission_id": "sub-1", + "eval_run_id": "run-1", + "replay_attempt": 1, + "plan_sha256": plan_sha256(plan), + "eval_plan": plan, + "k": plan["k"], + "selected_tasks": plan["selected_tasks"], + "scoring_policy": plan["scoring_policy"], + "scoring_policy_digest": plan["scoring_policy_digest"], + "attested_score": 0.75, + } + + +def test_replay_request_recomputes_complete_plan_and_policy_digests() -> None: + request = ReplayAuditRequest.from_mapping(_request()) + + assert request.plan_sha256 == plan_sha256(request.eval_plan) + assert request.scoring_policy_digest == scoring_policy_digest( + request.scoring_policy + ) + assert [task["task_id"] for task in request.selected_tasks] == ["task-a", "task-b"] + + +@pytest.mark.parametrize( + "mutate", + [ + lambda body: body["eval_plan"].update({"k": 3}), + lambda body: body["eval_plan"]["selected_tasks"].reverse(), + lambda body: body["eval_plan"]["scoring_policy"].update( + {"keep_policy": "drop_lowest_n"} + ), + lambda body: body["eval_plan"].pop("eval_app"), + lambda body: body["eval_plan"]["scoring_policy"].update({"drop_lowest_n": 1}), + ], +) +def test_replay_request_rejects_single_field_plan_mutations(mutate) -> None: + body = _request() + mutate(body) + with pytest.raises(ReplayAuditWireError): + ReplayAuditRequest.from_mapping(body) + + +def test_replay_request_rejects_forged_digests_and_aliases() -> None: + body = _request() + body["plan_sha256"] = "ff" * 32 + with pytest.raises(ReplayAuditWireError): + ReplayAuditRequest.from_mapping(body) + + body = _request() + body["scoring_policy_digest"] = "ff" * 32 + with pytest.raises(ReplayAuditWireError): + ReplayAuditRequest.from_mapping(body) + + body = _request() + body["eval_plan"]["scoring_policy"]["threshold"] = None + with pytest.raises(ReplayAuditWireError): + ReplayAuditRequest.from_mapping(body) + + +def test_replay_request_rejects_duplicate_nested_json_keys() -> None: + body = _request() + raw = json.dumps(body, separators=(",", ":")) + raw = raw.replace('"k":2', '"k":2,"k":2', 1) + with pytest.raises(ReplayAuditWireError): + ReplayAuditRequest.from_mapping(json.loads(raw), raw_body=raw.encode()) + + +def test_replay_request_rejects_partial_plan_before_assignment_payload() -> None: + body = _request() + body["eval_plan"] = { + "schema_version": 1, + "eval_run_id": "run-1", + "selected_tasks": body["selected_tasks"], + "k": 2, + "scoring_policy": body["scoring_policy"], + } + with pytest.raises(ReplayAuditWireError): + ReplayAuditRequest.from_mapping(body) diff --git a/tests/unit/test_supervisor_weights.py b/tests/unit/test_supervisor_weights.py index c4fede9c9..caf43d1b4 100644 --- a/tests/unit/test_supervisor_weights.py +++ b/tests/unit/test_supervisor_weights.py @@ -12,6 +12,7 @@ import logging import threading import time +from collections.abc import Iterator from types import SimpleNamespace from typing import Any @@ -138,8 +139,54 @@ def test_compute_uses_cli_path_with_zero_chain_calls( assert recorder["set_weights_calls"] == [] +@pytest.fixture +def scheduler_error_messages() -> Iterator[list[str]]: + """Capture ERROR records from ``base.supervisor.scheduler`` order-independently. + + The full suite runs, in alphabetical order, files that import bittensor + (which forces every already-created logger's level to CRITICAL) and files + that run alembic's ``fileConfig`` (which can set ``logging.disable`` and a + logger's ``.disabled`` flag). Any of those left set would swallow the + scheduler's error log and fail this test only in the full run. Snapshot and + neutralize every one of those process-wide knobs for the duration of the + test, capture via a handler on the scheduler's own logger (propagation off), + then restore the prior state so this test perturbs no sibling. + """ + logger = logging.getLogger("base.supervisor.scheduler") + + class _RecordingHandler(logging.Handler): + def __init__(self) -> None: + super().__init__(level=logging.ERROR) + self.messages: list[str] = [] + self._lock = threading.Lock() + + def emit(self, record: logging.LogRecord) -> None: + with self._lock: + self.messages.append(record.getMessage()) + + handler = _RecordingHandler() + prev_disable = logging.root.manager.disable + prev_disabled = logger.disabled + prev_level = logger.level + prev_propagate = logger.propagate + logging.disable(logging.NOTSET) + logger.disabled = False + logger.setLevel(logging.DEBUG) + logger.propagate = False + logger.addHandler(handler) + try: + yield handler.messages + finally: + logger.removeHandler(handler) + logger.propagate = prev_propagate + logger.setLevel(prev_level) + logger.disabled = prev_disabled + logging.disable(prev_disable) + + def test_compute_raise_is_logged_and_schedule_continues( monkeypatch: pytest.MonkeyPatch, + scheduler_error_messages: list[str], ) -> None: counts = {"n": 0} lock = threading.Lock() @@ -155,27 +202,8 @@ def boom(settings: Settings) -> FinalWeights: shutdown = threading.Event() worker = TaskWorker(task=fast, shutdown=shutdown) - class _RecordingHandler(logging.Handler): - def __init__(self) -> None: - super().__init__(level=logging.ERROR) - self.messages: list[str] = [] - self._lock = threading.Lock() - - def emit(self, record: logging.LogRecord) -> None: - with self._lock: - self.messages.append(record.getMessage()) - - # Attach directly to the scheduler logger: immune to other tests - # reconfiguring root logging (which breaks caplog in the full run). - # Re-enable it too: alembic's env.py fileConfig (run by any migration - # test) disables all previously-imported loggers suite-wide. - scheduler_logger = logging.getLogger("base.supervisor.scheduler") - handler = _RecordingHandler() - was_disabled = scheduler_logger.disabled - scheduler_logger.disabled = False - scheduler_logger.addHandler(handler) + worker.start() try: - worker.start() deadline = time.monotonic() + 5.0 while time.monotonic() < deadline: with lock: @@ -185,7 +213,9 @@ def emit(self, record: logging.LogRecord) -> None: shutdown.set() assert worker.join(timeout=5.0) finally: - scheduler_logger.removeHandler(handler) - scheduler_logger.disabled = was_disabled + shutdown.set() + worker.join(timeout=5.0) assert counts["n"] >= 3 # schedule kept ticking after each raise - assert any("raised; continuing schedule" in message for message in handler.messages) + assert any( + "raised; continuing schedule" in message for message in scheduler_error_messages + ) diff --git a/tests/unit/test_worker_proof_phala.py b/tests/unit/test_worker_proof_phala.py new file mode 100644 index 000000000..0a408df42 --- /dev/null +++ b/tests/unit/test_worker_proof_phala.py @@ -0,0 +1,408 @@ +"""Phala TDX tier for ExecutionProof: schema, report_data binding, build helper. + +Foundational schema shared by the image emitter (M1) and the verifier (M4) +(architecture.md sec 6/9). These tests pin the Phala tier value, the attestation +payload schema, the architecture-sec-6 ``report_data`` derivation, and the +``build_phala_execution_proof`` helper -- while asserting existing tier-0/1/2 +behavior is unchanged. +""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from base.schemas.worker import ( + PHALA_TDX_TIER, + ExecutionProof, + PhalaAttestation, + PhalaMeasurement, + WorkerSignature, +) +from base.validator.agent.signing import KeypairRequestSigner +from base.worker.proof import ( + PHALA_REPORT_DATA_TAG, + build_phala_execution_proof, + phala_report_data, + phala_report_data_hex, + verify_execution_proof, +) + +MANIFEST = "a" * 64 +UNIT_ID = "submission-phala-1" + + +def _signer(uri: str = "//WorkerPhala") -> KeypairRequestSigner: + import bittensor as bt + + return KeypairRequestSigner(bt.Keypair.create_from_uri(uri)) + + +def _measurement(rtmr3: str = "d" * 96) -> PhalaMeasurement: + return PhalaMeasurement( + mrtd="a" * 96, + rtmr0="b0" * 48, + rtmr1="b1" * 48, + rtmr2="b2" * 48, + rtmr3=rtmr3, + compose_hash="c" * 64, + os_image_hash="e" * 64, + ) + + +def _attestation() -> PhalaAttestation: + return PhalaAttestation( + tdx_quote="0xdeadbeef", + event_log=[{"event": "compose-hash", "digest": "c" * 64}], + report_data="ab" * 64, + measurement=_measurement(), + vm_config={"vcpu": 1, "memory_mb": 2048}, + ) + + +def _report_data_kwargs() -> dict[str, object]: + return dict( + canonical_measurement=_measurement(), + agent_hash="f" * 64, + task_ids=["task-b", "task-a", "task-c"], + scores_digest="9" * 64, + validator_nonce="nonce-123", + ) + + +# --- tier constant + schema ------------------------------------------------ + + +def test_phala_tdx_tier_constant_value() -> None: + assert PHALA_TDX_TIER == "phala-tdx" + + +def test_phala_measurement_canonical_excludes_rtmr3() -> None: + canonical = _measurement().canonical() + assert set(canonical) == { + "mrtd", + "rtmr0", + "rtmr1", + "rtmr2", + "compose_hash", + "os_image_hash", + } + assert "rtmr3" not in canonical + + +def test_phala_attestation_round_trips() -> None: + att = _attestation() + dumped = att.model_dump(mode="json") + assert set(dumped) >= { + "tdx_quote", + "event_log", + "report_data", + "measurement", + "vm_config", + } + assert set(dumped["measurement"]) == { + "mrtd", + "rtmr0", + "rtmr1", + "rtmr2", + "rtmr3", + "compose_hash", + "os_image_hash", + } + assert PhalaAttestation.model_validate(dumped) == att + + +def test_phala_attestation_accepts_architecture_aliases() -> None: + att = PhalaAttestation.model_validate( + { + "tdx_quote_b64": "0xcafe", + "report_data_hex": "ff" * 64, + "measurement": _measurement().model_dump(), + } + ) + assert att.tdx_quote == "0xcafe" + assert att.report_data == "ff" * 64 + # serialization uses the canonical field names. + assert "tdx_quote" in att.model_dump() + assert "report_data" in att.model_dump() + + +@pytest.mark.parametrize("missing", ["tdx_quote", "report_data", "measurement"]) +def test_phala_attestation_requires_core_fields(missing: str) -> None: + payload = { + "tdx_quote": "0xdead", + "report_data": "ab" * 64, + "measurement": _measurement().model_dump(), + } + payload.pop(missing) + with pytest.raises(ValidationError): + PhalaAttestation.model_validate(payload) + + +# --- report_data derivation (architecture sec 6) --------------------------- + + +def test_report_data_is_deterministic_32_bytes() -> None: + a = phala_report_data(**_report_data_kwargs()) # type: ignore[arg-type] + b = phala_report_data(**_report_data_kwargs()) # type: ignore[arg-type] + assert a == b + assert isinstance(a, bytes) + assert len(a) == 32 + + +def test_report_data_tag_is_bound() -> None: + assert PHALA_REPORT_DATA_TAG == "base-agent-challenge-v1" + + +def test_report_data_task_ids_order_independent() -> None: + kwargs = _report_data_kwargs() + kwargs["task_ids"] = ["task-a", "task-b", "task-c"] + forward = phala_report_data(**kwargs) # type: ignore[arg-type] + kwargs["task_ids"] = ["task-c", "task-a", "task-b"] + shuffled = phala_report_data(**kwargs) # type: ignore[arg-type] + assert forward == shuffled + + +def test_report_data_ignores_rtmr3_runtime_register() -> None: + base_kwargs = _report_data_kwargs() + base_kwargs["canonical_measurement"] = _measurement(rtmr3="d" * 96) + other = _report_data_kwargs() + other["canonical_measurement"] = _measurement(rtmr3="7" * 96) + assert phala_report_data(**base_kwargs) == phala_report_data(**other) # type: ignore[arg-type] + + +def test_report_data_sensitive_to_every_bound_component() -> None: + base_digest = phala_report_data(**_report_data_kwargs()) # type: ignore[arg-type] + + changed_measurement = _report_data_kwargs() + m = _measurement() + m.compose_hash = "0" * 64 + changed_measurement["canonical_measurement"] = m + + changed_agent = _report_data_kwargs() + changed_agent["agent_hash"] = "0" * 64 + + changed_tasks = _report_data_kwargs() + changed_tasks["task_ids"] = ["task-a", "task-b"] + + changed_scores = _report_data_kwargs() + changed_scores["scores_digest"] = "0" * 64 + + changed_nonce = _report_data_kwargs() + changed_nonce["validator_nonce"] = "nonce-999" + + for perturbed in ( + changed_measurement, + changed_agent, + changed_tasks, + changed_scores, + changed_nonce, + ): + assert phala_report_data(**perturbed) != base_digest # type: ignore[arg-type] + + +def test_report_data_nonce_changes_digest() -> None: + a = _report_data_kwargs() + a["validator_nonce"] = "nonce-A" + b = _report_data_kwargs() + b["validator_nonce"] = "nonce-B" + assert phala_report_data(**a) != phala_report_data(**b) # type: ignore[arg-type] + + +def test_report_data_accepts_measurement_mapping() -> None: + as_model = phala_report_data(**_report_data_kwargs()) # type: ignore[arg-type] + kwargs = _report_data_kwargs() + kwargs["canonical_measurement"] = _measurement().model_dump() + as_mapping = phala_report_data(**kwargs) # type: ignore[arg-type] + assert as_model == as_mapping + + +def test_report_data_hex_is_64_byte_zero_padded_field() -> None: + digest = phala_report_data(**_report_data_kwargs()) # type: ignore[arg-type] + hex_field = phala_report_data_hex(**_report_data_kwargs()) # type: ignore[arg-type] + assert len(hex_field) == 128 + field_bytes = bytes.fromhex(hex_field) + assert len(field_bytes) == 64 + assert field_bytes[:32] == digest + assert field_bytes[32:] == b"\x00" * 32 + + +# --- pinned cross-repo golden vector (drift guard) -------------------------- +# This fixed input -> expected digest/field is asserted in BOTH repos against +# their independent sec-6 implementations, using the same inputs +# (``_report_data_kwargs`` here): +# base: base.worker.proof.phala_report_data(_hex) +# agent-challenge: agent_challenge.canonical.report_data.report_data(_hex) +# (tests/test_canonical_report_data.py::GOLDEN_DIGEST_HEX) +# The agent-challenge helper is a self-contained replica because base is not +# importable inside the canonical eval image; if either implementation drifts, +# one repo's pinned-vector test fails. Do NOT change one side without the other. +GOLDEN_REPORT_DATA_DIGEST_HEX = ( + "dd2c57688b55e25df20e292b71e1cb97d8501e9280e1dd3475b3e61c30e38cc2" +) +GOLDEN_REPORT_DATA_FIELD_HEX = GOLDEN_REPORT_DATA_DIGEST_HEX + "00" * 32 + + +def test_report_data_matches_pinned_cross_repo_vector() -> None: + assert ( + phala_report_data(**_report_data_kwargs()).hex() # type: ignore[arg-type] + == GOLDEN_REPORT_DATA_DIGEST_HEX + ) + assert ( + phala_report_data_hex(**_report_data_kwargs()) # type: ignore[arg-type] + == GOLDEN_REPORT_DATA_FIELD_HEX + ) + + +# --- build_phala_execution_proof ------------------------------------------- + + +def test_build_phala_execution_proof_sets_tier_and_attestation() -> None: + signer = _signer() + proof = build_phala_execution_proof( + signer=signer, + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=_attestation(), + ) + assert proof.tier == PHALA_TDX_TIER + assert proof.attestation is not None + assert proof.attestation["tdx_quote"] == "0xdeadbeef" + assert proof.attestation["measurement"]["mrtd"] == "a" * 96 + assert proof.worker_signature.worker_pubkey == signer.hotkey + + +def test_build_phala_execution_proof_accepts_attestation_mapping() -> None: + signer = _signer() + proof = build_phala_execution_proof( + signer=signer, + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation={ + "tdx_quote_b64": "0xfeed", + "report_data_hex": "ab" * 64, + "measurement": _measurement().model_dump(), + }, + ) + assert proof.attestation is not None + assert proof.attestation["tdx_quote"] == "0xfeed" + + +def test_phala_proof_worker_signature_still_verifies() -> None: + signer = _signer() + proof = build_phala_execution_proof( + signer=signer, + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=_attestation(), + ) + assert verify_execution_proof(proof, unit_id=UNIT_ID) is True + assert verify_execution_proof(proof, unit_id="other-unit") is False + + +def test_phala_proof_round_trips_through_serialization() -> None: + signer = _signer() + proof = build_phala_execution_proof( + signer=signer, + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=_attestation(), + ) + restored = ExecutionProof.model_validate(proof.model_dump(mode="json")) + assert restored == proof + assert restored.tier == PHALA_TDX_TIER + att = PhalaAttestation.model_validate(restored.attestation) + assert att.measurement.compose_hash == "c" * 64 + + +# --- existing tier behavior unchanged -------------------------------------- + + +def test_execution_proof_int_tier_unchanged() -> None: + proof = ExecutionProof( + version=1, + tier=2, + manifest_sha256=MANIFEST, + worker_signature=WorkerSignature(worker_pubkey="pk", sig="0x00"), + ) + assert proof.tier == 2 + assert isinstance(proof.tier, int) + + +def test_execution_proof_string_tier_supported() -> None: + proof = ExecutionProof( + version=1, + tier=PHALA_TDX_TIER, + manifest_sha256=MANIFEST, + worker_signature=WorkerSignature(worker_pubkey="pk", sig="0x00"), + ) + assert proof.tier == "phala-tdx" + assert isinstance(proof.tier, str) + + +# --- cross-repo emitted-envelope conformance (drift guard) ------------------ +# The agent-challenge canonical image emits the attested-result envelope below +# (fixed input vector) on the ``execution_proof`` key of its +# ``BASE_BENCHMARK_RESULT=`` line. base's ``ExecutionProof``/``PhalaAttestation`` +# are not importable inside the lean image, so the emitter builds plain dicts and +# validates them with a self-contained conformance check; this pins the EXACT +# emitted shape against base's REAL models so the two cannot drift (VAL-IMG-025 / +# VAL-IMG-026). Regenerate via agent-challenge: +# agent_challenge.canonical.attested_result.emit_attested_benchmark_result(...) +# (see agent-challenge tests/test_canonical_attested_result.py). Do NOT change +# one side without the other. +GOLDEN_EMITTED_ENVELOPE: dict[str, object] = { + "version": 1, + "tier": "phala-tdx", + "manifest_sha256": "1" * 64, + "worker_signature": {"worker_pubkey": "", "sig": ""}, + "attestation": { + "tdx_quote": "abababababababab", + "event_log": [{"imr": 3, "event": "compose-hash", "digest": "c" * 64}], + "report_data": ( + "807faf7c13ac9798f2f841ad9f05949a19d6bb1ab0833f67101a5d3ce2bbaa1d" + + "00" * 32 + ), + "measurement": { + "mrtd": "a" * 96, + "rtmr0": "b0" * 48, + "rtmr1": "b1" * 48, + "rtmr2": "b2" * 48, + "rtmr3": "d" * 96, + "compose_hash": "c" * 64, + "os_image_hash": "e" * 64, + }, + "vm_config": {"vcpu": 1, "memory_mb": 2048}, + }, +} + + +def test_emitted_envelope_validates_against_execution_proof() -> None: + proof = ExecutionProof.model_validate(GOLDEN_EMITTED_ENVELOPE) + assert proof.tier == PHALA_TDX_TIER + assert proof.attestation is not None + att = PhalaAttestation.model_validate(proof.attestation) + assert att.tdx_quote == "abababababababab" + assert att.measurement.compose_hash == "c" * 64 + assert att.measurement.rtmr3 == "d" * 96 + # report_data is the 64-byte (128-hex) TDX field, left-aligned + zero-padded. + assert len(att.report_data) == 128 + assert att.report_data.endswith("00" * 32) + + +@pytest.mark.parametrize("missing", ["manifest_sha256", "worker_signature"]) +def test_emitted_envelope_missing_required_field_rejected(missing: str) -> None: + payload = {k: v for k, v in GOLDEN_EMITTED_ENVELOPE.items() if k != missing} + with pytest.raises(ValidationError): + ExecutionProof.model_validate(payload) + + +@pytest.mark.parametrize("missing", ["tdx_quote", "report_data", "measurement"]) +def test_emitted_attestation_missing_required_field_rejected(missing: str) -> None: + attestation = { + k: v + for k, v in GOLDEN_EMITTED_ENVELOPE["attestation"].items() # type: ignore[attr-defined] + if k != missing + } + with pytest.raises(ValidationError): + PhalaAttestation.model_validate(attestation) diff --git a/tests/unit/test_worker_proof_phala_verify.py b/tests/unit/test_worker_proof_phala_verify.py new file mode 100644 index 000000000..48af9e8c2 --- /dev/null +++ b/tests/unit/test_worker_proof_phala_verify.py @@ -0,0 +1,870 @@ +"""Phala-tier quote verifier for ExecutionProof (M4, VAL-VERIFY-001..014). + +Black-box behavioral tests for the validator/master Phala-tier verifier added to +``verify_execution_proof``: it accepts a wholly-valid attested envelope and +rejects every tamper / mismatch / stale / repurposed case, keeps the tier-0 +worker-signature + cross-unit-replay checks in force on the Phala tier, and +PARKS (raises, does not accept nor fraud-reject) when the quote-verification +dependency is transiently unavailable. + +Quotes are assembled offline with the base ``phala_quote`` helpers (the inverse +of the parser); the DCAP signature/TCB layer is modeled with an injectable +:class:`StaticQuoteVerifier`, and a fake-runner :class:`DcapQvlVerifier` test +pins the CLI accept/reject/park mapping. A real ``dcap-qvl`` run is an M6 live +assertion. +""" + +from __future__ import annotations + +import hashlib +import json +import subprocess + +import pytest + +from base.schemas.worker import ( + PHALA_TDX_TIER, + ExecutionProof, + PhalaAttestation, + PhalaMeasurement, + WorkerSignature, +) +from base.validator.agent.adapters.agent_challenge import rebind_worker_signature +from base.validator.agent.signing import KeypairRequestSigner +from base.worker.phala_quote import ( + DcapQvlVerifier, + QuoteVerificationError, + StaticQuoteVerifier, + VerifierUnavailableError, + build_rtmr3_event_log, + build_tdx_quote, +) +from base.worker.phala_verify import ( + InMemoryNonceValidator, + MeasurementAllowlist, + PhalaBinding, +) +from base.worker.proof import ( + PHALA_REPORT_DATA_TAG, + build_phala_execution_proof, + phala_report_data_hex, + verify_execution_proof, +) + +MANIFEST = "a" * 64 +UNIT_ID = "submission-verify-1" + +MRTD = "a1" * 48 +RTMR0 = "b0" * 48 +RTMR1 = "b1" * 48 +RTMR2 = "b2" * 48 +COMPOSE_PAYLOAD = bytes.fromhex("c3" * 32) + +AGENT_HASH = "f0" * 32 +TASK_IDS = ("task-b", "task-a", "task-c") +SCORES_DIGEST = "9a" * 32 + + +def _signer(uri: str = "//WorkerVerify") -> KeypairRequestSigner: + import bittensor as bt + + return KeypairRequestSigner(bt.Keypair.create_from_uri(uri)) + + +def _os_image_hash(mrtd: str, rtmr1: str, rtmr2: str) -> str: + preimage = bytes.fromhex(mrtd) + bytes.fromhex(rtmr1) + bytes.fromhex(rtmr2) + return hashlib.sha256(preimage).hexdigest() + + +def _build_attestation( + *, + agent_hash: str = AGENT_HASH, + task_ids: tuple[str, ...] = TASK_IDS, + scores_digest: str = SCORES_DIGEST, + validator_nonce: str, + compose_payload: bytes = COMPOSE_PAYLOAD, + mrtd: str = MRTD, + rtmr0: str = RTMR0, + rtmr1: str = RTMR1, + rtmr2: str = RTMR2, + report_data_hex: str | None = None, +) -> tuple[PhalaAttestation, dict[str, str]]: + """A self-consistent Phala attestation + its reconstructed canonical measurement.""" + + event_log, rtmr3 = build_rtmr3_event_log([("compose-hash", compose_payload)]) + compose_hash = compose_payload.hex() + os_image_hash = _os_image_hash(mrtd, rtmr1, rtmr2) + measurement = { + "mrtd": mrtd, + "rtmr0": rtmr0, + "rtmr1": rtmr1, + "rtmr2": rtmr2, + "compose_hash": compose_hash, + "os_image_hash": os_image_hash, + } + if report_data_hex is None: + report_data_hex = phala_report_data_hex( + canonical_measurement=measurement, + agent_hash=agent_hash, + task_ids=task_ids, + scores_digest=scores_digest, + validator_nonce=validator_nonce, + ) + quote = build_tdx_quote( + mrtd=mrtd, + rtmr0=rtmr0, + rtmr1=rtmr1, + rtmr2=rtmr2, + rtmr3=rtmr3, + report_data=report_data_hex, + ) + attestation = PhalaAttestation( + tdx_quote=quote, + event_log=event_log, + report_data=report_data_hex, + measurement=PhalaMeasurement( + mrtd=mrtd, + rtmr0=rtmr0, + rtmr1=rtmr1, + rtmr2=rtmr2, + rtmr3=rtmr3, + compose_hash=compose_hash, + os_image_hash=os_image_hash, + ), + vm_config={"vcpu": 1, "memory_mb": 2048}, + ) + return attestation, measurement + + +def _allowlist(measurement: dict[str, str]) -> MeasurementAllowlist: + return MeasurementAllowlist.from_measurements([measurement]) + + +def _fixture( + *, + unit_id: str = UNIT_ID, + agent_hash: str = AGENT_HASH, + task_ids: tuple[str, ...] = TASK_IDS, + scores_digest: str = SCORES_DIGEST, +): + """A fully-valid (proof, binding, verifier, allowlist, nonce store) tuple.""" + + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + attestation, measurement = _build_attestation( + agent_hash=agent_hash, + task_ids=task_ids, + scores_digest=scores_digest, + validator_nonce=nonce, + ) + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=unit_id, + attestation=attestation, + ) + binding = PhalaBinding( + agent_hash=agent_hash, + task_ids=task_ids, + scores_digest=scores_digest, + validator_nonce=nonce, + ) + return proof, binding, StaticQuoteVerifier(), _allowlist(measurement), nonces + + +def _verify(proof, binding, verifier, allowlist, nonces, *, unit_id: str = UNIT_ID): + return verify_execution_proof( + proof, + unit_id=unit_id, + expected_binding=binding, + quote_verifier=verifier, + allowlist=allowlist, + nonce_validator=nonces, + ) + + +# --- VAL-VERIFY-001: fully valid attested result is ACCEPTED ---------------- + + +def test_valid_attested_result_accepts() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + assert _verify(proof, binding, verifier, allowlist, nonces) is True + + +def test_backward_compatible_tier0_only_without_binding() -> None: + # No expected_binding => existing tier-0 signature semantics, unchanged. + proof, _binding, _verifier, _allowlist, _nonces = _fixture() + assert verify_execution_proof(proof, unit_id=UNIT_ID) is True + assert verify_execution_proof(proof, unit_id="other") is False + + +# --- VAL-VERIFY-002: invalid/forged quote signature is REJECTED ------------- + + +def test_forged_quote_signature_rejected() -> None: + proof, binding, _verifier, allowlist, nonces = _fixture() + invalid = StaticQuoteVerifier(valid=False) + assert _verify(proof, binding, invalid, allowlist, nonces) is False + + +def test_dcap_qvl_nonzero_exit_is_reject_not_park() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, returncode=1, stdout="", stderr="bad sig" + ) + + verifier = DcapQvlVerifier(runner=runner) + with pytest.raises(QuoteVerificationError): + verifier.verify("00" * 8) + + +# --- VAL-VERIFY-003: out-of-date / invalid TCB is REJECTED ------------------ + + +def test_out_of_date_tcb_rejected() -> None: + proof, binding, _verifier, allowlist, nonces = _fixture() + stale_tcb = StaticQuoteVerifier(tcb_status="OutOfDate") + assert _verify(proof, binding, stale_tcb, allowlist, nonces) is False + + +def test_dcap_qvl_reports_tcb_status() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args, returncode=0, stdout=json.dumps({"status": "OutOfDate"}), stderr="" + ) + + verdict = DcapQvlVerifier(runner=runner).verify("00" * 8) + assert verdict.tcb_status == "OutOfDate" + + +# --- VAL-VERIFY-004: measurement not in the validator allowlist is REJECTED -- + + +def test_measurement_not_in_allowlist_rejected_then_accepted() -> None: + proof, binding, verifier, _allowlist, nonces = _fixture() + empty = MeasurementAllowlist() + assert _verify(proof, binding, verifier, empty, nonces) is False + + # The identical quote accepted once its measurement is added to the allowlist. + _p2, _b2, _v2, allowlist_ok, _n2 = _fixture() + assert _verify(_p2, _b2, _v2, allowlist_ok, _n2) is True + + +def test_measurement_single_register_mismatch_rejected() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + # Allowlist a measurement whose MRTD differs by one register. + other = MeasurementAllowlist.from_measurements( + [ + { + "mrtd": "ff" * 48, + "rtmr0": RTMR0, + "rtmr1": RTMR1, + "rtmr2": RTMR2, + "compose_hash": COMPOSE_PAYLOAD.hex(), + "os_image_hash": _os_image_hash(MRTD, RTMR1, RTMR2), + } + ] + ) + assert _verify(proof, binding, verifier, other, nonces) is False + + +# --- VAL-VERIFY-005: event-log / RTMR3 that does not replay is REJECTED ------ + + +def test_event_log_replay_mismatch_rejected() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + att = PhalaAttestation.model_validate(proof.attestation) + # Swap in an event log that replays to a DIFFERENT RTMR3 than the quote binds. + other_log, _other_rtmr3 = build_rtmr3_event_log( + [("compose-hash", bytes.fromhex("dd" * 32))] + ) + att.event_log = other_log + proof.attestation = att.model_dump(mode="json") + assert _verify(proof, binding, verifier, allowlist, nonces) is False + + +def test_event_log_internally_inconsistent_rejected() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + att = PhalaAttestation.model_validate(proof.attestation) + # Mutate a payload so its logged digest no longer matches (self-inconsistent). + att.event_log[0]["event_payload"] = "ee" * 32 + proof.attestation = att.model_dump(mode="json") + assert _verify(proof, binding, verifier, allowlist, nonces) is False + + +# --- VAL-VERIFY-006: wrong agent_hash is REJECTED --------------------------- + + +def test_wrong_agent_hash_rejected() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + wrong = PhalaBinding( + agent_hash="00" * 32, + task_ids=binding.task_ids, + scores_digest=binding.scores_digest, + validator_nonce=binding.validator_nonce, + ) + assert _verify(proof, wrong, verifier, allowlist, nonces) is False + + +# --- VAL-VERIFY-007: wrong task set REJECTED; reorder still ACCEPTED --------- + + +def test_wrong_task_set_rejected() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + wrong = PhalaBinding( + agent_hash=binding.agent_hash, + task_ids=("task-a", "task-b"), + scores_digest=binding.scores_digest, + validator_nonce=binding.validator_nonce, + ) + assert _verify(proof, wrong, verifier, allowlist, nonces) is False + + +def test_task_set_reordered_still_accepted() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + reordered = PhalaBinding( + agent_hash=binding.agent_hash, + task_ids=("task-c", "task-b", "task-a"), + scores_digest=binding.scores_digest, + validator_nonce=binding.validator_nonce, + ) + assert _verify(proof, reordered, verifier, allowlist, nonces) is True + + +# --- VAL-VERIFY-008: wrong scores is REJECTED ------------------------------- + + +def test_wrong_scores_digest_rejected() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + wrong = PhalaBinding( + agent_hash=binding.agent_hash, + task_ids=binding.task_ids, + scores_digest="00" * 32, + validator_nonce=binding.validator_nonce, + ) + assert _verify(proof, wrong, verifier, allowlist, nonces) is False + + +# --- VAL-VERIFY-009: wrong domain-separation tag is REJECTED ---------------- + + +def test_wrong_domain_tag_rejected() -> None: + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + # Build a report_data whose preimage uses a DIFFERENT tag but is otherwise + # correctly bound; the verifier recomputes with PHALA_REPORT_DATA_TAG. + os_image_hash = _os_image_hash(MRTD, RTMR1, RTMR2) + measurement = { + "mrtd": MRTD, + "rtmr0": RTMR0, + "rtmr1": RTMR1, + "rtmr2": RTMR2, + "compose_hash": COMPOSE_PAYLOAD.hex(), + "os_image_hash": os_image_hash, + } + preimage = { + "tag": "some-other-protocol-v9", + "canonical_measurement": measurement, + "agent_hash": AGENT_HASH, + "task_ids": sorted(TASK_IDS), + "scores_digest": SCORES_DIGEST, + "validator_nonce": nonce, + } + digest = hashlib.sha256( + json.dumps(preimage, sort_keys=True, separators=(",", ":")).encode() + ).digest() + report_data_hex = digest.ljust(64, b"\x00").hex() + assert PHALA_REPORT_DATA_TAG not in preimage["tag"] + + attestation, _m = _build_attestation( + validator_nonce=nonce, report_data_hex=report_data_hex + ) + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=attestation, + ) + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=nonce, + ) + assert ( + _verify(proof, binding, StaticQuoteVerifier(), _allowlist(measurement), nonces) + is False + ) + + +# --- VAL-VERIFY-010: stale or replayed validator nonce is REJECTED ---------- + + +def test_fresh_nonce_first_use_accepts_then_reuse_rejected() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + assert _verify(proof, binding, verifier, allowlist, nonces) is True + # Same nonce reused (consumed) => rejected. + assert _verify(proof, binding, verifier, allowlist, nonces) is False + + +def test_never_issued_nonce_rejected() -> None: + nonces = InMemoryNonceValidator() + attestation, measurement = _build_attestation(validator_nonce="never-issued-nonce") + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=attestation, + ) + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce="never-issued-nonce", + ) + assert ( + _verify(proof, binding, StaticQuoteVerifier(), _allowlist(measurement), nonces) + is False + ) + + +def test_expired_nonce_rejected() -> None: + clock = {"t": 1000.0} + nonces = InMemoryNonceValidator(ttl_seconds=120, clock=lambda: clock["t"]) + nonce = nonces.issue() + attestation, measurement = _build_attestation(validator_nonce=nonce) + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=attestation, + ) + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=nonce, + ) + clock["t"] = 2000.0 # advance well past the TTL + assert ( + _verify(proof, binding, StaticQuoteVerifier(), _allowlist(measurement), nonces) + is False + ) + + +# --- VAL-VERIFY-011: absent nonce in report_data is REJECTED ---------------- + + +def test_absent_nonce_rejected() -> None: + nonces = InMemoryNonceValidator() + attestation, measurement = _build_attestation(validator_nonce="") + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=attestation, + ) + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce="", + ) + assert ( + _verify(proof, binding, StaticQuoteVerifier(), _allowlist(measurement), nonces) + is False + ) + + +# --- VAL-VERIFY-012: genuine quote repurposed from a different submission ---- + + +def test_repurposed_quote_rejected_for_other_submission() -> None: + # Submission B: a genuine, self-consistent, valid attested envelope. + nonces = InMemoryNonceValidator() + nonce_b = nonces.issue() + attestation_b, measurement = _build_attestation( + agent_hash="bb" * 32, + task_ids=("b-task-1", "b-task-2"), + scores_digest="cc" * 32, + validator_nonce=nonce_b, + ) + proof_b = ExecutionProof( + version=1, + tier=PHALA_TDX_TIER, + manifest_sha256=MANIFEST, + worker_signature=WorkerSignature(worker_pubkey="", sig=""), + attestation=attestation_b.model_dump(mode="json"), + ) + # Present B's envelope as submission A's result (A's expected binding). + nonce_a = nonces.issue() + binding_a = PhalaBinding( + agent_hash="aa" * 32, + task_ids=("a-task-1", "a-task-2"), + scores_digest="dd" * 32, + validator_nonce=nonce_a, + ) + proof_a = rebind_worker_signature(proof_b, signer=_signer(), unit_id="submission-A") + assert ( + verify_execution_proof( + proof_a, + unit_id="submission-A", + expected_binding=binding_a, + quote_verifier=StaticQuoteVerifier(), + allowlist=_allowlist(measurement), + nonce_validator=nonces, + ) + is False + ) + + +# --- VAL-VERIFY-013: tier-0 worker signature + unit binding still enforced --- + + +def test_cross_unit_replay_rejected_on_phala_tier() -> None: + proof, binding, verifier, allowlist, nonces = _fixture() + # Valid for the signed unit, rejected when presented for another unit. + assert ( + _verify(proof, binding, verifier, allowlist, nonces, unit_id="other-unit") + is False + ) + + +def test_placeholder_worker_signature_rejected() -> None: + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + attestation, measurement = _build_attestation(validator_nonce=nonce) + # The M1 image emits an EMPTY placeholder worker_signature. + proof = ExecutionProof( + version=1, + tier=PHALA_TDX_TIER, + manifest_sha256=MANIFEST, + worker_signature=WorkerSignature(worker_pubkey="", sig=""), + attestation=attestation.model_dump(mode="json"), + ) + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=nonce, + ) + assert ( + _verify(proof, binding, StaticQuoteVerifier(), _allowlist(measurement), nonces) + is False + ) + + +def test_rebind_makes_placeholder_envelope_verifiable() -> None: + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + attestation, measurement = _build_attestation(validator_nonce=nonce) + placeholder = ExecutionProof( + version=1, + tier=PHALA_TDX_TIER, + manifest_sha256=MANIFEST, + worker_signature=WorkerSignature(worker_pubkey="", sig=""), + attestation=attestation.model_dump(mode="json"), + ) + signer = _signer("//Validator") + bound = rebind_worker_signature(placeholder, signer=signer, unit_id=UNIT_ID) + assert bound.worker_signature.worker_pubkey == signer.hotkey + assert bound.tier == PHALA_TDX_TIER + assert bound.attestation == placeholder.attestation + + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=nonce, + ) + assert ( + _verify(bound, binding, StaticQuoteVerifier(), _allowlist(measurement), nonces) + is True + ) + # Re-bound to UNIT_ID => cross-unit replay still rejected. + nonces2 = InMemoryNonceValidator() + nonce2 = nonces2.issue() + att2, m2 = _build_attestation(validator_nonce=nonce2) + placeholder2 = ExecutionProof( + version=1, + tier=PHALA_TDX_TIER, + manifest_sha256=MANIFEST, + worker_signature=WorkerSignature(worker_pubkey="", sig=""), + attestation=att2.model_dump(mode="json"), + ) + bound2 = rebind_worker_signature(placeholder2, signer=signer, unit_id=UNIT_ID) + binding2 = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=nonce2, + ) + assert ( + verify_execution_proof( + bound2, + unit_id="different-unit", + expected_binding=binding2, + quote_verifier=StaticQuoteVerifier(), + allowlist=_allowlist(m2), + nonce_validator=nonces2, + ) + is False + ) + + +def test_non_phala_tier_with_binding_rejected() -> None: + # A tier-0-only proof presented where a Phala attestation is expected. + from base.worker.proof import build_execution_proof + + proof = build_execution_proof( + signer=_signer(), manifest_sha256=MANIFEST, unit_id=UNIT_ID + ) + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce="n", + ) + assert ( + verify_execution_proof( + proof, + unit_id=UNIT_ID, + expected_binding=binding, + quote_verifier=StaticQuoteVerifier(), + allowlist=MeasurementAllowlist(), + nonce_validator=InMemoryNonceValidator(), + ) + is False + ) + + +# --- VAL-VERIFY-014: verifier transient unavailability/timeout PARKS --------- + + +def test_verifier_outage_parks_not_accepts_nor_rejects() -> None: + proof, binding, _verifier, allowlist, nonces = _fixture() + outage = StaticQuoteVerifier(unavailable=True) + with pytest.raises(VerifierUnavailableError): + _verify(proof, binding, outage, allowlist, nonces) + + +def test_parked_result_accepts_once_verifier_restored() -> None: + proof, binding, _verifier, allowlist, nonces = _fixture() + outage = StaticQuoteVerifier(unavailable=True) + with pytest.raises(VerifierUnavailableError): + _verify(proof, binding, outage, allowlist, nonces) + # The park did not consume the nonce nor fraud-reject: a later pass accepts. + assert _verify(proof, binding, StaticQuoteVerifier(), allowlist, nonces) is True + + +def test_dcap_qvl_timeout_is_park_not_reject() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + raise subprocess.TimeoutExpired(cmd=args, timeout=1.0) + + verifier = DcapQvlVerifier(runner=runner) + with pytest.raises(VerifierUnavailableError): + verifier.verify("00" * 8) + + +def test_dcap_qvl_missing_binary_is_park_not_reject() -> None: + def runner(args: list[str]) -> subprocess.CompletedProcess[str]: + raise FileNotFoundError("dcap-qvl not found") + + verifier = DcapQvlVerifier(runner=runner) + with pytest.raises(VerifierUnavailableError): + verifier.verify("00" * 8) + + +# --- VAL-VERIFY-025: empty/missing/unparseable allowlist fails closed -------- + + +def test_empty_or_missing_allowlist_fails_closed_then_populated_accepts() -> None: + # A genuine, fully-valid attested envelope that WOULD verify against a + # populated allowlist. An empty/missing allowlist must reject it (never + # accept-any) WITHOUT consuming the nonce, so the same valid result still + # accepts once the allowlist is populated. + proof, binding, verifier, populated, nonces = _fixture() + + assert ( + verify_execution_proof( + proof, + unit_id=UNIT_ID, + expected_binding=binding, + quote_verifier=verifier, + allowlist=MeasurementAllowlist(), + nonce_validator=nonces, + ) + is False + ) + # Absent/unconfigured allowlist (None) also fails closed. + assert ( + verify_execution_proof( + proof, + unit_id=UNIT_ID, + expected_binding=binding, + quote_verifier=verifier, + allowlist=None, + nonce_validator=nonces, + ) + is False + ) + # The SAME valid result now accepts against the populated allowlist. + assert _verify(proof, binding, verifier, populated, nonces) is True + + +def test_unparseable_or_missing_allowlist_config_fails_closed(tmp_path) -> None: + nonces = InMemoryNonceValidator() + nonce = nonces.issue() + attestation, measurement = _build_attestation(validator_nonce=nonce) + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=UNIT_ID, + attestation=attestation, + ) + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=nonce, + ) + verifier = StaticQuoteVerifier() + + # Unparseable config => empty allowlist => reject (nonce not consumed). + unparseable = MeasurementAllowlist.from_json("{ not valid json ]") + assert bool(unparseable) is False + assert ( + verify_execution_proof( + proof, + unit_id=UNIT_ID, + expected_binding=binding, + quote_verifier=verifier, + allowlist=unparseable, + nonce_validator=nonces, + ) + is False + ) + # Missing config file => empty allowlist => reject (nonce not consumed). + missing = MeasurementAllowlist.from_file(tmp_path / "absent.json") + assert bool(missing) is False + assert ( + verify_execution_proof( + proof, + unit_id=UNIT_ID, + expected_binding=binding, + quote_verifier=verifier, + allowlist=missing, + nonce_validator=nonces, + ) + is False + ) + # A valid config carrying the measurement accepts the SAME valid result. + config = tmp_path / "allowlist.json" + config.write_text(json.dumps({"entries": [measurement]}), encoding="utf-8") + populated = MeasurementAllowlist.from_file(config) + assert bool(populated) is True + assert ( + verify_execution_proof( + proof, + unit_id=UNIT_ID, + expected_binding=binding, + quote_verifier=verifier, + allowlist=populated, + nonce_validator=nonces, + ) + is True + ) + + +# --- VAL-VERIFY-027: multi-entry allowlist (image rotation) ------------------ + + +def _rotation_case( + nonces: InMemoryNonceValidator, + *, + unit_id: str, + mrtd: str, + compose_payload: bytes, +): + """A self-consistent (proof, binding, measurement) for a given image.""" + + nonce = nonces.issue() + attestation, measurement = _build_attestation( + validator_nonce=nonce, mrtd=mrtd, compose_payload=compose_payload + ) + proof = build_phala_execution_proof( + signer=_signer(), + manifest_sha256=MANIFEST, + unit_id=unit_id, + attestation=attestation, + ) + binding = PhalaBinding( + agent_hash=AGENT_HASH, + task_ids=TASK_IDS, + scores_digest=SCORES_DIGEST, + validator_nonce=nonce, + ) + return proof, binding, measurement + + +def test_two_entry_allowlist_accepts_either_rejects_third() -> None: + nonces = InMemoryNonceValidator() + old_proof, old_binding, old_m = _rotation_case( + nonces, + unit_id="unit-old", + mrtd="a1" * 48, + compose_payload=bytes.fromhex("c3" * 32), + ) + new_proof, new_binding, new_m = _rotation_case( + nonces, + unit_id="unit-new", + mrtd="b2" * 48, + compose_payload=bytes.fromhex("d4" * 32), + ) + third_proof, third_binding, _third_m = _rotation_case( + nonces, + unit_id="unit-third", + mrtd="c3" * 48, + compose_payload=bytes.fromhex("e5" * 32), + ) + assert old_m != new_m + + rotation = MeasurementAllowlist.from_measurements([old_m, new_m]) + verifier = StaticQuoteVerifier() + + # Both the outgoing (old) and incoming (new) images verify against the + # two-entry allowlist during a rotation window. + assert ( + verify_execution_proof( + old_proof, + unit_id="unit-old", + expected_binding=old_binding, + quote_verifier=verifier, + allowlist=rotation, + nonce_validator=nonces, + ) + is True + ) + assert ( + verify_execution_proof( + new_proof, + unit_id="unit-new", + expected_binding=new_binding, + quote_verifier=verifier, + allowlist=rotation, + nonce_validator=nonces, + ) + is True + ) + # A third image, listed under neither entry, is rejected. + assert ( + verify_execution_proof( + third_proof, + unit_id="unit-third", + expected_binding=third_binding, + quote_verifier=verifier, + allowlist=rotation, + nonce_validator=nonces, + ) + is False + )