UCP agentic-commerce policy examples (apps/ucp) - #15
Conversation
|
Do you have a UCP test rig setup somewhere and tested these policies in an end-to-end case to make sure they work as advertised? |
|
I tested the policies and verified the specs
No. These are unit/contract-tested (verified against the real spec), but not end-to-end tested against the real thing. But we can try and go all the way (I think we should) if we want to do this; If we want to truthfully say "works as advertised end-to-end," a UCP mock (or Shopify's hosted UCP MCP server) behind a real DTwo gateway, with an agent driving create→update→complete_checkout, would allow us to pull that off. I didn't scope it yet, but I think it is very doable. |
Eight buyer-side egress governance policies for AI agents that transact over the Universal Commerce Protocol (UCP) through Shopify's hosted Cart/Checkout/Order MCP servers. Experimental — shared for discussion, not a hardened or endorsed set. Adds the `shopify` app, the `agentic-commerce` bundle, and the `retail` industry landing page; manifest regenerated (manifest:check passes). All eight verified green under opa 1.17.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
942f8b9 to
a6e815e
Compare
Findings from driving all eight policies end-to-end behind a live
gateway against a mock UCP MCP server:
- Gateways slugify federated tool names ("ucp-shop-complete-checkout"),
so suffix-matching the underscored name alone never fires and deny
policies fail open. All eight now match hyphenated/underscored/
collapsed shapes anchored at a separator, plus the bare un-prefixed
name (the apps/slack matching convention). New slugified-name deny
fixtures lock this in per policy.
- restricted-category-block read line_items/messages from top-level
args, unlike its siblings; it now reads the canonical args.checkout.*
shape first, keeping the old shape as a fallback.
- READMEs: order-pii-egress-redaction gains a known limitation
(structured-result redaction does not rewrite the serialized
content[].text copy); the two stateful policies replace hedging with
verified deployment requirements (declared writable session keys,
matching write attribution); apps/shopify README notes the set is
exercised end-to-end.
opa check --strict clean on all eight; 26 fixtures green; manifest
regenerated (manifest:check passes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
Followed through on this — there is now a rig, and it earned its keep immediately. The rigA mock UCP MCP server (spec-faithful checkout shapes: What end-to-end testing caught (fixes pushed in 0a6b7ee)
The stateful pair — answer to my own open questionBoth stateful policies were proven end-to-end through the live gateway: Also verified but worth stating: the org-constraint inputs ( Remaining maintainer calls (would like your take, @denisknj)
|
Converted from the tests/*.json fixtures (same PARC inputs): allow/deny plus exact-reason and transform assertions in the runner's schema. The JSON fixtures stay: they carry the session-write expectations the runner schema cannot express, and the end-to-end run uses them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
Per review: tests.yaml (run by pnpm test) is now the single test suite; the parallel tests/*.json fixtures are removed so the two cannot drift. Case names survive as comments; the session-state expectations ride along as expectedSessionWrites (ignored by the runner, asserted by the end-to-end gateway run). READMEs updated to reference cases instead of fixture files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
|
Maintainer call #3 resolved as convert (per otto): Still open for a maintainer: the slug ( |
|
|
|
Naming:
|
…overcharge Per review: - approved-cart-integrity: the approval was a write-once, content-scoped latch — replayable, and a stale approval could authorize a later same-content completion. The approval now records the checkout id and denies completion of any other checkout, and an allowed completion consumes the approval (session write clearing the marker), so a replay fails closed. No cancel handling: no cancellation tool exists in the confirmed UCP surface; id binding plus TTL covers abandonment. New cases: deny-replay, deny-checkout-mismatch; allow now asserts the consume writes. - attribution-disclosure: strict price equality relaxed to no-overcharge (submitted <= advertised allows) — lower totals are coupons and reprice-downs, content tampering is approved-cart-integrity's job, and the ceiling is checkout-spend-cap's. New case: allow-discount. Docs reframed; strict any-drift noted as a one-comparison variant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
Conceded on all three edges — the latch was write-once and content-scoped, and your edge 3 is the policy's own threat narrative ("consent and charge separated in time") turned against it. Fixed in 6197cc3 with both hardenings:
Both of your asked-for tests are in: |
Taking A (no-overcharge) — your companion-policy argument settles it: content tampering is In 6197cc3: Next: refreshing the end-to-end rig with the current policy heads and adding live replay + discount scenarios; results to follow. |
|
|
|
The two stateful policies should converge on one pattern
These can't both be idiomatic. Suggest standardizing both on the |
Use consistent indentation across the Rego blocks
One caveat to settle at the repo level: |
|
Live end-to-end results for the 6197cc3 hardening, as promised — the rig re-vendored the policies at this PR's head and drove the new behavior through a real gateway (mock UCP merchant, real per-call decisions):
Full suite: 23 passed, 1 xfail (the pre-existing text-channel redaction finding, unchanged), three consecutive green runs. Two nice side effects of this round: the policies' native shape-set matching let us delete the tool-name shim from the rig's composition layer (one less gateway-reality workaround), and the deny-reason strings in the live assertions were all taken from evaluation, not transcription. That closes out both findings end-to-end. Remaining open items are the naming calls (slug, |
|
|
|
Vendor-neutralize the agentic-commerce example set: UCP is an open standard, so the app slug becomes ucp and packages move from shopify.ingress.*/shopify.egress.* to ucp.ingress.*/ucp.egress.*. The retail industry landing page becomes commerce. Rename high-value-approval-gate to high-value-handoff-gate. Mechanical and spec fixes on the non-completion-gating policies: - restricted-category-block: gate content where it enters the cart (create_checkout / update_checkout / update_cart) and drop the dead complete_checkout leg (completion args carry no line_items on the real arg shape). - high-value-handoff-gate: adopt the shared exactly-one-total idiom so an ambiguous totals array can never eval-conflict. - All rego reindented to 4 spaces. Remove the per-policy README.md files: the repo convention (see CONTRIBUTING) is that a policy's human docs live in its policy.md frontmatter description, and no other app ships per-policy READMEs. Their content is folded into the policy.md descriptions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
On the confirmed UCP MCP binding, complete_checkout carries only the
checkout id (top level) plus finalization data — no totals, no cart. Any
policy gating completion on cart facts must observe those facts from an
earlier checkout RESPONSE into policy-accessible session state, then
enforce at the completion request keyed on args.id.
Convert the four completion-gating policies to one shared stateful
pattern:
- Observe leg (tool_post_invoke, mode "output"): record what is needed —
an observed total, currency, or approval baseline — into one
object-valued key keyed by checkout id, merged with prior state via
object.union, discovering its own namespace by a unique marker key
(never a hard-coded writer id). Grand total is resolved only when there
is exactly one numeric type=="total" entry, else an unresolvable marker
is stored so completion fails closed.
- Enforce leg (tool_pre_invoke, mode "input"): read the observation for
args.id and decide, failing closed when nothing was observed.
- Nested decision envelope: decision := {allow, session_writes}, with the
package-level allow/reason/session_writes rules retained.
Per policy:
- checkout-spend-cap: now stateful; observed total vs mandate cap and
currency; fail closed on no/ambiguous observation or no cap.
- cumulative-spend-ceiling: marker discovery replaces the hard-coded
writer id; the running-total advance now rides the decision envelope
(it was previously emitted top-level only and was non-functional
standalone); an allowed completion consumes the observed id so a replay
cannot double-count.
- approved-cart-integrity: approval is sourced from a server
ready_for_complete RESPONSE (the request-status trigger can never fire
on the real arg shape); a non-ready response or cancel_checkout
invalidates; completion allows on a live keyed approval and consumes it;
a divergent echoed cart is denied as defense in depth. cancel_checkout
exists on the confirmed surface and is now handled.
- attribution-disclosure: the no-overcharge leg is now stateful (observed
total vs advertised); disclosure reads the completion's attribution and
is documented as an agent-supplied compliance nudge, not enforced
provenance.
Tests rewritten to the real spec arg shapes with observe cases and the
reviewer-requested deny paths (replay-after-complete, cancel-then-complete,
no-observation, duplicate type=="total", second-checkout-accumulated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
Reflects the renamed app slug (ucp), industry (commerce), policy packages, and the high-value-handoff-gate rename, with recomputed policy checksums. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
Conceded — and you also correctly called the contradiction with the "verified end-to-end" note. On inspection, the e2e composition layer was reading the package-level Fixed in fabec5b, both sides:
Your asked-for assertion is in the live suite and green: the second checkout's deny reason proves the accumulated prior total was read back — While in there the policy also adopted marker discovery (dropping the hardcoded writer-id literal — your convergence point) and gained replay protection: an allowed completion consumes its observation, so completing the same checkout twice cannot double-count. |
Both confirmed — and chasing them surfaced something bigger. Checking the extractor against the spec's The fix in fabec5b restructures the policy around where totals actually exist: it now observes the merchant's create/update/get responses — server-computed totals — into session state keyed by checkout id, and enforces at Live coverage: over-cap deny citing the observed amount ($1,390.00 — provably from the earlier response, since the completion call carried no totals), no-observation deny, duplicate-totals deny, currency-mismatch deny, under-cap allow. |
Agreed on all points, and the doc now says so in nearly your words (fabec5b): the disclosure value is agent-supplied, so the check enforces "this field is non-empty" — a compliance nudge, not enforced provenance; real assurance would need the disclosed value validated against a gateway-supplied trusted source under Two related shape fixes in the same commit: disclosure is read from where a spec-shaped completion actually carries it ( |
Done in fabec5b — and the convergence went further than the original two. After the arg-shape findings (your other thread), four policies are stateful (spend-cap, ceiling, attribution's price leg, cart-integrity), and all four now follow the single reference shape you asked for:
There's now exactly one shape for the next stateful policy to copy. |
Standardized on 4-space across all eight in fabec5b; |
You called it. Confirmed against the spec (checkout core + MCP binding, tag 2026-04-08), and it's more consequential than the id sourcing:
The rebuild in fabec5b sources everything from where it actually lives:
Full live re-run against the redesigned set: allow, deny, redaction, and all stateful flows — approve → complete → consume (asserted in session storage) → replay denied; tamper-via-update invalidates → deny; cancel → deny; approval for checkout X does not authorize Y; completion with no prior observation fails closed. Three consecutive fully-green runs (plus an independent fourth), one xfail unchanged (the documented text-channel redaction limitation). Honest scope note that fell out of this: with approval keyed to the server's |
Went with your leans on both (fabec5b):
And on the per-day-budget scoping comment: docs updated as recommended — the policy is now explicitly a per-session cumulative ceiling with sliding-TTL reset semantics (each write slides the TTL; the total never resets on a calendar boundary), pointing at a gateway-supplied trusted clock as the prerequisite for a |
|
Round 3 is in as fabec5b (three commits: renames + mechanical fixes; the stateful observe/enforce redesign; manifest regen). CI fully green — 85 tests across the catalog, every expected reason captured from real The one-paragraph version of the redesign: the spec check you prompted showed a With the naming settled per your calls ( |
…-end run The redesigned observe/enforce set has been re-verified behind a live gateway since this note was written; align the status text with the results reported on the PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
23671e0 to
36f6155
Compare
What this adds
Eight buyer-side governance policies for AI agents that transact over the Universal Commerce Protocol (UCP) through UCP checkout MCP servers — plus the vendor-neutral
ucpapp, anagentic-commercebundle, and acommerceindustry landing page.complete_checkoutwhen the observed grand total exceeds the org capget_order/get_checkoutresponsescontinue_urlFour policies are stateful because the spec forces it: a UCP
complete_checkoutrequest carries no cart content (inputs areid,signals?,attribution?,payment?) — totals and line items exist only in the merchant's responses. So each of these policies observes create/update/get responses into policy-accessible session state (keyed by checkout id) and enforces at the completion call, failing closed when the needed observation is absent. All four follow one shared reference pattern: marker discovery overinput.context.session.policies(no hardcoded writer ids), one object-valued key per concern merged with prior state, set-semantics reads, and the nesteddecision := {allow, session_writes}envelope.The idea (for discussion)
UCP standardizes how a trusted AI platform transacts with merchants, and merchants enforce fraud / payment / PII / scope server-side. What neither covers — and UCP by design leaves to "the platform" — is an enterprise's control over the agents it runs: which merchants/tools they may call, org spend and data-egress limits beyond a single AP2 consent mandate, and per-call Rego on tool args/results. These policies explore that buyer-side egress seam. Complementary to UCP and its implementors, not a competing trust referee.
Verification
opa check --strict-clean; every policy ships atests.yamlrun bypnpm test(CI-green, 85 cases repo-wide) with exact-reason assertions captured from realopaevaluation. Session-write expectations ride along asexpectedSessionWrites(runner-ignored, asserted end-to-end).id; completion without cart;cancel_checkoutpresent; status/totals strictly server-side). All eight policies; the composition layer consumes each stateful policy's owndecisionenvelope, so the documented contract is the thing under test. Latest results in the round-3 comment below.input.payload.name/input.kind/input.useror stripped claims.manifest.jsonregenerated;pnpm manifest:checkpasses.Honest caveats
continue_url, that session is invisible to the gateway. These are agent-autonomy guardrails on the MCP path, not end-to-end guarantees.on_drop), and both hooks bound (observe attool_post_invoke, enforce attool_pre_invoke).order-pii-egress-redactionrewrites the structured result; the JSON copy serialized incontent[].textis not rewritten — a stated limitation for text-first clients.input.context.mandate.*(caps/allowlists/budget/advertised price) is gateway-supplied policy input, not a UCP field — UCP carries the buyer's constraints only as an AP2 mandate that is unstructured to a checkout policy.attribution-disclosure's disclosure check is a compliance nudge, not enforced provenance — the value is agent-supplied.approved-cart-integritybinds to is the merchant'sready_for_completeresponse; whether that transition encodes a human approval is a property of the deployment's flow.Resolved during review
apps/ucp, industrycommerce, bundleagentic-commerce,high-value-approval-gate→high-value-handoff-gate.tests.yamlis the canonical test format (fixtures converted, JSON removed).opa fmtconvention tracked separately).