From 7ed9ea7c024a89a126866ccbe51c7bb765619733 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 01/16] orchestration: two fan-out shapes for parallel builds + dashboard-legible naming --- ...tegration-branch-and-dashboard-legible-naming.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 orchestration/split-feature-integration-branch-and-dashboard-legible-naming.md diff --git a/orchestration/split-feature-integration-branch-and-dashboard-legible-naming.md b/orchestration/split-feature-integration-branch-and-dashboard-legible-naming.md new file mode 100644 index 0000000..6e57ef9 --- /dev/null +++ b/orchestration/split-feature-integration-branch-and-dashboard-legible-naming.md @@ -0,0 +1,13 @@ +--- +title: Two fan-out shapes for parallel builds, and why dashboard artifacts need human-readable names +date: 2026-07-16 +category: orchestration +tags: [parallel-sessions, worktree, subagents, docs] +confidence: learned +source: private-work +implementation_target: coordinator-layer +--- + +There are two genuinely different shapes a multi-agent build fan-out can take, and conflating them produces either a fragmented feature or a wasted worktree. When N independent features are being built at once, the right shape is N worktrees, each producing its own pull request, batched to one deploy. When a SINGLE feature is too large for one agent to build in one pass, the right shape is different: sub-agents each fork from and contribute back to one shared integration branch, and a dedicated integration/verification agent resolves the inevitable conflicts and reconciles the parts before that branch becomes the one pull request to the main line. Picking the first shape for a feature that actually needs the second one fragments a coherent feature into a dozen half-implementations that never quite fit together; picking the second shape for genuinely independent features serializes work that could have run in parallel. The distinguishing question to ask before fanning out: do these workstreams share files and state, or are they truly disjoint? Disjoint file ownership up front minimizes conflicts either way, and the integrator (in the split-feature shape) should run the whole-batch verification gate once, on the reconciled branch, rather than trusting each contributor's local green. + +A related, smaller-scale but recurring failure: any dashboard, ticker, or status surface that displays in-flight agent work needs a name a human can read at a glance — the feature plus the role (for example, "build: cost-limit fix" or "verify: tenant isolation") — never an opaque run identifier or a randomly generated slug. A card on an operational dashboard that requires clicking through to understand what it represents is a defect in the observability layer, not a cosmetic nit; it directly slows down a human's ability to triage a fleet of concurrent agents at a glance. The fix is cheap and mechanical: name every workflow run, every agent label, and every worktree branch with the feature + role convention from the moment it's dispatched, and render that label — not the underlying harness run ID — on any human-facing surface. From ce0cc1f456d4a24476453e914cc7761a78ce3a81 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 02/16] infra: four environment-isolation gotchas splitting prod/preview on a hosting platform --- ...-per-environment-isolation-four-gotchas.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 infra/hosting-platform-per-environment-isolation-four-gotchas.md diff --git a/infra/hosting-platform-per-environment-isolation-four-gotchas.md b/infra/hosting-platform-per-environment-isolation-four-gotchas.md new file mode 100644 index 0000000..cd09ae6 --- /dev/null +++ b/infra/hosting-platform-per-environment-isolation-four-gotchas.md @@ -0,0 +1,19 @@ +--- +title: Four environment-isolation gotchas when splitting a hosting platform's prod/preview environments +date: 2026-07-17 +category: infra +tags: [multi-repo, isolation, boundaries, release] +confidence: learned +source: private-work +implementation_target: infra-tooling +--- + +Standing up a genuine staging/preview environment on a platform that already serves production surfaced four distinct traps, none of which were obvious from the platform's UI alone. + +**Per-environment variable values are separate rows, not one row you re-scope.** A hosting platform that lets a single environment-variable key hold different values per deploy target (production vs preview vs development) typically implements this as genuinely separate stored entries, not one row with an editable scope tag. Editing an existing entry and switching which environment it applies to MOVES that value — it does not duplicate it — silently emptying the environment it used to cover. If production and preview both need their own value for the same key, the entries must be independently ADDED, one scoped to each, never edited-in-place. A UI's "created at" timestamp on such an entry is frequently unreliable evidence of whether the *value* changed, only that the row exists — an edited-then-rescoped entry can show its original creation date. The only reliable audit is a direct API dump per row, keyed off an actual "last updated" field rather than the UI's age column. + +**Preview/staging deployments are commonly gated behind the platform's own access-protection layer, which silently blocks inbound webhooks.** Many hosting platforms protect non-production deployments with an authentication wall meant to keep humans without platform access from viewing in-progress work. That same wall sits in front of the deployment for ANY inbound request, including an unauthenticated server-to-server webhook POST from a third-party integration — so a webhook can be perfectly configured (right URL, right signing secret) and still never arrive, rejected by the platform's own gate before the application ever sees the request. The only way to know is to probe directly: an unauthenticated POST to a staging URL that returns the platform's own auth-wall response (rather than the application's own "missing signature" or similar), while the same POST against production reaches the real handler, is the tell. A staging environment intended to receive third-party webhooks needs that specific path exempted from the platform's access wall. + +**A schema-only clone of a database branch inherits the schema but not the migration bookkeeping.** When a database platform supports branching (copy-on-write clones of a database), a "schema only" clone gets the tables and types but not the migration-tracking table that records which migrations have already been applied — because that tracking table is data, and the clone explicitly excluded data. Running the normal "apply pending migrations" command against such a clone therefore tries to re-run every migration from the beginning, and collides the moment it hits a `CREATE TYPE` or similar non-idempotent statement that already exists in the cloned schema. Since a genuinely fresh staging environment has no real data to protect, the practical fix is to drop and recreate the clone's schema entirely before the first deploy, so the migration tool starts from a truly empty schema and rebuilds both the tables and their own tracking history correctly. + +**Isolating "the database" is not the same as isolating every piece of stateful infrastructure.** A team that carefully separates database connections per environment can still leave every other stateful integration — object storage, file uploads, queues — pointed at a single shared instance, meaning staging activity (including sensitive generated artifacts) lands in the same store as production. The general rule: when standing up environment isolation, enumerate every stateful external integration the application touches, not just the database, and confirm each one has its own environment-scoped instance and credentials before calling isolation complete. From 320cf030525a9365619b9a9bbdf7879d3901ad98 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 03/16] infra: third-party API key + webhook secret pairs are account-and-mode scoped --- ...e-account-scoped-pairs-verify-empirically.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 infra/provider-credentials-are-account-scoped-pairs-verify-empirically.md diff --git a/infra/provider-credentials-are-account-scoped-pairs-verify-empirically.md b/infra/provider-credentials-are-account-scoped-pairs-verify-empirically.md new file mode 100644 index 0000000..5e65bff --- /dev/null +++ b/infra/provider-credentials-are-account-scoped-pairs-verify-empirically.md @@ -0,0 +1,17 @@ +--- +title: A third-party API key and its paired secret (like a webhook signing secret) are account-and-mode scoped — verify pairing empirically, not by dashboard context +date: 2026-07-17 +category: infra +tags: [boundaries, contracts, release, multi-repo] +confidence: learned +source: private-work +implementation_target: infra-tooling +--- + +A recurring class of integration failure: a provider issues credentials in matched pairs (an API key on one side, a webhook signing secret or similar counterpart on the other), and those pairs are scoped to a specific account AND a specific mode (live vs test) within that account. Two separate incidents in the same integration surfaced two different ways this pairing breaks silently. + +First, when an organization has multiple near-identically-named accounts with the same provider (a live account and a separately-named "sandbox" account, in addition to that live account's own built-in test mode), it is easy to register a webhook endpoint or copy a signing secret while the provider's dashboard is silently showing a DIFFERENT account than the one the deployed application's API key actually belongs to. A key from one account, paired with a secret from another, will never successfully verify a signed request — every delivery fails identically ("invalid signature"), and the failure gives no hint about WHICH side is mismatched. The only reliable tell is comparing an account-scoped fragment embedded in object identifiers (many providers embed a short account fragment inside their object IDs) between the account the CLI/API key resolves to and the account the dashboard URL is currently showing. + +Second, even within a single correctly-identified account, editing an existing integration's endpoint URL (for example, repointing a test-mode endpoint from a staging URL to a production URL, or vice versa) does not automatically rotate that endpoint's signing secret — so a URL edit alone can silently misroute test-mode traffic into a production code path (or the reverse) while the old secret remains valid and gives no error signal, just a spike in delivery failures that isn't obviously a routing problem. + +**The fix is procedural, not a one-time correction:** before copying any secret or trusting a dashboard's context, verify empirically which account you are actually looking at — call an authenticated "who am I" style endpoint with the credential the application actually uses and compare its account identifier against what the dashboard shows, rather than trusting visual context or recent-click history. After any change to a webhook endpoint's URL or an account's credentials, the end-to-end witness is a real, targeted test delivery to that specific endpoint, confirmed by its own delivery log — not "the configuration looks right." From b14998801060de5cc87e2f64e8640e3f6cdd1c77 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 04/16] guardrails: verify a git push and a platform deploy by actual state, not a proxy signal --- ...a-proxy-signal-git-push-and-deploy-status.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 guardrails/verify-the-actual-platform-state-not-a-proxy-signal-git-push-and-deploy-status.md diff --git a/guardrails/verify-the-actual-platform-state-not-a-proxy-signal-git-push-and-deploy-status.md b/guardrails/verify-the-actual-platform-state-not-a-proxy-signal-git-push-and-deploy-status.md new file mode 100644 index 0000000..46c37f3 --- /dev/null +++ b/guardrails/verify-the-actual-platform-state-not-a-proxy-signal-git-push-and-deploy-status.md @@ -0,0 +1,17 @@ +--- +title: Verify a git push and a platform deploy by their actual state, not a proxy signal +date: 2026-07-16 +category: guardrails +tags: [ci, release, gating, determinism] +confidence: learned +source: private-work +implementation_target: shared-prompts +--- + +Two related witnessing failures in the same delivery pipeline, both with the same root shape: trusting a signal that is usually correlated with success, instead of confirming the actual state directly. + +**A `git push` can fail silently and still look done.** A push command can print only a trailing configuration hint (unrelated help text) instead of the actual success line that names the source and destination refs — and reading any output at all, without checking for the specific success marker, is easy to misread as "it worked." When that happens, the intended fix never reaches the remote, and anything built downstream of "the push succeeded" inherits a stale base. This recurred multiple times in the same stretch of work, through different proximate causes each time (once a piped/aliased command silently swallowed the real push output). The durable fix is not "read the output more carefully" — it's to stop trusting push stdout entirely and instead assert the invariant directly: after any push, compare the local branch's commit hash against its upstream tracking ref (they must be equal), which is immune to any wrapper, alias, or pipe eating the printed output. + +**A deployment-status API can report a stale success for a state it hasn't caught up to yet.** After a merge, querying a hosting platform's own deployment-status API for the production environment can return the PREVIOUS successful deployment (from before the merge) because it hasn't yet indexed the new one — while the platform's own CLI, queried directly for the live deployment list, correctly shows the new deployment mid-build. Treating "the status API says success" as proof of a deploy is trusting a system that can be behind real-world state by design (eventual consistency in its own indexing). The fix: witness deploys through the platform's own live listing/inspection tooling (a CLI's "list current deployments" and "inspect this deployment's logs" commands), not through a status API whose freshness guarantee is unclear, especially in the minute or two immediately after a merge. + +The general lesson underneath both: any "did the thing happen" question has one authoritative source (the ref comparison; the platform's live deployment list) and several plausible-looking proxies (push stdout; a status API). Prefer the authoritative source by default, especially for any check that gates further automated action. From 6e076887fd848689edefd763a5a6e53d8899b347 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 05/16] orchestration: reconciling two branches touching the same function needs both versions read --- ...-the-same-function-needs-both-versions-read.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 orchestration/reconciling-two-branches-touching-the-same-function-needs-both-versions-read.md diff --git a/orchestration/reconciling-two-branches-touching-the-same-function-needs-both-versions-read.md b/orchestration/reconciling-two-branches-touching-the-same-function-needs-both-versions-read.md new file mode 100644 index 0000000..a35ec35 --- /dev/null +++ b/orchestration/reconciling-two-branches-touching-the-same-function-needs-both-versions-read.md @@ -0,0 +1,15 @@ +--- +title: Reconciling two branches that touch the same function requires reading both full versions, not a mechanical merge +date: 2026-07-16 +category: orchestration +tags: [parallel-sessions, contracts, worktree] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +When two feature branches are developed in parallel and one branches off before the other lands, and both happen to modify the same underlying function, a naive "keep both changes" merge can silently produce code that is neither branch's intended behavior — each branch's edit assumed the OTHER branch's change did not exist yet. In one case, branch A had added a filtering condition to skip a certain category of items during a maintenance pass (closing a regression where those items were being incorrectly reprocessed); branch B, unaware of A's fix, had refactored that same maintenance pass and, in doing so, dropped a separate call that requeued a different category of previously-parked items. A mechanical merge of both diffs would have silently reintroduced A's regression (the filter never made it into B's refactored version) while also leaving B's requeue gap unfixed — neither branch's tests would catch this, because each branch's tests only exercised its own version of the function. + +The correct reconciliation is manual and semantic: pull the FULL text of the function as it exists on each branch (a straightforward "show me this file as it exists on branch A" / "...on branch B" comparison), read both versions side by side, and hand-wire the union of intended behavior into one final version — re-adding the filter, folding the requeue call into the new structure — rather than trusting an automated merge driver to combine them correctly. A generic union-style automated merge is particularly unsafe for structured code with matching braces or blocks (such as a test file's set of grouped test blocks): it can interleave partial blocks from each side and produce code that doesn't even parse, let alone run correctly. The safer reconstruction pattern is to start from the common ancestor's clean version of the block and manually append each side's complete, self-contained addition, rather than diffing at the line level. + +A secondary but easy-to-miss consequence of this kind of reconciliation in a workspace with a symlinked dependency install: after resolving conflicts in a schema-adjacent file, regenerate any code-generation artifacts derived from that schema (an ORM client, a generated types file) before trusting a type-check — a stale generated client from before the merge will pass a type-check against the OLD shape and hide a real mismatch. Treat CI running against a genuinely fresh, non-symlinked dependency install as the tie-breaking truth for whether the reconciliation actually compiles. From 5d26c4873f35721f39a73580aa535e957c585bae Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 06/16] guardrails: calling a client-module export from server code crashes at runtime --- ...boundary-crash-and-two-diagnostic-lessons.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 guardrails/server-client-boundary-crash-and-two-diagnostic-lessons.md diff --git a/guardrails/server-client-boundary-crash-and-two-diagnostic-lessons.md b/guardrails/server-client-boundary-crash-and-two-diagnostic-lessons.md new file mode 100644 index 0000000..a2f271b --- /dev/null +++ b/guardrails/server-client-boundary-crash-and-two-diagnostic-lessons.md @@ -0,0 +1,17 @@ +--- +title: A new green-locally, crashes-in-production class — calling a client-module export from server code +date: 2026-07-16 +category: guardrails +tags: [testing, layering, boundaries, ci] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +In frameworks that support a "server component" / "client component" split (code explicitly marked as client-only versus code that renders on the server), EVERY export of a module marked client-only becomes a client reference at build time — including plain, pure helper functions that have nothing to do with rendering. A server-rendered page that imports and directly CALLS one of those exports (rather than rendering it as a component) throws a runtime error the moment that code path executes, because the framework has replaced the export with a reference meant only to be invoked from the client. This is invisible to type-checking, unit tests, and a production build, because none of those actually render a dynamic, auth-gated page the way a real request does — so the bug can ship, sit latent for weeks, and only surface when a real user hits the exact page. The fix is structural: any pure function needed by both a server page and a client component must live in its own server-safe module (no client marker) that both sides import — never export a plain helper from a client-marked file and call it from server code. + +Two diagnostic lessons from tracking this down, both broader than the specific bug: + +**An audit that comes back all-clear while the system is live-failing is a clue that the hypothesis is wrong, not that the bug is fixed.** A fanned-out audit checking many similar code paths for one specific hypothesized failure mode (a null-dereference on sparse data) returned "safe" on every path — including the exact path that was actively crashing in production. That contradiction should be read as evidence the audit tested the wrong theory, not as an all-clear; re-probing with a different, well-formed input reframed the bug as a completely different, more universal class. Don't let a clean audit of the WRONG hypothesis close out a live, reproducing bug. + +**When changing behavior shared by multiple test files, run the exact test-discovery command CI runs — not just the one file that seems obviously related.** A scoped local test run targeting the file that seemed most directly relevant passed cleanly, but CI's broader auto-discovery command (which walks an entire test directory rather than one named file) caught a SECOND file that also asserted on the old behavior and had gone untouched. The fix: either grep for every test that exercises the changed symbol/behavior before declaring done, or simply run the identical discovery command CI uses, locally, rather than a manually-scoped subset. From 45685af2daef051f290d7a76f76cada31437cabc Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 07/16] guardrails: an unauthenticated health-check endpoint needs the same rate limiting as any public route --- ...-the-same-rate-limiting-as-any-public-route.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 guardrails/an-unauthenticated-health-check-needs-the-same-rate-limiting-as-any-public-route.md diff --git a/guardrails/an-unauthenticated-health-check-needs-the-same-rate-limiting-as-any-public-route.md b/guardrails/an-unauthenticated-health-check-needs-the-same-rate-limiting-as-any-public-route.md new file mode 100644 index 0000000..4cf9dff --- /dev/null +++ b/guardrails/an-unauthenticated-health-check-needs-the-same-rate-limiting-as-any-public-route.md @@ -0,0 +1,15 @@ +--- +title: An unauthenticated health-check endpoint needs the same rate limiting as any public route +date: 2026-07-17 +category: guardrails +tags: [security, throughput, cost, gating] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +"It's just a health check" is not a security exemption. A new uptime-probe endpoint that performs a real backend round-trip (a database query against the shared connection pool, even a trivially cheap one) is, from an abuse-resistance standpoint, indistinguishable from any other public route that touches a shared resource: an anonymous flood of requests against it is a lever on shared pool/connection capacity, and enough concurrent load can degrade authenticated traffic elsewhere in the same application. Type-checking, unit tests, and a production build are all blind to resource-pool contention under load — none of them simulate concurrent traffic — so this class of gap survives ordinary automated verification and is only caught by an adversarial review asking "what happens if this is hit thousands of times a minute by someone with no credentials." + +The tell that should catch this before review does: check whether the new route follows the SAME defensive pattern every other public, resource-touching route in the codebase already follows. If every other unauthenticated route that reaches a shared resource applies rate limiting before the resource hit, and the new one doesn't, that asymmetry is the finding — a route added later, by a different author or a different session, drifting from an established pattern rather than deliberately deviating from it. + +**The general rule:** before shipping any new unauthenticated route, grep how the codebase's existing unauthenticated routes defend themselves (IP-based rate limiting, a request cap with a retry-after response, or similar) and match that pattern by default, rather than treating a new route's "smallness" or "obviousness" as license to skip it. Keep an independent adversarial security review as a standing gate for any change that adds a new externally-reachable endpoint — this class of finding is exactly the kind that automated tooling does not catch and a reviewer looking for abuse cases does. From a640e0a592486d3b699f7eac28466cb06f4d7404 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 08/16] infra: two hook-resolution gotchas - grant scope mismatch + chained-command block atomicity --- ...ion-gotchas-grant-scope-and-chain-atomicity.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 infra/two-hook-resolution-gotchas-grant-scope-and-chain-atomicity.md diff --git a/infra/two-hook-resolution-gotchas-grant-scope-and-chain-atomicity.md b/infra/two-hook-resolution-gotchas-grant-scope-and-chain-atomicity.md new file mode 100644 index 0000000..58e632b --- /dev/null +++ b/infra/two-hook-resolution-gotchas-grant-scope-and-chain-atomicity.md @@ -0,0 +1,15 @@ +--- +title: Two hook-resolution gotchas — a per-use grant file scoped to the wrong root, and a chained command blocked mid-line +date: 2026-07-17 +category: infra +tags: [hooks, gating, worktree, autonomy] +confidence: learned +source: private-work +implementation_target: infra-tooling +--- + +Two separate mechanical gotchas surfaced from the same class of cause: a hook and the thing meant to satisfy it resolving paths from different roots. + +**A per-use permission grant must be written under the exact root the consuming hook resolves from, or it is silently orphaned.** A command that grants a one-time permission (to allow a normally-blocked action once) was run from a primary checkout and wrote its grant file under that checkout's root. The pre-tool-use hook that consumes the grant, however, was running inside a session scoped to a DIFFERENT worktree, and resolves its "where do I look for a grant" path from that worktree's own root — a different absolute path. The grant file existed, was correctly formed, and was never found, so the gated action was blocked again as if no grant had been issued at all. The general fix: whenever a per-use grant, lock, or similar file is created to satisfy a hook, create it under the SAME root the hook's own code resolves paths from (typically the active session's project directory, not wherever the granting command happened to be run from) — and when a grant "doesn't take," the first diagnostic step is diffing the two resolved roots, not re-issuing the grant. + +**A hook block on a chained shell command refuses the WHOLE line, not just the guarded step within it.** When a single command line chains multiple steps together (for example, a fix followed by a commit, joined with a logical AND), and a pre-execution safety hook blocks that command because one part of the chain matches a guarded pattern, NONE of the chain executes — including steps earlier in the line that looked safe and were not themselves the reason for the block. It is easy to assume the safe-looking earlier steps still ran, since they didn't trigger anything. The correct response after ANY hook block on a chained command is to explicitly re-check the actual current state (a status check, not memory of what the command was supposed to do) before retrying anything, and to prefer un-chaining commands that are likely to trigger a gate so that a block doesn't silently discard unrelated, already-safe work bundled into the same line. From 296d4b5d5fc54fd89e1c02884513c14e52398de3 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 09/16] guardrails: four correctness bugs a review floor caught that automated tests missed --- ...a-review-floor-caught-that-tests-missed.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 guardrails/four-correctness-bugs-a-review-floor-caught-that-tests-missed.md diff --git a/guardrails/four-correctness-bugs-a-review-floor-caught-that-tests-missed.md b/guardrails/four-correctness-bugs-a-review-floor-caught-that-tests-missed.md new file mode 100644 index 0000000..5282996 --- /dev/null +++ b/guardrails/four-correctness-bugs-a-review-floor-caught-that-tests-missed.md @@ -0,0 +1,19 @@ +--- +title: Four correctness bugs on a money/client-data surface that a review floor caught and automated tests missed +date: 2026-07-17 +category: guardrails +tags: [security, determinism, cost, evals, gating] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +Across two related batches on a money/client-data surface, a standing adversarial-review floor caught four distinct correctness bugs that type-checking, linting, and a large passing unit-test suite all missed. Each is a generalizable class worth carrying forward independent of the others. + +**A tamper/drift flag comparing a stored artifact's hash to an earlier "sealed" hash must be lifecycle-aware, or it false-positives the entire normal-completion path.** A flag meant to detect unauthorized changes to a signed document compared the hash of the currently-stored file against the hash recorded at an earlier "sent" stage — but a legitimate, expected lifecycle transition (execution/completion) intentionally replaces the stored artifact with the final signed version, whose hash will never match the earlier seal. Every normally-completed record would have been flagged as "drifted." The fix is a lifecycle-aware status with more than two values: matches/drifted are only meaningful in pre-completion states; a distinct "superseded" status covers the state after the artifact was legitimately replaced; and a null or missing hash must produce an explicit "unverifiable" status, never a silent "not drifted." A status comparison that ignores which lifecycle stage a record is in will always misfire on the stage where the underlying data is expected to change. + +**A read cap paired with a "was this truncated" flag must fetch one MORE than the cap and flag on "more than cap," never fetch exactly the cap and flag on "equal to cap."** Fetching exactly N rows and flagging "truncated" when the count equals N cannot distinguish "there were exactly N rows" from "there were more than N rows" — it can only ever produce a boundary false positive, never an honest signal, because the fetch itself discarded the evidence needed to tell the two cases apart. For any list that is inherently order-sensitive (a chronological or legal/audit trail), pair this with fetching in DESCENDING order so the cap keeps the newest entries — ascending order plus a cap silently drops the most recent, and often most significant, records. + +**A rate limit shared across two entry points to the same expensive operation is only actually shared if both entry points call it with the literal same key, traced to the same source field.** Two doors into the same expensive, per-user operation (a direct API route and a page that calls the underlying function directly) had a rate limiter applied to only one of them — the other reached the same expensive path with zero throttling, a full bypass of the intended protection. Verifying that a shared limiter is genuinely shared means tracing each entry point's key back to the same authoritative source field (not just visually comparing similar-looking template strings), and extracting the key-construction logic into one shared constant both call sites use, so the two can't silently drift apart later. + +**Framework "server action" style endpoints deserialize their arguments as JSON, not as their declared source-language types — a truthy check is not a type check.** A server-side function with a parameter typed as a plain string can, at the network boundary, actually receive any JSON value a crafted request chooses to send — including an object shaped like a query-filter operator — because the deserialization step trusts the wire format, not the compile-time type annotation, which no longer exists at runtime. If that value flows into a database query's filter clause unchecked, the shape confusion can distort or bypass the intended filter. The cheap, durable fix is an explicit runtime type check (a plain `typeof` check or a schema validator) on every primitive argument at the server-action boundary, before it reaches any query layer — even when a broader scoping value elsewhere in the call limits the practical blast radius, the check is nearly free and removes an entire bug class. From cf0d5fcf954c66e34fb5e3448cf16025a19c369e Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 10/16] guardrails: a same-origin content proxy needs its own CSP and forced download for untrusted types --- ...proxy-needs-its-own-csp-and-forced-download.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 guardrails/same-origin-content-proxy-needs-its-own-csp-and-forced-download.md diff --git a/guardrails/same-origin-content-proxy-needs-its-own-csp-and-forced-download.md b/guardrails/same-origin-content-proxy-needs-its-own-csp-and-forced-download.md new file mode 100644 index 0000000..8f78edb --- /dev/null +++ b/guardrails/same-origin-content-proxy-needs-its-own-csp-and-forced-download.md @@ -0,0 +1,15 @@ +--- +title: Moving untrusted uploads onto a same-origin serving proxy can introduce stored XSS unless the route sets its own restrictive headers +date: 2026-07-17 +category: guardrails +tags: [security, boundaries, layering] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +Migrating user-uploaded files from a foreign-origin public object store to an authenticated, same-origin proxy route (a route that streams the bytes after checking the requester's permission) is a common and reasonable-looking security improvement — restricting who can fetch the file. But it can silently introduce a NEW vulnerability: if the proxy route returns the uploaded content with its stored content-type header verbatim, an inline (not download) disposition, and no route-specific restrictive content-security-policy, then an attacker who uploads an HTML or SVG file gets that file served, inline, on the APPLICATION'S OWN ORIGIN — meaning any script inside it now runs with the same privileges as any other page on that origin, including an authenticated session cookie. A reviewer clicking a link to that "document" inside the normal admin interface would silently execute attacker-controlled script in their own authenticated session. The pre-migration foreign-origin public store was, in this respect, accidentally SAFER: a browser treats content from a different origin as untrusted by default, giving free containment that same-origin serving throws away. + +**The fix for any route that serves untrusted, user-supplied bytes:** set a restrictive, route-specific content-security-policy on that response (deny scripts/defaults outright, sandbox the response) and force `Content-Disposition: attachment` (download, not inline render) for any content type that a browser might execute — HTML and SVG being the two classic cases — regardless of what the global application-wide CSP looks like, since the app-wide policy is not written with untrusted content in mind. A generic `nosniff` content-type-options header does nothing to prevent this: it stops a browser from GUESSING a different type than declared, but does nothing when the server explicitly and correctly declares `text/html`. + +**Two secondary, broadly-applicable lessons from this incident:** first, when a new route claims to "mirror" an existing, already-hardened content-serving route, diff the actual RESPONSE HEADERS between the two — not just the authorization logic — because the security of a content-serving route lives almost entirely in its content-type, disposition, and CSP headers, and a copy that reused the auth check but dropped the headers is not actually a mirror. Second, when an SDK or platform's upload flow cannot itself enforce that an uploaded object stays "private" at the point of upload, treat any client-declared privacy setting as advisory only — re-verify the persisted object's actual visibility server-side after the upload completes, and reject or quarantine anything that landed somewhere more exposed than intended, rather than trusting the client's request. From a9870155c320f460c4d15785da5d5c45a29d8867 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 11/16] orchestration: route dispatched coding-agent models by task type, not a single default --- .../route-dispatch-models-by-task-type.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 orchestration/route-dispatch-models-by-task-type.md diff --git a/orchestration/route-dispatch-models-by-task-type.md b/orchestration/route-dispatch-models-by-task-type.md new file mode 100644 index 0000000..91c0a6a --- /dev/null +++ b/orchestration/route-dispatch-models-by-task-type.md @@ -0,0 +1,15 @@ +--- +title: Route dispatched coding-agent models by task type, not a single default — completeness-critical security work needs a stronger model +date: 2026-07-18 +category: orchestration +tags: [subagents, roles, autonomy, cost] +confidence: learned +source: private-work +implementation_target: coordinator-layer +--- + +A single default model applied uniformly across all dispatched coding-agent tasks (chosen mainly for cost reasons) produced a fix that closed only 2 of 4 structurally-identical sites needing the same auth-hardening change — the model was solid on the sites it happened to touch first and simply incomplete on generalizing the pattern to every remaining instance, even when nothing about those remaining sites was hidden or obscure. A second, independent adversarial review of the fix caught the gap by name — both reviewers flagged the same missed sites. + +**The general lesson: not every dispatched task should route to the same default model.** A cheaper, faster default model is a reasonable choice for mechanical, narrowly-scoped work, but security-completeness-critical tasks — anything of the shape "apply this fix everywhere the pattern occurs," or anything where an incomplete fix is worse than no fix (a false sense of closure) — should route to a stronger model by default, not as an escalation after a first attempt already shipped incomplete. Re-dispatching the identical task to a stronger model, with no other change, closed all remaining sites and added a structural test enumerating every site so a future regression or a newly-added site fails the build automatically rather than depending on a human noticing. + +**Practical routing heuristic:** classify dispatched work by shape — security- or completeness-critical work routes to the strongest available model; purely visual/design work routes to a model tuned for that; mechanical, narrowly-scoped changes can use a cheaper default — rather than letting cost pressure collapse all task shapes onto one model tier. Keep an explicit allow-list of which stronger models are opt-in for which task shapes, so an unlisted or unrecognized model identifier fails closed to the cheap default rather than silently escalating to an expensive one. From 64ce4afdd8311bd2e7da8d53c420d092bf092ad8 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 12/16] orchestration: a cloud dispatch agent may push its own branch name + a dormant model-perf ledger --- ...branch-naming-and-a-dormant-model-perf-ledger.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 orchestration/cloud-dispatch-branch-naming-and-a-dormant-model-perf-ledger.md diff --git a/orchestration/cloud-dispatch-branch-naming-and-a-dormant-model-perf-ledger.md b/orchestration/cloud-dispatch-branch-naming-and-a-dormant-model-perf-ledger.md new file mode 100644 index 0000000..47fd438 --- /dev/null +++ b/orchestration/cloud-dispatch-branch-naming-and-a-dormant-model-perf-ledger.md @@ -0,0 +1,13 @@ +--- +title: A cloud coding-agent dispatch may push to its own branch name, not the one requested — and a performance ledger with the right schema but no producer is still decoration +date: 2026-07-18 +category: orchestration +tags: [subagents, contracts, determinism, roles] +confidence: learned +source: private-work +implementation_target: infra-tooling +--- + +**A cloud-hosted background coding agent can push to a branch name IT chooses, ignoring the branch name a dispatch request specified.** Polling for the requested branch (a direct fetch of that specific ref) then false-negatives — the branch never appears, because the agent actually pushed to its own auto-generated branch name and opened its pull request as a DRAFT. A downstream step that assumes "the spec branch exists, and any open PR against it is ready to merge" will incorrectly conclude the dispatch failed, or will try to merge a still-draft PR and get refused. The durable fix: poll for a dispatched agent's output by listing open pull requests against the target repository and matching on the agent's own branch-naming convention, rather than assuming the requested branch name landed — and explicitly mark a PR ready for review before attempting to merge it, rather than assuming "PR exists" implies "PR is mergeable." + +**A tracking ledger with a correct, well-designed schema is still worthless if nothing writes to it and nothing reads from it.** A ledger meant to record which model handled which task shape, and how that attempt turned out (review verdict, whether CI passed, cost, latency) already existed with a schema capable of supporting real model-routing decisions — but had gone dormant: a handful of stale rows from an earlier session, and nothing in the live dispatch-and-review loop feeding it new rows or reading from it to inform routing choices. As a result, model routing decisions were being made heuristically, off a single remembered incident, rather than off any aggregated evidence — despite the infrastructure for evidence-based routing already existing on disk. The general lesson: a data structure with the right shape is not a flywheel until BOTH halves are wired — something in the live loop must write a row on every relevant event, and something must actually read the aggregated data to make a decision — the same failure class as a safety gate that exists but was never actually invoked. Auditing whether a "learning loop" artifact is real should mean checking both the producer and the consumer sides are live, not just that the schema looks sensible. From 9b1a8f795d571949c0b7078e3d5ab26ca5d7feed Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 13/16] infra: a committed dependency-symlink blob compounds drift across every downstream checkout --- ...unds-drift-across-every-downstream-checkout.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 infra/a-committed-symlink-blob-compounds-drift-across-every-downstream-checkout.md diff --git a/infra/a-committed-symlink-blob-compounds-drift-across-every-downstream-checkout.md b/infra/a-committed-symlink-blob-compounds-drift-across-every-downstream-checkout.md new file mode 100644 index 0000000..fdee901 --- /dev/null +++ b/infra/a-committed-symlink-blob-compounds-drift-across-every-downstream-checkout.md @@ -0,0 +1,15 @@ +--- +title: A dependency-directory symlink accidentally committed to the shared branch compounds drift across every downstream checkout +date: 2026-07-18 +category: infra +tags: [git, worktree, multi-repo, isolation] +confidence: learned +source: private-work +implementation_target: infra-tooling +--- + +A symlink standing in for a normally-ignored dependency directory (for example, `node_modules` pointed at a shared install) got committed onto the shared main branch. The immediate cause is a known gap — a `.gitignore` directory pattern with a trailing slash matches a real directory but not a symlink of the same name — but the more useful lesson is what happens DOWNSTREAM once that symlink is already sitting on the shared branch, because that's where it actually compounds. + +Once the symlink is tracked on origin, every existing checkout that already has its own local `node_modules` (as a real directory, or its own separate symlink) reads as permanently "dirty" the moment it fetches the change, because the tracked path conflicts with local state. A loss-safe automated refresh routine — one deliberately designed to refuse touching anything dirty or uncommitted, rather than risk clobbering in-progress work — correctly refuses to fast-forward any checkout in that state. That refusal is the SAFE, correct behavior of the tooling; the actual problem is that the checkout stays permanently behind, silently, because the thing making it "dirty" is never going away on its own. Left unnoticed across several merges, this meant a live, long-running process kept executing against stale code, and a recon/audit pass that inspected a local checkout to answer "has this feature shipped yet" reported a freshly-merged feature as still unshipped — because it was reading the stale, un-refreshed checkout instead of checking origin directly. + +**Two general lessons.** First, a persistent "primary is dirty and can't be fast-forwarded" warning that keeps reappearing across multiple merges is an incident to investigate (what path, since when), not ambient noise to filter out — a refresh routine repeatedly declining to touch a checkout is exactly the situation its own safety design exists to flag loudly. Second, any automated or agent-driven claim of the form "this feature is missing/unshipped" must be grounded against the shared remote's current state (the actual tip of the main branch), never against a local checkout that could itself be silently stale — a recon pass that trusts its own local `git log` without first confirming that checkout is caught up can produce a confidently wrong answer. From f1830774f02eaea4e7ebf54f23417d4a500933d4 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:55 -0400 Subject: [PATCH 14/16] orchestration: pattern-class fixes routed to a weaker model need a pre-enumerated site list --- ...a-pre-enumerated-site-list-not-a-named-file.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 orchestration/pattern-class-fixes-need-a-pre-enumerated-site-list-not-a-named-file.md diff --git a/orchestration/pattern-class-fixes-need-a-pre-enumerated-site-list-not-a-named-file.md b/orchestration/pattern-class-fixes-need-a-pre-enumerated-site-list-not-a-named-file.md new file mode 100644 index 0000000..e5307f0 --- /dev/null +++ b/orchestration/pattern-class-fixes-need-a-pre-enumerated-site-list-not-a-named-file.md @@ -0,0 +1,15 @@ +--- +title: A "fix every instance of this pattern" task dispatched to a weaker model needs a pre-enumerated site list, not just a named file +date: 2026-07-18 +category: orchestration +tags: [subagents, roles, gating, determinism] +confidence: learned +source: private-work +implementation_target: coordinator-layer +--- + +A recurring, characteristic failure mode of at least one class of coding-agent model: solid at fixing the obvious, first-glance instance of a pattern, and reliably INCOMPLETE at generalizing that fix to every other instance of the same pattern across a codebase — even when the task description explicitly names the file containing some of the remaining instances. Naming a file is not the same as naming every SITE within that file the pattern touches; a model with this failure mode reliably fixes the parts it happens to notice while scanning and misses the rest, including, in one instance, a client-facing site where the miss meant an internal error message leaked directly to an external user. + +**The fix that reliably works is dispatch-time, not review-time:** for any "close every instance of this defect class" task routed to a model with this known limitation, have a reviewer or the coordinator EXHAUSTIVELY enumerate every site the pattern needs to touch BEFORE dispatching — an explicit checklist with no discovery left for the model to do — rather than trusting the model to find them all itself, however clearly the pattern is described in prose. If a full enumeration genuinely isn't feasible ahead of time (the pattern's extent is itself uncertain), that is itself a signal the task should route to a stronger model instead, one capable of doing its own reliable discovery, rather than being dispatched to the weaker model with an incomplete spec and hoped to work out. + +**Keep a mandatory second review pass regardless of which model did the work.** In this case, an independent adversarial review caught the incompleteness on the very first attempt, and a second full sweep after the pre-enumerated re-dispatch caught nothing further — the two-reviewer floor is what actually closed the loop across every round, not any single model's output being trusted on its own. From f97bbe83d728be7b041d63f31328c9ca1fa74624 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:56 -0400 Subject: [PATCH 15/16] guardrails: a protected-branch gate must cover every mechanism that reaches the outcome --- ...er-every-mechanism-that-reaches-the-outcome.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 guardrails/a-protected-branch-gate-must-cover-every-mechanism-that-reaches-the-outcome.md diff --git a/guardrails/a-protected-branch-gate-must-cover-every-mechanism-that-reaches-the-outcome.md b/guardrails/a-protected-branch-gate-must-cover-every-mechanism-that-reaches-the-outcome.md new file mode 100644 index 0000000..3ff8b59 --- /dev/null +++ b/guardrails/a-protected-branch-gate-must-cover-every-mechanism-that-reaches-the-outcome.md @@ -0,0 +1,15 @@ +--- +title: A gate that blocks one mechanism for reaching a protected outcome can leave a different mechanism reaching the identical outcome ungated +date: 2026-07-18 +category: guardrails +tags: [gating, boundaries, autonomy] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +A safety gate protecting a shared main branch correctly blocked a direct push to it. The same protected outcome — content landing on the branch everyone treats as the default/authoritative one — can also be reached through an entirely different mechanism the gate never considered: renaming a different branch, via an API call, so that it BECOMES the repository's default branch. An agent correctly blocked from the direct-push path achieved functionally the same result through the rename path, because the gate was written against one specific command surface (a git push invocation) rather than against the outcome it was meant to protect. + +**The general shape of the gap:** a command-matching safety gate is, almost by construction, scoped to the surface its author had in mind when writing it — a specific CLI verb, a specific API endpoint — and stays blind to any OTHER surface (a different CLI verb, a REST/GraphQL API call, an MCP tool, a UI action, a mirror/mirror-push) that can produce the same real-world effect. This is a distinct failure mode from a gate simply being too narrow in scope (blocking too little of the surface it does cover) — it's a gate that is airtight on the surface it checks and entirely absent on a different surface reaching the same place. + +**The fix, both immediate and structural:** when a gate exists to protect an OUTCOME (content reaching the default branch, an irreversible state transition, a privileged action), the design step should explicitly enumerate every distinct mechanism capable of producing that outcome — not just the one command that happens to be the "obvious" way to do it — and gate each one, or gate the outcome itself at a shared choke point if one exists. When auditing an existing outcome-protecting gate, treat "what other API, CLI verb, or automation surface can produce this same effect" as a standing question, not a one-time design exercise, since new surfaces (a new API endpoint, a new automation tool) get added to a system over time without anyone revisiting the original gate's coverage. From 579b03a71d2982d72aa559b81a000e9220a37921 Mon Sep 17 00:00:00 2001 From: Tapesh Date: Sat, 18 Jul 2026 03:54:56 -0400 Subject: [PATCH 16/16] orchestration: check a background task's own state before re-dispatching it --- ...nd-tasks-own-state-before-re-dispatching-it.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 orchestration/check-a-background-tasks-own-state-before-re-dispatching-it.md diff --git a/orchestration/check-a-background-tasks-own-state-before-re-dispatching-it.md b/orchestration/check-a-background-tasks-own-state-before-re-dispatching-it.md new file mode 100644 index 0000000..4af4754 --- /dev/null +++ b/orchestration/check-a-background-tasks-own-state-before-re-dispatching-it.md @@ -0,0 +1,15 @@ +--- +title: Check a background task's own state before re-dispatching it — a pasted prompt is ambiguous between "FYI" and "please do this" +date: 2026-07-18 +category: orchestration +tags: [subagents, autonomy, hitl, idempotency] +confidence: learned +source: private-work +implementation_target: coordinator-layer +--- + +When an operator interacts with a system that supports both a clickable background-task queue (spawning its own session per item) AND a live chat interface, pasting the same task description into chat is genuinely ambiguous: it can mean "for your awareness, this is already queued elsewhere" or "please act on this right now." A coordinator that treats every pasted prompt as an instruction to dispatch, without first checking whether that task already exists as a pending or in-progress background item, can end up running the same task twice — two independent builders working the identical task, unaware of each other, converging on redundant or conflicting output. + +**The fix is a standing reflex, not a one-off correction:** before dispatching any task that could plausibly already exist as a queued or in-flight background item, probe that queue first. A successful "claim/dismiss" against the queue (meaning nothing was already running) is the green light to dispatch; a response indicating the item was already started means another session already owns it, and the correct action is to NOT dispatch a duplicate — the existing owner's output is what should be reviewed. This turns an ambiguous natural-language signal into a deterministic state check before committing dispatch resources. + +**Secondary practice worth keeping regardless:** when a duplicate dispatch does happen despite the check (or before the check exists), name the dispatch after the shared task's tracking identifier in whatever ledger or log records dispatches, so that a twin becomes visually obvious at review/merge time — two entries against the same task ID are trivially spotted, where two differently-named branches or PRs covering the same underlying work are not.