diff --git a/evals/a-test-environment-that-can-only-reach-a-primitives-fallback-branch-is-hollow-green.md b/evals/a-test-environment-that-can-only-reach-a-primitives-fallback-branch-is-hollow-green.md new file mode 100644 index 0000000..6992b3b --- /dev/null +++ b/evals/a-test-environment-that-can-only-reach-a-primitives-fallback-branch-is-hollow-green.md @@ -0,0 +1,13 @@ +--- +title: A test environment that can only ever reach a primitive's fallback branch produces hollow-green coverage +date: 2026-07-16 +category: evals +tags: [evals, hollow-green, reversion-litmus, serverless] +confidence: learned +source: private-work +implementation_target: shared-prompts +--- + +A background-work primitive that only functions inside a live request always throws when invoked from a unit-test or CI environment, because that environment never has an active request scope. Every test that exercised the code path routed through that primitive's fallback branch instead of its real registration path — not by test design, but because the environment made the real branch structurally unreachable. A downstream helper the fallback branch calls happened to perform its side effect synchronously, before any await, so the test's mocked assertion passed whether the work was wrapped in the primitive or simply invoked bare. Reverting the fix that added the primitive would leave every test green. + +The generalizable litmus, applicable beyond this one primitive: before trusting a test's coverage of a fix, ask "would this suite still pass if the fix were fully reverted?" If the answer is yes, the suite is not exercising the changed code path — it may be structurally unable to, because the test environment can only ever reach one branch of a runtime-dependent primitive. The fix in that situation is not a bigger unit suite; it's a targeted mock or spy on the primitive itself that asserts it was actually invoked with the expected work, so the test can fail when the wrapping is removed even though the environment can never drive the real branch end-to-end. diff --git a/guardrails/a-client-side-timeout-abandons-a-query-without-cancelling-it.md b/guardrails/a-client-side-timeout-abandons-a-query-without-cancelling-it.md new file mode 100644 index 0000000..742a93a --- /dev/null +++ b/guardrails/a-client-side-timeout-abandons-a-query-without-cancelling-it.md @@ -0,0 +1,13 @@ +--- +title: A client-side timeout abandons a slow query without cancelling it +date: 2026-07-16 +category: guardrails +tags: [deadline, database, concurrency, resource-leak] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +A common deadline pattern races a database call against a timer and rejects if the timer wins. That pattern bounds how long the caller waits, but it does not touch the underlying query: the database keeps executing it and the connection stays checked out of the pool for as long as the query actually takes. Under sustained slowness, every "timed-out" request still leaks a live connection, which shrinks the pool further and makes the next request more likely to time out too — the deadline meant to contain the problem quietly deepens it instead. + +The generalizable lesson: a race-based timeout is an abandon, not a cancel. It changes what the calling code observes without changing what the database is doing. A deadline that must actually bound resource usage — not just caller-perceived latency — needs a mechanism the database itself will honor, such as a statement-level timeout scoped to the query or transaction, or a client driver call that genuinely cancels the in-flight query. When reviewing a newly added deadline, check which side of the boundary it operates on: does it stop the work, or does it just stop waiting for the work? diff --git a/guardrails/a-deadline-guarding-one-hop-of-a-fail-closed-path-leaves-sibling-hops-exposed.md b/guardrails/a-deadline-guarding-one-hop-of-a-fail-closed-path-leaves-sibling-hops-exposed.md new file mode 100644 index 0000000..c91bb7d --- /dev/null +++ b/guardrails/a-deadline-guarding-one-hop-of-a-fail-closed-path-leaves-sibling-hops-exposed.md @@ -0,0 +1,13 @@ +--- +title: A deadline guarding one hop of a fail-closed path leaves sibling hops exposed +date: 2026-07-16 +category: guardrails +tags: [deadline, fail-closed, concurrency, cost] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +A fix added a short deadline to the read step of a fail-closed spend-limiting path — the step that checks current usage before allowing an expensive downstream call. The same change also introduced several write steps further down the same path: a conditional update that atomically claims the spend, a batch insert, and an event write, all awaited with no deadline of their own. Under the exact concurrent-burst scenario the atomic claim exists to handle, one of those write steps can queue behind a database row lock indefinitely — so the request hangs and eventually times out at the outermost layer instead of failing closed quickly. The step that got the deadline was, ironically, the one least likely to contend; the step actually vulnerable to contention was left bare. + +The general rule: a deadline added to make a path fail closed only holds if every I/O call in that path — including calls inside error-handling branches — carries its own bound. Adding a timeout to the first hop a reviewer happens to look at, without auditing the full call graph the request traverses, produces a guarantee that looks complete but silently excludes exactly the hop most likely to be slow under the load the fix was meant to survive. Before declaring a path fail-closed, enumerate every awaited call between the entry point and the response, not just the one that prompted the fix. diff --git a/guardrails/a-partial-rollout-of-a-fix-commonly-skips-the-highest-traffic-caller.md b/guardrails/a-partial-rollout-of-a-fix-commonly-skips-the-highest-traffic-caller.md new file mode 100644 index 0000000..7892940 --- /dev/null +++ b/guardrails/a-partial-rollout-of-a-fix-commonly-skips-the-highest-traffic-caller.md @@ -0,0 +1,13 @@ +--- +title: A partial rollout of a fix commonly leaves the highest-traffic caller on the old behavior +date: 2026-07-16 +category: guardrails +tags: [fan-out-completeness, partial-rollout, enumerate-callers] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +A more robust rate-limiting mechanism was built to replace a per-instance limiter that didn't hold up across multiple running instances, and it was wired into the two call sites where the original bug had been reproduced. Those two call sites were not the busiest ones sharing the same underlying cost-limited resource — a higher-traffic set of call sites hitting the identical resource were left calling the old, per-instance-only limiter, because they weren't part of the original repro and so weren't in scope when the fix was rolled out. + +The generalizable pattern: when closing a finding that affects a shared resource or a shared code path, the repro that surfaced the bug is rarely the only caller that shares the vulnerable pattern, and it is very often not the highest-traffic one — the highest-traffic caller is disproportionately likely to be a different code path that happens to share the same underlying dependency. Fixing only the call site in the repro gives a false sense of closure. The rule: before considering a fan-out finding closed, grep every caller of the pattern being fixed (not just the one in the bug report), migrate all of them, and if any are deliberately left out, document that scope explicitly rather than leaving it implicit. diff --git a/guardrails/a-serverless-keep-alive-primitive-throws-for-two-reasons-swallowing-both-hides-a-regression.md b/guardrails/a-serverless-keep-alive-primitive-throws-for-two-reasons-swallowing-both-hides-a-regression.md new file mode 100644 index 0000000..7b3a3b2 --- /dev/null +++ b/guardrails/a-serverless-keep-alive-primitive-throws-for-two-reasons-swallowing-both-hides-a-regression.md @@ -0,0 +1,13 @@ +--- +title: A serverless keep-alive primitive throws for two different reasons — swallowing both hides a real regression +date: 2026-07-16 +category: guardrails +tags: [serverless, silent-regression, telemetry, durability] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +Serverless platforms offer a primitive for scheduling background work that should keep running after the visible response is sent. That primitive throws in two structurally different situations: a benign one (it is invoked outside of any active request — for example from a test runner or a script, where there is no response to extend) and a genuine one (it is invoked inside a real request, but the hosting platform or a self-hosted adapter never wired up the mechanism that lets work outlive the response). A catch block that swallows both cases identically — logging nothing, or silently falling back to a fire-and-forget call — makes a platform or deployment misconfiguration invisible. The background work silently stops being durable, with zero signal in any monitoring system, which is exactly the failure mode the primitive was adopted to prevent. + +The rule: when adopting a "run after the response" primitive, distinguish its two failure classes rather than treating any throw as routine. The benign, no-request-scope case can usually be identified by a distinct error marker the platform attaches to it; anything else that throws from the same call site should be surfaced loudly (an error log, a metric, an alert) so a real platform regression is visible where it happens, not discovered later as an unexplained gap in the data the background work was supposed to produce. diff --git a/guardrails/recurring-per-surface-ui-drift-is-a-missing-shared-primitive-fix-it-once-with-a-guard-test.md b/guardrails/recurring-per-surface-ui-drift-is-a-missing-shared-primitive-fix-it-once-with-a-guard-test.md new file mode 100644 index 0000000..481c84e --- /dev/null +++ b/guardrails/recurring-per-surface-ui-drift-is-a-missing-shared-primitive-fix-it-once-with-a-guard-test.md @@ -0,0 +1,13 @@ +--- +title: Recurring per-surface UI drift is a missing shared primitive — fix it once and add a guard test +date: 2026-07-16 +category: guardrails +tags: [design-system, shared-primitive, recurrence-gate, consistency] +confidence: learned +source: private-work +implementation_target: agent-guardrails +--- + +A single small UI element — a link inviting a user to report a problem — had drifted into several visually different implementations across separate surfaces of the same product: different colors, different presence or absence of an icon, different styling conventions, each one hand-rolled independently by whichever surface needed it. This is the same underlying pattern already known from design-review triage — a repeated finding across many surfaces signals a missing shared primitive one layer down — but observed here from the implementation side: the fix was one shared component, applied everywhere the element was needed, replacing every hand-rolled variant. + +The addition that made the fix durable rather than a one-time cleanup was a small automated consistency check wired into the build: a test that fails if any surface reintroduces its own version of the element instead of using the shared component. Without that guard, the same drift reliably recurs the next time a surface is built or redesigned, because nothing stops a future change from hand-rolling the element again. The generalizable rule: when consolidating N independently-drifted implementations of the same UI element into one shared primitive, ship a mechanical test alongside it that fails the build on reintroduction — the consolidation alone fixes the present instances, but only the guard test prevents the pattern from regenerating.