ci(splinter): drop 10 stale ore_cllw allowlist rows, add unused-row check (CIP-3489) - #402
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe splinter test script now tracks allowlist rows that match no findings, reports stale rows in console and GitHub Actions summaries, and fails CI when stale rows or unmatched findings exist. Encrypted-JSONB allowlist entries and justification text were also updated. ChangesSplinter allowlist validation
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
9199d3f to
30c8ef2
Compare
…heck (CIP-3489)
The CLLW-OPE migration removed the `ore_cllw` surface, but its splinter allowlist
rows survived and nothing could catch them: the awk only tested `if (key in allow)`
per finding and never iterated `allow[]`, so no allowlist row could ever be
reported as unused — a future `ore_cllw` would arrive pre-waived.
- Remove the 10 inert rows: eql_v3_internal.ore_cllw_{eq,neq,lt,lte,gt,gte} and
eql_v3.{ore_cllw,has_ore_cllw} + eql_v3_internal.{ore_cllw,has_ore_cllw}. No
such functions exist in src/v3 any more, so the rows waived nothing.
- Fix regressed prose on the two live rows that still routed "via ore_cllw":
eql_v3.lt (jsonb_entry lt now routes ord_term → ope_cllw) and eql_v3.-> (the
chained ste_vec recipe extracts via eq_term / ord_term).
- Add a `for (k in allow)` unused-row check: allowlist rows that match no
finding now fail the gate (with a CI step-summary section), enforcing the same
"registered but referenced by nothing" invariant the known-failures gate uses.
Comment/blank heredoc lines are skipped so they are never counted as unused.
Verified end-to-end against a live install (splinter@55db5b1): the gate stays
clean (raw=1180, allowlisted=1180, unmatched=0, unused=0, exit 0) after the row
removals, and an injected bogus row is correctly reported unused and fails the
gate (exit 1).
30c8ef2 to
fa31d49
Compare
The merge queue ejects PRs when the sqlx shards fail with `53200 out of
shared memory`. Despite the wording this is not RAM exhaustion — it is
the shared lock table running out of slots, exactly as the error's own
hint ("You might need to increase max_locks_per_transaction") says.
max_locks_per_transaction is not a per-transaction cap; it sizes the
CLUSTER-WIDE lock table as max_locks_per_transaction * max_connections.
The shipped uninstaller drops both eql_v3 schemas with DROP ... CASCADE,
which locks every one of the ~5,682 objects it removes and holds them all
until commit (sqlx::raw_sql sends the script as one implicit
transaction). Measured: 5,689 locks for a single uninstall against a
stock table of 64 * 100 = 6,400 slots — 89% of the entire cluster's
budget in one transaction. Overlapping uninstalls exhaust it, and the
backends that then fail are mostly innocent tests that were merely
installing EQL, which is why this surfaces as `failed to apply
migrations` in unrelated tests.
Sharding is what tipped it over: the queue runs 2 shards where PRs run 4,
doubling test density per shard so the uninstall-heavy tests overlap.
Partitioning is a deterministic hash, so they land in the same shard on
every run — hence shard 2/2 failing on all four PG versions while the
same commit passes on the PR. This is not specific to one PR: #382, #401
and #402 ejected with the identical signature.
Raise max_locks_per_transaction to 1024 (102,400 slots), sized for the
worst case of nextest's 16 concurrent tests each uninstalling (~91k
locks). Verified on postgres:17: 6 concurrent uninstalls fail 5/6 at the
default and pass 6/6 at 1024. Costs ~42MB of shared memory (145MB ->
187MB), well inside the container's 2GB limit.
Add lock_capacity_tests.rs as a preflight gate so an undersized lock
table fails once, immediately, with a message naming the cause and the
fix — instead of six unrelated tests dying 20 minutes into CI with what
reads like an OOM. NB: a Postgres container created before this change
keeps the old setting, so local devs must recreate it
(mise run postgres:down && mise run postgres:up postgres-17).
Deliberately not "fixed" by giving the queue 4 shards: that would only
hide an undersized lock table until the catalog grew again.
Closes CIP-3489.
The CLLW-OPE migration removed the
ore_cllwsurface fromsrc/v3, but itstasks/test/splinter.shallowlist rows survived — and nothing could ever catch them. The awk classifier only testedif (key in allow)per finding and never iteratedallow[], so no allowlist row could ever be reported as unused. A future re-introduction ofore_cllwwould arrive pre-waived.Changes
eql_v3_internal.ore_cllw_{eq,neq,lt,lte,gt,gte}andeql_v3.{ore_cllw,has_ore_cllw}+eql_v3_internal.{ore_cllw,has_ore_cllw}. Confirmed 0 non-commentore_cllwreferences remain insrc/v3, so these rows waived nothing. The liveope_cllwandore_block_256comparator rows are untouched.ore_cllw":eql_v3.lt(thejsonb_entrylt wrapper now routesord_term→ope_cllw) andeql_v3.->(the chained ste_vec recipe extracts viaeq_term/ord_term).for (k in allow)unused-row check (tobyhede's point on the thread). Allowlist rows that match no finding now fail the gate — with a diagnostic and a CI step-summary "Stale allowlist rows" section — enforcing the same "registered but referenced by nothing" invariant theknown-failuresgate applies to suppressed tests. Comment (#…) and blank heredoc lines are skipped so they are never counted as unused.Verification (live install, splinter@55db5b1, PG17)
eql_v3.bogus_nonexistent_fnallowlist row → gate reports it "matched NO finding" and exits 1.CI-only change, no changeset (per CLAUDE.md exclusions).
Summary by CodeRabbit