feat(competition): engine lane - submit ranking code, scored in a sandbox - #563
Merged
Merged
Conversation
benchmark v2 grows the axis recall-only benchmarks cannot measure: citation-correctness (the surfaced answer must be spelled by a quote whose byte-offset receipt verifies), receipt-coverage (fraction of surfaced claim items carrying a verifying receipt), and supersede-hygiene (a landed update must retire the stale value from the live claim set). the first two are guards that score 1.00 stock; the third is a lever at 0.00 alongside knowledge-update, pointing at lifecycle-driven supersession. memory_contract.MemoryContract implements ditto's five mining-contract tools (save, search, search subjects, fetch by id, search in subject) over the real store, so a shared-contract head-to-head can run both engines on one task set. saves route through the receipt-gated capture loop and return nothing durable when the gate is off — the review gate is never silently bypassed, even inside a harness. reference baseline refreshed to 0.52 +- 0.03 over seeds 1-6 (the category set change is a bench-contract version bump).
the verifiability categories change the category set, so seeds 1-6 rescore: composite 0.52 +- 0.03 replaces 0.57 +- 0.04 as row 0. the koth fold compares only max-bench-version scores, so the reigning-kit number and the challenger number stay on the same contract.
…dbox the engine-submission half of the koth ladder, and the real ditto equivalent: contributors submit a retrieval STRATEGY as python code (a rank(query, candidates, *, limit) -> ids function under contrib/strategies/), and vouchbench scores it. the kit lane only tuned coefficients; this is where a new fusion or reranker actually competes. the load-bearing rule: a kit is data and auto-merges; a strategy is code and NEVER auto-merges. vouch is a library people install, so a winning strategy earns the leaderboard and payout and ships only after human review merges it as a default - the review gate applied to engine code. scoring runs untrusted code, so vouch.strategy.run_sandboxed executes each submission in a python -I child that sets rlimits and installs an audit hook blocking network, subprocess, os.exec/fork, ctypes.dlopen, and filesystem writes before importing the file; the parent enforces a wall-clock timeout and treats any failure as no-reordering. the engine hook in build_context_pack is off by default and byte-identical when unset (retrieval.strategy, a dotted path, wires a merged strategy). koth-engine-gate.yml runs pull_request_target with contents:read, no secrets, no write token, no merge. hardened against an adversarial review. two confirmed defects fixed with regression tests: the audit hook read its blocklists from module globals (disarmable in two lines by reassigning __main__ globals) - now bound in closure cells; and the result travelled on the strategy's own stdout (any print corrupted it, silently scoring a good strategy as a no-op) - the child now writes the result to a private dup of stdout with fd 1 pointed at /dev/null. the docstring is honest that in-process python isolation is best-effort and the real boundary is the ephemeral runner plus human merge.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this is the engine-submission lane of the koth ladder - the part that makes vouch a real ditto competitor rather than a config knockoff. stacks on the kit lane (#562).
ditto onboards miners by letting them submit the retrieval harness itself; the kit lane only tuned coefficients, which has a low ceiling. this lane lets contributors submit real ranking CODE - a retrieval strategy that decides the order the reader sees candidates in, which is where a new fusion, a learned reranker, or a novel signal actually lives.
the interface is small: a file under contrib/strategies/ exposing rank(query, candidates, *, limit) -> list[ids]. a Candidate is kind/id/summary/score - data only; the strategy never gets the KB, the filesystem, or a socket. ordering is authoritative but bounded (invented ids ignored, omitted candidates appended), so a strategy can reorder and de-prioritise but never fabricate or hide a result - the same discipline the built-in rerank stage already follows.
the load-bearing difference from the kit lane: a kit is data and auto-merges; a STRATEGY is code and NEVER auto-merges. vouch is a library people install, so a winning strategy earns the leaderboard place and the payout, and ships only after a human reviews the code and merges it as a new default. that human review is the review gate applied to engine code - exactly the boundary ditto draws with its tee plus team deployment.
scoring runs untrusted code, so vouch.strategy.run_sandboxed executes each submission in a fresh python -I child that, before importing the file, sets rlimits (cpu/as/nofile) and installs an audit hook blocking network, subprocess, os.exec/fork, native ctypes.dlopen, and filesystem writes; the parent enforces a wall-clock timeout and treats any failure as no-reordering. the honest boundary is documented: an in-process python guard cannot stop a determined native escape, so it is defense in depth - the real boundary is the ephemeral ci runner (koth-engine-gate.yml: pull_request_target, contents:read, no secrets, no write token, no merge) plus the mandatory human merge.
the engine hook in build_context_pack is off by default and byte-identical when unset (parity proven: strategy=None == baseline == sandboxed-baseline == 0.6). a merged strategy is wired via retrieval.strategy (a dotted path, loaded in-process because it is now trusted, reviewed code).
hardened against an adversarial review (fan-out of sandbox-escape / gate-security / correctness agents, each finding verified by an independent refuter). two confirmed defects fixed with regression tests: (1) the audit hook read its blocklists from module globals, so untrusted top-level code could disarm the network/subprocess block in two lines by reassigning
__main__._BLOCKED_EXACT- now bound in closure cells holding the original frozensets, unreachable by a module-global rebind; (2) the result travelled on the strategy's own stdout, so anyprint()corrupted the json and silently scored a good strategy as a no-op - the child now keeps a private dup of stdout for the result and points fd 1 at /dev/null. the docstring is honest that in-process python isolation is best-effort (a determined gc/native introspection can defeat any such hook) and the real boundary is the ephemeral read-only-token runner plus the no-auto-merge rule.tests/test_strategy.py proves the sandbox blocks network, file-writes, and subprocess, kills infinite loops, survives crashing strategies, resists the module-global disarm, isolates the result channel from strategy stdout, and keeps apply_ordering set-preserving. gates green: ruff, mypy (110 files), full pytest.
files: src/vouch/strategy.py (interface + sandbox), src/vouch/context.py (optional strategy stage), src/vouch/bench.py (strategy arm), .github/scripts/score_strategy.py, .github/workflows/koth-engine-gate.yml, contrib/strategies/{baseline,example_lexical,README}, docs/koth-strategy-lane.md, tests/test_strategy.py.