Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,16 @@ uv run python scripts/prepare_task.py \
The prepared OpenCode workspace deliberately excludes `verifier/`, `oracle/`,
and `evidence/`.

## Reward Integrity

Article-suite agents run without network access, trusted task files are
root-owned, and submitted policies execute in an unprivileged Landlock worker
outside the root verifier process. A deterministic trajectory and artifact
audit forces reward to zero on confirmed integrity violations.

See [`docs/reward-integrity.md`](docs/reward-integrity.md) for the threat model,
violation codes, and required adversarial tests.

## OpenCode Article-Suite Experiment

OpenCode is the default and only leaderboard harness for the nine-task suite.
Expand Down
7 changes: 7 additions & 0 deletions docs/learning-beyond-gradients-suite.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ fixed public starter

The task workspace never contains `verifier/`, `oracle/`, or `evidence/`.

The agent and environment also run with `network_mode: no-network`. Managed
OpenCode permissions deny web retrieval, external-directory access, and
subagents. During final evaluation, submitted policy code runs as UID `agent`
inside a Landlock worker and cannot share the root verifier's filesystem or
Python process. The trajectory and final artifact must pass the deterministic
integrity gate before scientific reward is accepted.

## Atari57 exception

Atari57 is one aggregate task rather than 114 standalone tasks. Its artifact
Expand Down
75 changes: 75 additions & 0 deletions docs/reward-integrity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Reward integrity

GenesisBench scores scientific or control performance only after a deterministic
integrity gate passes.

## Threat model

Submitted code is untrusted. It must not:

- access `/oracle`, `/verifier`, verifier logs, or hidden configuration;
- use network retrieval or copy a completed upstream answer;
- modify trusted evaluators or runtime modules;
- create symlinks, import hooks, compiled extensions, or special files;
- monkeypatch the verifier or execute dynamic code;
- tamper with reward artifacts;
- fabricate experiment-accounting records.

## Prevention

All article-suite tasks run the agent, environment, and verifier without
network access. OpenCode web retrieval, external-directory access, and
subagents are denied by the managed run configuration.

The task image keeps `/app/evaluate.py`, starter artifacts, task context, and
`/opt/genesisbench` root-owned and read-only. Only the submission directory and
`/app/work` are agent-writable.

The verifier never imports submitted Python into its root process. A persistent
policy worker runs as UID `agent` under Landlock with access only to:

- system libraries;
- a copied submission bundle;
- a shared observation buffer;
- an episode-specific copied MuJoCo XML when the task permits model-based
planning.

The worker cannot read `/oracle`, `/verifier`, `/logs`, or hidden suite files.

## Detection

Before scientific scoring, `genesisbench.integrity` audits:

- the root-published ACP trajectory;
- the submitted artifact tree;
- prohibited upstream answer hashes;
- forbidden imports, paths, file types, symlinks, and dynamic-code hooks;
- untrusted Atari57 completion claims.

The report is preserved at `/logs/verifier/integrity.json`.

## Reward composition

Integrity is a binary multiplier:

```text
final_reward = scientific_reward if integrity_pass
final_reward = 0 otherwise
```

`reward.json` keeps BenchFlow's numeric reward-only contract. The scientific
score and violation evidence remain separately reviewable in
`genesis-score.json` and `integrity.json`, so confirmed misconduct is not
conflated with an ordinary low-performing policy.

## Required acceptance tests

Every task change must keep these gates green:

1. schema and task validation;
2. starter/reference/oracle regression tests;
3. clean submission passes integrity;
4. web retrieval, hidden-path access, source copying, symlinks, import hooks,
reward tampering, and fabricated accounting receive reward zero;
5. Docker/Daytona policy-isolation smoke;
6. complete repository test suite and lint.
5 changes: 5 additions & 0 deletions experiments/article_suite/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ the nine task packages derived from the article:
Every run uses BenchFlow's registered `opencode` ACP harness. OpenHands is not
part of this suite.

All task and agent network access is disabled. The managed OpenCode
configuration denies `webfetch`, `websearch`, external-directory access, and
subagents. Final policies execute in an unprivileged Landlock worker, and any
confirmed trajectory or artifact integrity violation forces reward to zero.

## Experiment protocol

The current leaderboard protocol is defined in `protocol.toml`:
Expand Down
6 changes: 6 additions & 0 deletions scripts/run_article_suite.py
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,12 @@ def _opencode_config(model: dict[str, Any]) -> dict[str, Any]:
}
config: dict[str, Any] = {
"$schema": "https://opencode.ai/config.json",
"permission": {
"external_directory": "deny",
"task": "deny",
"webfetch": "deny",
"websearch": "deny",
},
"provider": {
provider: {
"models": {
Expand Down
Loading