Spec-agnostic task validation: openenv validate-task for external task.md packages (PostTrain Arena / BenchFlow)

[thegovind]

PostTrain Arena (proposed NeurIPS 2026) takes containerized RL task packages instead of models. The package format is BenchFlow's task.md: a task.md, an environment/ (Dockerfile + seed data), a verifier/ (pytest that writes a reward file), and an oracle/solve.sh reference solution. Validation runs with docker + python3 only, no framework install.

We want OpenEnv to author and validate these packages. This issue proposes a spec-agnostic openenv validate-task: support a couple of specs out the gate, clearly declare the ones we can't run locally, and let others register their own task spec.

Why OpenEnv envs don't drop in

An OpenEnv env is a long-lived server you drive over WebSocket. A task.md package is a one-shot container: the agent runs to completion, then a verifier reads the filesystem and writes the reward.

flowchart TB
    subgraph OE["OpenEnv today · served"]
      direction LR
      T["RL trainer"] -->|"reset / step / state<br/>over WebSocket"| S["Env server<br/>FastAPI, long-lived"]
      S -->|"reward in Observation"| T
    end

    subgraph BF["PostTrain / BenchFlow · packaged"]
      direction LR
      C["Task container<br/>one per trial"] --> AG["Agent runs<br/>to completion"]
      AG --> V["Verifier reads<br/>filesystem"]
      V --> R["reward.txt"]
    end

    OE == "missing: producer + validator" ==> BF

    classDef oe fill:#dbeafe,stroke:#2563eb
    classDef bf fill:#fef3c7,stroke:#d97706
    class T,S oe
    class C,AG,V,R bf

Loading

Neither model is wrong, but a served env is not a task.md package and there is no path between them yet.

Proposal: openenv validate-task

Detect the spec from signature files, dispatch to the matching validator, run structural checks, and optionally do a docker oracle replay. A registry lets anyone add a spec without touching core.

flowchart TD
    Dir["task directory"] --> Detect{"detect spec<br/>by signature file"}
    Registry["spec registry<br/>(pluggable)"] -.-> Detect

    Detect -->|"task.md"| PT["PostTrain / BenchFlow"]
    Detect -->|"openenv.yaml"| OEV["OpenEnv"]
    Detect -->|"task.toml"| HV["Harbor"]
    Detect -->|"registered"| TP["third-party spec"]

    PT --> Gate{"runnable<br/>locally?"}
    OEV --> Gate
    HV --> Gate
    TP --> Gate

    Gate -->|"yes"| Run["structural checks<br/>+ docker oracle replay"]
    Gate -->|"no"| Skip["declared unsupported + reason<br/>hosted verifiers · GPU/TPU · multi-agent local run"]

    Run --> Report["pass / fail report"]

    classDef ok fill:#dcfce7,stroke:#16a34a
    classDef plug fill:#ede9fe,stroke:#7c3aed
    classDef warn fill:#fef3c7,stroke:#d97706
    class PT,OEV,HV,Run,Report ok
    class Registry,TP plug
    class Skip warn

Loading

Support out the gate:

• PostTrain / BenchFlow task.md (port their check_task.py + run_local.sh logic, ~200 lines, no deps)
• OpenEnv native (openenv.yaml)

Declared unsupported, with a printed reason:

• Hosted Verifiers / PrimeIntellect source-envs (they run on their own surface, not a local task dir)
• Multi-agent scenes and simulated-user (structural check yes, local execution no)
• GPU/TPU-bound tasks the local runner can't build

Not from scratch

The dispatch-by-signature and registry patterns already exist in and around this repo:

• BenchFlow already dispatches with detect_adapter (today only Harbor task.toml).
• openenv import ([feature] Import external envs (ors, verifiers) #726) already ships an importer registry for ORS/OpenReward and Verifiers.
• envs/tbench2_env already wraps a task.toml + tests/ layout and scores with pytest to a binary reward, which is the same shape as a task.md verifier.
• This lines up with the auto-validation RFC (RFC: 008 Environment auto validation #778) rather than opening a new track.

Scope

• openenv validate-task <dir> [--spec auto|posttrain|openenv|...] with a spec registry
• PostTrain task.md structural validator (frontmatter, required files, ## prompt, verifier/rubrics/)
• OpenEnv native validator (wraps existing openenv validate)
• openenv run-local: docker build + oracle replay (expect 1.0) + empty trial (expect not 1.0)
• Registry hook + docs so a third party can add a spec
• One end-to-end example task.md (cyber domain) that passes the command

Open questions

• Extend the existing openenv validate or add a separate validate-task verb?
• Where does the spec registry live so it stays importable without heavy deps?
• Vendor PostTrain's checker or depend on it? It's designed to be copy-free.

[AmyTao]

Hi @thegovind! This is Amy from the BenchFlow community. I think a separate validate-env command would be better here, to avoid conflicting with what the existing validate command already does.

[thegovind]

Direction update after reviewing RFC 008 (#942) and its implementation stack:

• @AmyTao's suggestion is right in substance: task-package validation should have a separate command instead of extending the existing openenv validate flow.
• RFC 008 now provides the spec-neutral adapter, planner, executor, and report foundation, and explicitly reserves one-shot task packages for this issue.
• This issue should therefore narrow to the missing PostTrain/BenchFlow lane: the task.md adapter and structural checks, a one-shot Docker oracle/empty-trial runner, a separate task CLI, docs, and one end-to-end example.
• The task command should not wrap OpenEnv-native validation; served OpenEnv environments remain under openenv validate. The exact task command can be settled as openenv validate-task versus openenv task validate. I would avoid validate-env, because that is too close to the existing environment command.
• Public plugins can support local author validation, but official certification must use pinned trusted adapters.

I have proposed to @zkwentz that I own this focused follow-up rather than opening a competing validation-core PR. The likely implementation sequence is:

1. PostTrain/BenchFlow adapter plus structural policy.
2. One-shot local runner and shared report integration.
3. Task CLI, plugin/docs surface, and example package.

I am holding off on a branch from main. If we develop before #942–#947 land, the clean base is #947 (rfc008-06-cli); otherwise this should start from updated main after that foundation merges.

[AmyTao]

@thegovind Hi Govind! Could you point me to the branch you are working on for the implementation? I'd like to test on the posttrain arena side! thx