← back to the main dashboard · curation cost per rubric

MLE-bench Lite — harness 2 (9/14): interpreter, CWM-only, dual channel

MLE-bench Lite, all 22 competitions held out · Haiku 4.5 agent · official mlebench grader · 2-hour budget per competition · harness 2 (pulled 2026-09-14): agent told its channel, installs never routed, judge latency recorded; every arm writes final.py which the harness runs once, uncapped, at collection. CWM library = on-policy all + task criterion + preamble. Dual arms run each routed command for real under a 30 / 60 / 120 s budget AND grade the change set.

1Experiment setting

Each task is a Kaggle competition: the agent gets the competition description and data and must leave a valid submission file; the official grader scores it against the real leaderboard. Medal = the submission clears the bronze threshold; the efficiency page uses 'beats the no-execution floor' as the solved bar.

A Haiku 4.5 agent works in the repository by issuing bash commands. The settings differ only in what answers those commands:

In every world-model setting an execution command is still run in the pod (its side effects persist) and only the observation the agent sees is replaced by the world model's answer; reads, searches and edits are always real. Same settings as the 9/07 SWE-bench page.

The router is a fixed rule, not a model. The command string is split on every bash separator (\n ; && || | &), heredoc bodies are dropped and wrapper prefixes (sudo, timeout N, VAR=x) skipped; a command is code execution if any segment's first word is an interpreter or test runner (python, pytest, tox, make, manage.py, runtests.py, bash, sh, …), a script or repo executable (./run.sh, /testbed/bin/x), or contains a substitution / xargs / find -exec that runs one. Everything else — cat, ls, grep, sed -i, git, heredoc writes — is a read or edit and goes to the interpreter. A mixed command such as cat > t.py <<EOF … EOF && python t.py counts as execution.

The CWM (Haiku 4.5 or Opus 5) reads the task, the agent's current diff and the command, grades the diff against the 12 rubrics retrieved from the mined library (nearest to the task, the files the diff touches and the command; in the hybrid arms a standing task criterion — “does the change implement what the issue asks?” — is added to the list) and returns the per-rubric verdict. Files written by a run are quarantined so the agent cannot read the output back; every trajectory is audited for leakage. No setting ever sees the hidden grading tests.

2Results

any medal, % of 22

above the leaderboard median, % of 22

MLE-bench's headline metric is the medal rate (top chart); "above the leaderboard median" is the secondary, easier bar. Valid-submission counts are in the table.

Efficiency: median wall-clock per instance

Wall-clock per instance from the first command to the last (minutes), from the per-command timing records of each pod.

Efficiency: median agent LLM calls per instance

Efficiency: median seconds per LLM call (wall clock / calls, per instance)

Wall clock is calls × seconds per call. This chart isolates the second factor: the time the agent waits per step, which contains the model call and, in the world-model arms, the judge verdict; in the interpreter it contains the real execution. An arm that is slower here costs more per step; an arm that is slower only in wall clock took more steps.

Efficiency: median cost per instance (agent + world model, $; judge cost = graded observations x that arm's $/grade)

Mock: median wall clock per competition under the canonical architecture (code execution never runs)

Computed from the recorded per-command timings of the runs above. "Execution removed" subtracts the time of every command the router classified as code execution from the hybrid arm's wall clock (its judge calls are already inside that wall clock). The interpreter rows replace each of its execution commands by one judge call of 10 / 15 / 20 s. Totals over the set: interpreter 23.6 h, hybrid as run 18.0 h, hybrid with execution removed 3.9 h (83% below the interpreter). These are latency mocks: without execution nothing writes the submission file, and agents behave differently when they never see a training result.

What the agent does: command mix per setting

Every agent command classified by the routing rule (inner ring: interpreter vs code execution) and by type (outer ring). "Write file" = heredoc / echo redirects; "edit / file ops" = sed -i, mv, cp…; "run tests" = pytest and friends; "python script" = python x.py; "inline python" = python -c / heredoc.

3Example traces

Trace pair pending: needs the same instance in h2-mle-dual60-opus and h2-mle-cwm-opus.

4Curation cost per rubric

On-policy mining from the DABench + DA-Code data-science pool (run_da_pool.py): the same Haiku agent runs on pool tasks with real execution, every failed rollout is distilled into one lesson by Opus, lessons are deduplicated and cut into the 10 / 100 / all libraries. No CWM grading during mining and no precision filter, so the per-rubric cost is the "cheap" recipe of the SWE-bench page. Prices: Opus 5 $5 / $25 per M tokens, Haiku 4.5 $1 / $5, GKE Spot ≈ $0.045 per node-hour.