← back to the main dashboard · curation cost per rubric

SWE-fficiency

SWE-fficiency, all 498 tasks held out · Haiku 4.5 agent · our GKE grader mirrors the official harness (workload timing + covering tests) · rubrics collected from injected slowdowns on SWE-smith repos only (268 pool rollouts, 69 failures, 69 lessons: the library is 10 / all = 69, and the "100" arms are replicates of "all")

1Experiment setting

Each task is a real repository (pandas, scipy, sympy, astropy, scikit-learn, numpy, matplotlib, dask, xarray) at a commit where a given workload is slow. The agent gets the issue text, the timing workload and the covering tests, and must make the workload faster without changing behaviour. A task counts as resolved when every covering test still passes and the workload is at least as fast as before; the table under the chart also reports the benchmark's own score (harmonic mean of speedup relative to the expert patch).

A Haiku 4.5 agent works in the repository by issuing bash commands. The settings differ only in what answers those commands:

In every world-model setting an execution command is still run in the pod (its side effects persist) and only the observation the agent sees is replaced by the world model's answer; reads, searches and edits are always real. Same settings as the 9/07 SWE-bench page.

The router is a fixed rule, not a model. The command string is split on every bash separator (\n ; && || | &), heredoc bodies are dropped and wrapper prefixes (sudo, timeout N, VAR=x) skipped; a command is code execution if any segment's first word is an interpreter or test runner (python, pytest, tox, make, manage.py, runtests.py, bash, sh, …), a script or repo executable (./run.sh, /testbed/bin/x), or contains a substitution / xargs / find -exec that runs one. Everything else — cat, ls, grep, sed -i, git, heredoc writes — is a read or edit and goes to the interpreter. A mixed command such as cat > t.py <<EOF … EOF && python t.py counts as execution.

The CWM (Haiku 4.5 or Opus 5) reads the task, the agent's current diff and the command, grades the diff against the 12 rubrics retrieved from the mined library (nearest to the task, the files the diff touches and the command; in the hybrid arms a standing task criterion — “does the change implement what the issue asks?” — is added to the list) and returns the per-rubric verdict. Files written by a run are quarantined so the agent cannot read the output back; every trajectory is audited for leakage. No setting ever sees the hidden grading tests.

2Results

resolved (tests pass and workload not slower), % of 498

Efficiency: median wall-clock per instance

Wall-clock per instance from the first command to the last (minutes), from the per-command timing records of each pod.

Efficiency: median agent LLM calls per instance

Efficiency: median seconds per LLM call (wall clock / calls, per instance)

Wall clock is calls × seconds per call. This chart isolates the second factor: the time the agent waits per step, which contains the model call and, in the world-model arms, the judge verdict; in the interpreter it contains the real execution. An arm that is slower here costs more per step; an arm that is slower only in wall clock took more steps.

Efficiency: median cost per instance (agent + world model, $; judge cost = graded observations x that arm's $/grade)

What the agent does: command mix per setting

Every agent command classified by the routing rule (inner ring: interpreter vs code execution) and by type (outer ring). "Write file" = heredoc / echo redirects; "edit / file ops" = sed -i, mv, cp…; "run tests" = pytest and friends; "python script" = python x.py; "inline python" = python -c / heredoc.

3Example traces

Instance scikit-learn__scikit-learn-25713, the same task under both settings. Colours: agent command and reasoning, interpreter real output, CWM world-model text, withheld nothing returned; rc = the exit code the agent saw.

4Curation cost per rubric

On-policy mining from the injected-slowdown pool on SWE-smith repos (make_perf_pool.py): the same Haiku agent runs on pool tasks with real execution, every failed rollout is distilled into one lesson by Opus, lessons are deduplicated and cut into the 10 / 100 / all libraries. No CWM grading during mining and no precision filter, so the per-rubric cost is the "cheap" recipe of the SWE-bench page. Prices: Opus 5 $5 / $25 per M tokens, Haiku 4.5 $1 / $5, GKE Spot ≈ $0.045 per node-hour.