Rubric libraries: can a coding world model with rubrics stand in for running code?

The Haiku agent's own rollouts on other data-science and ML benchmarks, none resembling an MLE-bench competition. Each task was run several times (seeds). The libraries are mined from the 2nd to 4th collections below.

Failed commands: found by exit code and error text. The 1st collection (5,397 rollouts: InfiAgent-DABench 4,337, DA-Code 1,060) was used only by the original Wrong-answer pipeline.

What "final" means. After a rollout ends, its submitted answer is graded. The original miner (Wrong-answer) kept only rollouts graded wrong and gave Opus four things: the task, the last version of each script the agent wrote, the submitted answer and the grader's verdict. It never saw the commands the agent ran or their outputs: no tracebacks, time limits or missing packages, none of the intermediate steps. Rollouts graded correct were skipped even when commands failed along the way (395 of the 1,192 failed commands in its source). So its rubrics describe mistakes visible in the final code and answer (wrong metric, split or format), not the failures that happen while code runs, which is what the CWM predicts at every step.

Failure events. First collection: failed commands found by exit code and error text; wrong final answers. Collection pool and MLE-bench: failed commands, plus the cause of each wrong result found at grading.

Two rollouts behind Wrong-answer rubrics (click to open):

expected -> file result.csv: WRONG/MISSING

### Validation split and feature set that don't mirror the actual prediction setting
- **Applies when**: `task` -- a model must produce predictions for a supplied test file whose rows come from an ordered (e.g., time-indexed) process, and the scripts choose which of the shared columns to use as features and how to hold out data for validation.
- **Pattern**: The attempt builds features by blanket-excluding columns (dropping some that exist in *both* train and test and are strongly related to the target) and then measures quality with a random/shuffled train-test split of ordered data, reporting the resulting optimistic error/R² as evidence the submission is good — without ever checking predi

expected -> file submission.csv: WRONG/MISSING

### Optimizing/validating with a metric other than the one the task specifies
- **Applies when**: `task` -- the task states an explicit scoring metric (e.g., a rank/agreement/ordinal or cost-weighted metric) and the scripts must choose a model, tune it, or decide when the answer is good enough.
- **Pattern**: The scripts never implement or compute the stated metric; they select the "best" model and ensemble weights using a generic proxy (accuracy, weighted F1, plain log-loss) and treat the target as unordered classes, so the reported/validated numbers say nothing about the actual grading score. Often compounded by a contaminated check (the final ensemble is fit on all training rows and then 

Reading every command of every rollout fixed what the miner could see, not how often it wrote about each failure. Crashes are 76% of the pool's failures and 13% of the 3,550-rubric library. The miner writes up to 6 rubrics per rollout and merges near-duplicates: a crash seen hundreds of times collapses into a few rubrics, while every modelling lesson is new text and is kept. Nothing in the generation counts frequency: per pool failure, the library holds 21× more rubrics for other failures than for crashes.

Distance to the pool's mix: Wrong-answer 78, whole rollouts 64 (0 = same mix, 100 = no overlap). Categories: Sonnet 5.5 reading each rubric.

Two curation methods match the library to the collection pool's failure mix: Filtered cuts the whole-rollout library to the pool's proportions, Quota writes rubrics per category until each category's share is filled. Category split retrieval then gives each category its share of the 12 rubrics the CWM sees. Every library so far, by date built:

Each library that was run, against MLE-bench execution

Click a library. Execution = the failures MLE-bench Lite shows when every command runs (the ceiling runs).

Audit of the 29 modelling-quality rubrics: 14 came from score contrasts across eight tasks; 15 came from wrong-output cases. All had support 1. Incomplete script bundles and confounded changes prevented a reliable final-program check. This audit does not establish that all 952 rubrics are bad.

Fresh collection: 128/128 attempts recorded; 125 valid; 0 collection errors, not retried. Sixteen new OpenML tasks, eight attempts each; 12 tasks for generation and four for development. Exact scripts, scored replay and all attempts are retained.

New-only quality library: 1 accepted lessons. Curation + review API cost per accepted rubric: $2.87 (collection excluded). A lesson needs source-pair support plus positive discrimination on two other tasks, including development, with no reversed development prediction. No old rubrics are blended in.

Partial replication: user-requested early stop. 6 queued arms were cancelled. Already-active arms finish; all results and unresolved failures are retained. Counts below keep the original planned denominators, not a completed four-repeat comparison.

Gated screen: compare the frozen new library on the existing eight validation tasks, which never enter curation. Reuse the existing Quota predictions rather than rejudge them. At least 12 lessons, complete coverage, better ordering and rubric-specific discrimination are required. Previously inspected validation is not fresh held-out evidence.

The four-repeat plan was stopped early by the user. Every row uses the same fixed-schedule checks (defined in the settings table): a real final.py run, stopped after at most 180 seconds, on every second code-running command, plus a real run at submit that holds only a crashed submission. No CWM submit veto. The quality-only row runs only after the screen passes and controls complete.

MATS only: eight Spot collection workers, then at most 16 non-Spot evaluation workers; never overlapping. No selective retries; unresolved infrastructure losses stay visible. The check policy—not realized calls or runtime—is fixed. Lite remains development evidence.

Quality-curation funnel: 15 proposed lessons → 7 passed grounding/format checks → 6 unique → 1 accepted.

• Wrong-program citations: 8 proposals failed because 21 quotes were attributed to the wrong program. The saved requests contained the correct programs; this was generated evidence misattribution, not a parser bug.
• Applicability: 3 rejected source contrasts marked the weaker program as violating the lesson but the stronger program as outside its scope. The frozen gate requires both to be applicable.
• Quality review: 4 lessons failed the requirement that every comparison pass its quality audit. These checks varied with the comparison context. Rejection is not proof that every proposed lesson is false.

All 114/114 reviews were usable. Offline replay reproduced the frozen selection and library exactly. Criteria, judgments and accepted lessons are unchanged.

Collection accounting: among 6 flagged records audited, 1 recovered successfully but retained a stale error; 3 failed before an episode began and 2 lacked a resolved final collection. The collector fix preserves failed-read history and clears the active error only after completed collection. Historical records and their conservative status remain unchanged.

Next: use a separately versioned protocol for correctly attributed evidence and both-applicable near-misses. This diagnosis does not establish an efficiency or medal improvement.

Saved-evidence audit, 2026-10-04. No new model calls, benchmark episodes or acceptance changes. Rejection reasons overlap. Evidence: curation audit · collection audit.

Completed run: where candidates were lost

One candidate was blocked only by the restart's failed-authentication review. Another had only output-format gaps. These are not proof that the lessons are false; neither is accepted retrospectively.

Repair: v3 implements native structured outputs and harness-bound citation hashes. See the v3 section for run status; support gates remain unchanged. No benchmark scaling yet.

Offline diagnosis: 281 saved calls replayed, no new model calls. Frozen outputs reproduced exactly.

Curation: insufficient valid lessons. 0 accepted lessons. This is a new protocol, not a regrading of the original one-lesson library.

Same collection data: 19 valid-program contrasts; 16 eligible for generation. The original 12 training / four development task split is retained. No MLE-bench or validation examples enter curation.

Unchanged support gates: source contrast plus two other positive tasks, at least one development positive, no development reversals, complete reviews and at least 12 accepted lessons before the CWM screen. Scores remain observational evidence, not isolated causal effects.

No collection or Lite runs start automatically. A separately invoked paid curation can proceed to the existing pool screen only if its library passes the gate. More accepted lessons, better medal rates and a rubric-specific speedup are all still unproven.

Evidence: offline plan and verification. The original protocol, calls, decisions and library are preserved.

Curation: insufficient valid lessons. 4 accepted lessons. These changes affect offline rubric curation, not the coding agent's hybrid CWM feedback or interpreter checkpoints.

Same 19 pool contrasts, 16 eligible for generation. Same source/cross-task/development support gates and 12-lesson screen minimum. V1/v2 evidence is frozen, not repaired.

Offline implementation evidence. No new validity, medal or above-median result.

Compare full-trace, outcome-aware curation with final-program contrasts; then compare lessons collected by Haiku and Sonnet. The fresh teacher collections use the same 16 training tasks and eight attempts per task, not benchmark examples.

Continues after the workstation ran low on disk space. Completed teacher collections and completed curation results are not repeated. Saved model responses are hash-checked and reused; interrupted requests remain in the record and replacements are separately accounted for. The original deadline and acceptance checks are unchanged. This is a continuation, not another independent replicate.

Overnight observer: 2 recorded alert(s); see the monitoring report.

Same wall, turn, command and sandbox limits for both teachers. API spend is not matched; a dollar-based early stop is disabled for both because the Sonnet SDK tariff is unreliable. This does not test expansion to new task families. Per-rubric cost is curation-only, excludes teacher collection, and is shown only when accounting is complete.

At most eight Spot workers on MATS. This continuation uses two API-only curation lanes and launches no collection. Agent comparisons remain gated. Original reports stay preserved; no automatic episode retries or benchmark launches.

Does following a proposed lesson improve the same program? Cases are selected using source checks only, not favorable development scores. These are candidate lessons, not accepted library rubrics.

Each scoped correction is independently reviewed, then compared with the original on the same pool data and limits. Three repeated executions per version; at most 60 executions for this case list. 60 program executions attempted; 27 matched pairs valid on both sides.

The first attempt lost a GCP Spot node. Its 32 attempted slots are never replayed: 29 valid results and three unknowns remain in the record. A bounded continuation runs only the 28 untouched slots, with the same programs, runtime, scoring and original deadline. This table combines both portions of the same 60-slot schedule, not new replicates.

Score change is corrected minus original balanced accuracy, averaged over paired-valid repeats. Validity failures and missing executions remain in the report; repeats are not independent datasets. Correction types were assigned before execution, not inferred from scores. Diagnostic-only and risk-guard corrections may leave predictions unchanged even when they satisfy the lesson.

Within-source mechanism diagnostic, not a medal, transfer, or agent-speedup result. At most four MATS Spot workers; only pool data. No new teacher collection or benchmark run.

Start with the completed, measured corrections. Generate new lessons from the actual before/after programs, then check whether applying a lesson helps on another task. Fresh correction proposals use public task context and code—not benchmark errors or scores.

Model responses: 239 / 239 reserved requests; 0 failed, 0 unresolved.

1 lesson passed static screening; the 12-lesson library gate was not met. Static judgments are not measured transfer success.

Rejected lessons do not end the whole study: the next planned source-expansion batch follows. The original quality checks and 12-lesson library gate stay unchanged. Expensive cross-task reviews run only after a source audit passes; source rejections stay in the denominator. Source-audited candidates in a diagnostic are not an accepted CWM library.

29 measured collection-pool contrasts on 11 training tasks. Each fold excludes one whole task from generation. All conditions share source anchors, generation repetitions, models and validation. No benchmark or old development outcomes enter the study.

Single example uses one source pair. Multiple examples adds the other training evidence. Counterexample-aware uses the same additional evidence but explicitly reconciles improvements, regressions and unchanged results, and requires consistent scope for satisfying alternatives.

All review decisions are frozen. Unchanged-score cases and distinct-task support are reported separately from favorable ordering; missing responses remain unknown.

132 matched generation calls planned; at most 2580 model requests for the frozen design. Deadline: 2026-10-06T07:50:50.693106+00:00. Four concurrent calls; no automatic retries, new code runs or cloud resources. Scientific rejection advances the remaining groups. This is task-excluded TRAIN cross-validation, not a fresh benchmark result, an agent efficiency gain or an accepted library.

Compare each unchanged original with a Haiku task-only correction (no lesson) and a matched Haiku lesson-guided correction, selecting other TRAIN collection-pool programs marked violates before either correction or its outcomes. One lesson per correction: this tests lesson content, not the 12-rubric retrieval policy or a full agent episode.

complete; phase complete; updated 2026-10-06 01:59 UTC; deadline 2026-10-06T08:41:48.675293+00:00.

Request compatibility: 8 original requests returned HTTP 400; they remain failed requests, not negative lesson results. The native-schema compatibility check was provider_accepted in 9.42 seconds. The continuation reuses that response and runs only previously unstarted units. One intermediate schema-invalid response is also preserved, not repaired. The current wire schema is the exact native schema; explicit field instructions and native local validation remain. No deadline or overall request cap was increased.

Execution handoff: the first service could not find the cloud CLI tools. The corrected service environment passed a read-only MATS preflight. Execution reuses the frozen cases and approved programs; the earlier stop record is preserved. New model requests in this handoff: 0.

Model schedule scope_review: 28 / 28 finished; 0 active model calls. Requests: 286 reserved, 278 returned.

Curation: 9 grounded candidates from 9 planned generation requests; source audit 4 passed / 5 rejected or unknown (of 9); independent source consistency 4 / 4; 4 distinct lessons selected.

Target applicability: violates 59; satisfies 28; not applicable 76; insufficient evidence 0; unknown 18 (of 181 reviews). Selected: 19 cases on 10 tasks.

Comparison coverage: both corrections were approved in 4 / 19 cases across 4 tasks. Only paired-valid executions can establish their score difference; approval is not a score gain.

Local patch-validation failures before scope review: no lesson: 1; lesson-guided: 8. These are unusable proposed edits, not measured score losses.

Main contrast, lesson-guided minus no lesson: mean balanced-accuracy difference +0.0208; mean execution-seconds difference +0.7268. Paired-valid coverage 12 / 57; 45 pairs without both valid scores. Execution state: complete.

Same-case unchanged-program baseline

Identical 4 cases across 4 tasks in all three rows; 12 dependent repetitions. A gain over no-lesson edits is not necessarily a gain over leaving the program unchanged.

Lesson-guided: 3 better / 1 worse cases than no lesson; 2 better / 2 worse than the unchanged original.

3 no-lesson timeout executions belong to 1 case(s) without an executed lesson-guided counterpart. They are not a paired validity advantage.

Measured writer defects (earlier study): 131 / 132 generations were schema-valid but yielded only 25 native candidates; 111 emitted lessons had empty applicability. Independent reviews: 82 / 99 usable, 17 unusable, including 13 truncated. Prospective repair uses explicit nonblank-field instructions, native local validation and a 6,000-token review cap; formatting repair is not evidence of better lessons.

Source-reviewed is not deployment-approved: native gates and the 12-lesson library gate remain unchanged. Effects condition on both arms valid; declined, rejected, missing and ambiguous outcomes are not zero scores. Three repeats preserve program seeds; repeats and cases sharing a task are dependent. TRAIN-only reused tasks, not a benchmark efficacy result, CWM-agent efficiency result or accepted library.

Envelope: 8 concurrent API requests / 4 GKE workers on MATS only; up to 24 cases × 3 arms × 3 repeats = 216 program attempts, 1,000 API requests and eight hours. Public JSON: plan · progress and results · writer diagnosis · request compatibility · correction coverage · matched final results.

Change under test: return one complete corrected program instead of substring replacements, in both the no-lesson and lesson-guided arms. Same frozen cases, lessons, models, output budgets, seeds and validation gates; the unchanged original remains the third arm. No selection of previous winners.

The previous study lost 8 / 19 guided proposals and 1 / 19 unguided proposals before scope review. This experiment first measures usable correction coverage, then score and runtime against both controls. A formatting fix is not evidence of better lessons.

Status: execution complete; interrupted outcomes retained; phase stopped; deadline 2026-10-06T15:14:05.190416+00:00.

Current model schedule: 29 / 29 finished; 0 active. Total requests: 69 reserved, 69 returned.

Execution-only recovery after infrastructure stops. 1 previously unstarted executions; the earlier 114 valid executions and 4 interrupted/unknown outcomes are retained. 0 new model requests. No attempted program is retried; the original deadline and scientific gates are unchanged.

1 earlier invalid execution is also retained, not retried.

Finish the sole unstarted slot after the disk-reserve stop; retain every previous outcome.

Saved-output audit: three file-reference rejections came from changed module docstrings containing existing paths, not new file access. Three other outputs do not parse as Python; one uses blocked dynamic class construction. Original rejections stay in place. These checks do not establish whether the rejected edits would improve predictions.

No observed outcome includes unapproved and unstarted slots, as well as interrupted executions.

Local construction/validation failures, No lesson: introduced_data_or_file_reference: 1; python_parse_error: 1.

Local construction/validation failures, Lesson-guided: introduced_data_or_file_reference: 2; introduced_or_changed_external_access: 1; python_parse_error: 2.

Same-case comparison

Identical 5 paired-valid cases across 5 tasks; 15 dependent repetitions. Means average each case's repetitions first; repeated executions are not new tasks.

Partial-evidence comparison: interrupted outcomes remain unknown and are excluded only from paired-valid means, never from the all-case accounting above.

Stop reason: all_admitted_slots_exhausted_partial_evidence.

All selected cases stay in the denominators. Rejected, declined and missing results are not zero scores. TRAIN-only diagnosis, not a medal-rate or full-agent efficiency result. No library deployed and no acceptance gate relaxed. Limit: 76 model requests, 8 concurrent; 4 MATS CPU workers, 300 seconds per execution, 3 repetitions and an eight-hour deadline. Public JSON: plan · progress and results · saved-output diagnosis.

Rates are over all finished runs of 22 competitions each (mean valid submissions in brackets). Runs of the same setup vary a lot (the floor: 7, 15 and 9 valid), so gaps of a few points are noise. Mismatch columns: total variation distance on the 22 failure categories with crashes split by type, 0 = identical mix, 100 = no overlap. Execution = the failures MLE-bench Lite shows when every command runs (three ceiling runs: failed commands plus the cause of each wrong result); pool = the collection pool's failures, which the libraries were matched to. Hover a header for its definition.

Recorded time against the ceiling

Historical records lack exact episode boundaries. This window includes logged setup waits and the final collection run, but not unlogged provisioning, grading or earlier retry attempts; it is not pure agent time. Means include valid and invalid episodes. Real execution excludes installs and read-only harness probes. The hybrid changes execution policy as well as feedback; this ratio does not isolate the rubric contribution.

Same 22 competitions, same agent. The real run at submit is capped at 3 min and only holds a submission whose final.py crashes or that the CWM still flags. 'No rubrics': the CWM judges the change against the task statement alone (no library). The earlier scheduled-check sweep was stopped after GKE transport errors entered agent feedback; its partial grades are not included here or used for causal inference. It also differed in CWM submit vetoes. The fixed-policy controls below remove that veto and add a task-only CWM control. Feedback counts cover messages shown to the agent, not silent accepted-submit checks. Real-run time excludes environment/data inspection, installs and final collection. 'Checks on every change' runs the deliverable on every routed code request, without a CWM call.

Same execution policy, fixed-schedule checks (defined in the settings table): the agent's 2nd, 4th, 6th… code-running commands each trigger a real run of final.py, stopped after at most 180 seconds; the other code commands are not executed. This counts commands; it is not a timer. At submit final.py runs again; only a real crash holds the submission (at most twice), and the CWM has no submit veto. The two CWM rows also show the CWM's prediction on every code command. Same agent and limits; Quota uses the frozen 952-rubric library, with no new mining. Four full 22-competition runs per row.

• Quota rubrics + CWM minus Task-only CWM: validity difference (pp): +13.6 [+1.1, +28.4]; real-time difference (min): -0.9 [-3.1, +1.1]; episode + final difference (min): -2.2 [-5.4, +1.5]
• Validity sensitivity: +6.8 to +13.6 pp if lost finals had failed or succeeded. Adverse-endpoint 95% CI: [-3.4, +19.3] pp.
• Quota rubrics + CWM minus Checks only, no CWM: validity difference (pp): +22.7 [+11.3, +35.2]; real-time difference (min): -3.2 [-5.2, -1.2]; episode + final difference (min): -4.3 [-7.2, -1.5]
• Validity sensitivity: +17.0 to +22.7 pp if lost finals had failed or succeeded. Adverse-endpoint 95% CI: [+4.5, +30.7] pp.
• Task-only CWM minus Checks only, no CWM: validity difference (pp): +9.1 [-3.4, +21.6]; real-time difference (min): -2.3 [-4.5, -0.1]; episode + final difference (min): -2.2 [-6.0, +1.4]
• Validity sensitivity: +3.4 to +15.9 pp if lost finals had failed or succeeded. Adverse-endpoint 95% CI: [-10.2, +18.2] pp.

Collection-loss caveat: all episodes remain in the denominator. Lost finals are not proven code failures; the observed CIs are not clean causal estimates. Sensitivity changes only missing final validity, not runtime. Nine agent-loop OOM terminations per row remain real failures. Times cover the last saved attempt, including failed outcomes and CWM latency; final collection is included in the last time column, but earlier retries, provisioning and grading are not. CIs resample competitions, keeping four repeats together. Same check rule does not force identical realized check counts or duration.

What this tests: Quota versus task-only isolates adding the curated library under this policy; Quota versus checks-only tests the whole feedback channel. Higher validity at longer runtimes is a tradeoff, not an efficiency win. These controls do not by themselves prove adaptive triggering or distribution matching is the cause.

Reading: the package has a validity advantage over checks even if their five lost finals all succeeded. The extra benefit of curated rubrics over task-only CWM is less certain after collection-loss sensitivity. A rubric-specific speedup is not established; medals are 7 versus 6, and above-median outcomes are 12 versus 12.

Quality lessons exist, but this category is not retrieved. S9P01 is the existing category “Modelling choice leaves score on the table”.

Its support gives 0.20 of 12 slots before integer allocation, then 0 afterwards. The fixed split spends ten slots on crashes, timeouts and missing packages, one on output format and one on missing-value checks. Across 924 saved verdicts, S9P01 was never retrieved, marked applicable or flagged. Baseline-validation, metric-selection and hyperparameter-validation categories also had zero retrieved exposures.

Logged verdicts are not all grade-wrapper attempts or cached feedback deliveries. Other categories and the task criterion can still contain modelling advice. This establishes a coverage gap, not that filling it will win medals; training limits, CWM accuracy and agent use of advice remain untested.

Previous validation collection — probe: 64/64 new OpenML attempts recorded; 64 valid. Eight new small classification tasks, eight real interpreter attempts each. These tasks are reserved for validation, not rubric mining.

The frozen CWM comparison used current retrieval, two protected quality slots, and the task criterion alone. The protected-slot screen did not clear its complete-coverage gate, so it did not launch a Lite arm. The new curation experiment above uses separate training tasks, not these validation examples.

The existing collection had no unused task-disjoint examples. OpenML tests small-tabular quality discrimination, not medal gains on large competitions. Its datasets are public. Final collection now uses a durable exit record; a MATS smoke passed and its pod was deleted. New retries preserve all attempts rather than overwriting their costs.

Pure CWM: no code runs for real during the episode (no run on a clean prediction, none at submit); every library is curated from scratch by the Quota miner at that size (planned size; a category whose new failures were already covered stops early, so large sizes can come out smaller), served by Category split. The 1,000 row is the method's own library (952).

Doubled wall clock, step and cost limits; the per-command time limit (10 min) is unchanged.

One run per row of the full 75-competition MLE-bench, same harness as Lite (CPU pods, 2 h agent budget), counted on the competitions graded in all three rows (in brackets). Left out: three whose mle-bench graders fail on pandas 3 (tgs-salt, tensorflow2-question-answering, vinbigdata) and three whose ceiling pods were evicted in every attempt because the agent unpacked the data past the pod's 10 GB disk (freesound-audio-tagging-2019, inaturalist-2019, iwildcam-2019; the floor and the CWM row had valid submissions on the first two, so leaving them out favours the ceiling). The 53 competitions outside Lite were never used for error analysis or any choice: the first clean test of the method.