mle-rubrics-onpolicy-1000

### No held-out validation against the competition's stated metric before submitting
- **Applies when**: `task` -- the task specifies an explicit evaluation metric (e.g., log loss, AUC, RMSE) for predictions on an unlabeled test set, and the script trains models and writes a submission file.
- **Pattern**: The script fits one or more models on 100% of the labeled data, blends them with hand-picked weights, and writes predictions straight to disk — never computing the stated metric on a validation split, cross-validation folds, or even against a trivial baseline (e.g., class-prior probabilities). Hyperparameters, imputation choices, and ensemble weights are therefore unjustified, and there is no evidence the output is better than random or than a constant prediction.
- **Detection procedure**:
  1. Read the task and note the exact scoring metric and its inputs (probabilities vs. labels, per-row normalization, clipping).
  2. Scan the script for any train/validation split, `cross_val_score`/`KFold`, or explicit computation of that metric on labeled data; check whether printed diagnostics include a metric value at all.
  3. Check whether model/ensemble choices (weights, depth, learning rate) are tied to any measured score, or are literal constants written by the author.
  4. Inspect the answer/submission: is any score, baseline comparison, or sanity check (row count equal to test rows, ids matching test ids, probability ranges/sums, no degenerate constant column) reported?
- **Discriminator**: A real violation is when *no* estimate of the stated metric exists anywhere for any candidate model, so the attempt cannot distinguish a good submission from a broken one. It is *not* a violation if the script measures the metric via CV/holdout (even briefly) and uses it to pick among options, nor if the metric is unmeasurable because labels genuinely do not exist for any subset.
- **Consequence**: The submission may be systematically miscalibrated, mis-ordered, mislabeled by class column, or simply far worse than a simple baseline; the grader reports a failing/incorrect submission with no diagnostic trail, since the agent had no internal score to catch it.

### Ships model predictions with no held-out validation and no distribution sanity check
- **Applies when**: `task` -- a script trains a regressor/classifier on one file and writes predictions for another file as the deliverable, with no ground truth available for the predicted rows.
- **Pattern**: The attempt fits a single model on 100% of the training rows, immediately predicts on the target rows, and writes the output without (a) any hold-out/CV error estimate, (b) a comparison of the predicted value distribution against the training target distribution, or (c) a check that the feature columns/dtypes/encodings used at prediction time are the same as those learned at fit time. Degenerate output (e.g. the vast majority of predictions collapsed to the same value, or a range far narrower than the target's) is accepted as-is.
- **Detection procedure**:
  1. Read the task: identify the deliverable (file, required column names/order, row alignment, rounding/units) and note that no labels exist for the predicted rows.
  2. Read the script: check whether a train/validation split or cross-validation with a printed error metric exists, and whether the same column names, imputation, and category encodings are applied consistently to both datasets (watch for near-identical but non-identical column spellings, or one file's schema being assumed for the other).
  3. Read the script's output stage: check whether the predicted values' summary statistics are compared to the training target's summary statistics, and whether any explicit guard rejects a degenerate/off-scale prediction vector.
  4. Inspect the submitted answer: compute the share of identical values and the min/max; if predictions are dominated by a single value or their spread is an order of magnitude off the target's documented spread, and no validation metric was reported, flag it.
- **Discriminator**: A real violation is an unvalidated pipeline whose output is visibly degenerate or whose feature handling silently differs between fit and predict; it is *not* a violation if the script reports a hold-out/CV score and the prediction distribution plausibly matches the training target (a skewed target legitimately yields many small values, provided the reported validation error supports it).
- **Consequence**: The written file passes format checks but its values are near-constant/mis-scaled, so the grader's accuracy or error tolerance against the reference targets fails, marking the deliverable WRONG.

### Hypothesis test run on the full table instead of the task-specified population/subset
- **Applies when**: `task` -- the task asks for a statistical test, metric, or comparison whose population is qualified by filters (date range, category/tournament type, subgroup, one-sided direction) stated in the prompt, README, or problem context.
- **Pattern**: The attempt loads the raw files and computes the statistic on every row of both groups, silently dropping the qualifying conditions (and/or defaulting to a two-sided test when a directional hypothesis was specified), so the reported p-value/metric describes a different population than the one asked about.
- **Detection procedure**:
  1. Read the task and any README, and list every explicit or implied restriction on the rows/columns to be analyzed (time window, subset of categories, groups compared) plus the alternative hypothesis direction and significance level.
  2. Read the scripts and check that each restriction appears as a concrete filter/dtype conversion (e.g., date parsing then range filter, category equality filter) before the statistic is computed, and that the test call matches the stated alternative and test type (paired vs independent, equal-variance assumption).
  3. Compare the row counts used in the test against the raw file row counts; if the script never prints or reduces counts, treat the population as unverified.
  4. Check the reported statistic's magnitude for plausibility given the intended (usually much smaller) subset — extreme p-values (e.g., 1e-100 or smaller) usually signal a far larger n than intended.
- **Discriminator**: A real violation is a missing or incorrect filter/direction that changes which rows enter the computation; a look-alike that is fine is a script that applies the filters in a different but equivalent way (e.g., filtering at load time, using a query string) and can show the reduced counts/subset consistent with the task description.
- **Consequence**: The reported p-value (and possibly the reject/fail-to-reject decision) is computed on the wrong sample, so it will not match the expected value in the output file and the check fails even though the file format is correct.

### Output template file never actually inspected
- **Applies when**: `task` -- the task says to write results into an output file "following the exact structure/format of" a provided sample/template file, and the scripts build that output.
- **Pattern**: The attempt never loads or prints the sample/template file; it hard-codes guessed column names, column order, row ordering, rounding/units and header text from the prose of the task, then asserts in the answer that the output "matches the required format."
- **Detection procedure**:
  1. In the task, note that a reference/sample output file is supplied and is the authority on schema and formatting.
  2. Search the scripts for any read/print/comparison of that sample file (e.g., loading it, checking its columns, dtypes, row count, decimal places, sort order).
  3. If absent, check whether the emitted frame's column names, column order, sort order and numeric formatting are instead invented in code or copied from the task wording.
  4. Check the answer for unverified claims of format compliance (no printed diff against the template).
- **Discriminator**: A real violation is when no code path ever reads the template, so agreement with it is pure luck; it is fine if the script reads the template (or explicitly reindexes/renames/rounds/sorts to the template's columns and formatting) and prints a shape/column/dtype comparison, even if the final naming happens to be hard-coded afterwards.
- **Consequence**: The graded file is judged WRONG/MISSING on a strict file comparison — mismatched header names or order, wrong row ordering, or unrounded/differently scaled values — even when the underlying aggregation logic is right.

### Unverified output artifact: schema/content of the saved file never checked against the requested spec
- **Applies when**: `task` -- the task requires writing results to a named file with an explicitly specified column naming scheme and one row per input record, and the scripts generate that file programmatically.
- **Pattern**: The attempt builds a feature matrix after silently dropping/imputing/encoding columns, fits a model, writes the file, and then reports a prose summary (counts, chosen hyperparameter, feature legend) without ever re-reading the written file to confirm it exists at the expected path and that its header names, column count, row count, and label values match the requested format; degenerate or minimal-complexity results (e.g., the smallest possible number of groups) are accepted without a sanity check.
- **Detection procedure**:
  1. From the task, list the required artifact path, the exact column-naming convention, the expected number of rows (records) and the expected value semantics of the label column.
  2. In the scripts, locate the write call and check whether the DataFrame passed in has columns generated to match the convention exactly (no index column, no leftover original names, no extra/missing feature columns relative to the matrix actually clustered) and whether every input record survives preprocessing (no silent row drops from NaN handling or filtering).
  3. Check for a read-back/assert step after writing: does any code load the file and print/verify shape, header, row count, and label distribution? Compare that to what the final answer claims.
  4. Inspect the reported result for degeneracy or implausibility (a single dominant group, the minimum possible number of clusters, row count ≠ dataset size) that no validation step addressed.
- **Discriminator**: A real violation is when the answer's claims about the file are asserted from in-memory variables or narrative only, with no post-write verification and no shape/format assertion — or when preprocessing changed the row/column set without reconciling it to the spec. It is *not* a violation if the script (or a follow-up run) reloads the artifact and asserts the header pattern, row count equal to the number of input records, and valid label values, even if the modeling choices are debatable.
- **Consequence**: The grader reads the artifact and finds it missing, misnamed, mis-headered, or with the wrong number of rows/columns (or a degenerate labeling), scoring the file check WRONG/MISSING despite a confident-sounding summary.

### Submission artifact never validated against the provided template (path, columns, ids, dtype)
- **Applies when**: `task` -- the task requires writing a prediction/result file whose format is defined by a sample/template file, and the scripts generate that file at a hard-coded location.
- **Pattern**: The attempt builds the output frame from its own assumptions (its own column names, its own output directory, its own value dtype such as forcibly rounding/casting continuous predictions), prints a few rows as "verification", and never programmatically loads the template to confirm the file is written where the task expects, with the same column names/order, the same number and set of identifiers, and value types consistent with the evaluation metric.
- **Detection procedure**:
  1. In the task/README, note the required output filename, its expected location relative to the working directory, and the template file that defines its schema.
  2. In the scripts, find every write of the output file: check the path string, the constructed column names/order, and any post-processing of predictions (rounding, clipping, int casting, sorting).
  3. Check whether any script actually reads the template and asserts equality of columns, row count, and id set/order against the written file — printing `head()` or shapes without comparison does not count.
  4. In the answer, check whether it states the verified output location and schema match, or only narrates modeling choices and CV scores.
- **Discriminator**: A real violation is when no code compares the produced file to the template/required path, or when values are transformed in a way the metric does not ask for (e.g., integer rounding of a continuous/log-scale target). It is fine if the script asserts column equality, id alignment and row counts against the template (even implicitly by copying the template and overwriting the prediction column) and writes to the location the task specifies.
- **Consequence**: The grader reports the expected result file as missing or wrong (not found at the expected path, mismatched columns/ids, or degraded score from unnecessary value transformation), so the attempt scores 0 despite a plausible-looking model and CV metrics.

### Dropping rows with missing values instead of imputing, shrinking the required output
- **Applies when**: `task` -- the task asks for a per-record output file (labels, predictions, scores) covering a dataset that contains missing values or mixed dtypes.
- **Pattern**: The script handles missing data with a blanket `dropna()` (or restricts to rows/columns that are fully populated), so the produced file contains far fewer records than the input dataset, and the answer reports the reduced count as if it were the full result.
- **Detection procedure**:
  1. Read the task for the required output granularity — does it implicitly require one row per input record, and does it state any filtering? If no filtering is stated, every input record must appear.
  2. In the script, look for `dropna`/subsetting before the model fit and for whether the output is written from the reduced frame.
  3. Compare the row count claimed in the answer (or in the written file) against the raw dataset's row count; also check the number of feature columns against the number of usable numeric columns.
  4. Flag if rows were silently lost and no imputation (mean/median/etc.) or justification was applied.
- **Discriminator**: A real violation is unrequested row loss that changes the output's coverage; it is fine if the task explicitly asks to filter/subset, or if only a couple of records are dropped for a documented reason **and** the task does not require complete coverage.
- **Consequence**: The output file has the wrong shape/row count and cluster labels that cannot be aligned to the expected per-record results, so the file comparison fails outright.

### Output schema deviation: extra/renamed columns in the required result file
- **Applies when**: `task` -- the task specifies an exact output file with an exact set/naming of columns, and the script builds that file from an intermediate DataFrame.
- **Pattern**: The script writes the file but adds identifier/bookkeeping columns, keeps original names, mis-indexes the required suffix numbering, or otherwise emits a superset/variant of the requested schema instead of exactly the stated columns (and does not assert the schema before saving).
- **Detection procedure**:
  1. From the task statement, write down the exact required column list (names, naming convention, and whether anything else is allowed) and any index/ordering requirement.
  2. In the script, trace the DataFrame that is passed to the write call: list every column added (`insert`, `copy` of source columns, `reset_index`) and every rename mapping, plus the `index=` argument.
  3. Compare that final column list to the required list; also check the numbering convention starts/increments as the task implies and that no ID/label leftovers survive.
  4. Check the answer/verification output: does it print the saved file's `columns.tolist()` and compare to the spec, or merely print a head and declare success? Also check the answer's described pipeline matches the script actually run.
- **Discriminator**: A real violation is a saved file whose column set differs from the specification (extra column, unrenamed column, off-by-one naming, index written as a column). A look-alike that is fine: extra columns exist only in in-memory/intermediate frames or in separate diagnostic files, while the required file contains exactly the specified columns.
- **Consequence**: The grader reads the result file, fails the schema/column check (or mis-aligns the feature vector), and marks the expected file WRONG/MISSING regardless of the clustering quality.

### Validation split and feature set that don't mirror the actual prediction setting
- **Applies when**: `task` -- a model must produce predictions for a supplied test file whose rows come from an ordered (e.g., time-indexed) process, and the scripts choose which of the shared columns to use as features and how to hold out data for validation.
- **Pattern**: The attempt builds features by blanket-excluding columns (dropping some that exist in *both* train and test and are strongly related to the target) and then measures quality with a random/shuffled train-test split of ordered data, reporting the resulting optimistic error/R² as evidence the submission is good — without ever checking predictions against a simple baseline or against the target's own distribution on a held-out block that resembles the test rows.
- **Detection procedure**:
  1. From the task and test file schema, list the columns actually available at prediction time; then read the scripts' feature-selection code and note any available column that is excluded or silently dropped (e.g., via an `exclude` list, `select_dtypes`, or all-NaN filtering) despite being predictive of the target.
  2. Inspect the holdout logic: does it use `train_test_split` (shuffled) / random CV on rows that are sequential in time, rather than a chronological or block split matching the test period?
  3. Check whether the answer's reported metric is the only evidence of correctness — i.e., no baseline comparison (persistence/mean/available forecast column) and no sanity check that predicted distribution (mean, range, count, ordering) matches the training target and the expected output rows.
  4. Flag if (1) or (2) holds and (3) holds.
- **Discriminator**: A real violation excludes usable, task-legitimate predictors and/or validates in a way that leaks temporally adjacent rows, so the quoted metric cannot be trusted; a look-alike that is fine excludes only columns genuinely absent/unusable in the test file (true leakage or all-missing), and validates on a chronological holdout that reproduces the test-time information set, with a baseline comparison reported.
- **Consequence**: Validation metrics look strong (low MAE, high R²) while the submitted predictions are systematically off on the real test rows, so the graded file fails the accuracy/tolerance check despite the correct file name and row count.

### Incomplete deliverables for a spec-driven plotting task
- **Applies when**: `task` -- the task points to an external format spec (e.g. a YAML/JSON config) and grading depends on machine-readable artifacts (plot data/series arrays) in addition to a rendered image.
- **Pattern**: The attempt produces only the image and narrates that it "follows the spec", without saving the accompanying data/spec artifacts the grader expects, and without keeping reproducible scripts; internal inconsistencies (e.g. a title naming one date range while the described data covers another) are left unresolved.
- **Detection procedure**:
  1. Read the task and any referenced spec file; enumerate every required output artifact (image, serialized plot spec, numeric array of plotted values) and every named property (figsize, color, title, axis labels, ticks, ordering, aggregation level).
  2. Check the working directory / scripts for each enumerated artifact actually being written, and check that a script exists that reproduces them from the raw data.
  3. Cross-check the answer's claimed properties against the spec verbatim (string equality of title/labels, tick list, series length) and against the described data range for contradictions.
  4. Verify the plotted series is derived with the stated granularity and ordering (e.g. correct time aggregation, sorted x-values, no dropped or duplicated periods) rather than asserted.
- **Discriminator**: A real violation is missing required artifacts or a property that mismatches the spec text (including inconsistent date ranges/labels); it is fine if all required files are written and the only differences are cosmetic extras (markers, gridlines, legend) not constrained by the spec.
- **Consequence**: Grader checks on the expected data artifacts report WRONG/MISSING and the attempt scores 0 even though an image exists.

### Missing required output artifact (and fabricated fallback data instead of the real inputs)
- **Applies when**: `task` -- the task names a specific output file/format (e.g., a result file matching a provided sample) and points to input data that the scripts must locate and load.
- **Pattern**: The scripts implement the statistical/modeling logic but never write the requested file; input loading is done by guessing candidate paths with a silent fallback that synthesizes or hard-codes plausible data, so the reported numbers come from invented inputs and the deliverable file is absent. The final answer is a prose/console report instead of the specified artifact.
- **Detection procedure**:
  1. From the task, list every required deliverable: exact output filename, expected columns/rows/ordering, rounding/units, and the exact input file(s) to be used.
  2. Grep the scripts for any write of that deliverable (`to_csv`, `open(..., 'w')`, etc.) and check the written schema against the provided sample; also check whether the sample file was read/inspected at all.
  3. Trace the data-loading path: does it read the actual provided file(s), or try a list of guessed paths / `except: pass` / `if df is None:` branch that generates random or literal numbers? Confirm the script would fail loudly rather than proceed on fake data.
  4. Compare the reported numbers to the branch that could have produced them (e.g., suspiciously round sample sizes, seeded `np.random.normal` values, a p-value of exactly 0) to see whether the answer came from the fallback.
- **Discriminator**: A real violation is when no code path writes the named file with the sample's schema, or the reported values could only come from synthetic/hard-coded inputs. It is fine if the script robustly searches for the input but raises on failure, and does write the deliverable with the required columns/precision — even if the analysis logic is only in a helper module or the console also prints a summary.
- **Consequence**: The grader finds the expected result file missing or containing values derived from invented data, so all output checks fail regardless of whether the statistical method was correct.

### Ignoring referenced specification files and required output artifacts
- **Applies when**: `task` -- The task points to auxiliary instruction/config files (e.g., a tips/README/spec text file, a YAML/JSON formatting config) and/or expects a set of deliverable files beyond the obvious main output.
- **Pattern**: The scripts never open, parse, or echo the referenced instruction/config files; the agent instead hardcodes its own assumptions (grouping definitions, columns, titles, colors, axis labels, figure size, ranges) and writes only the single most obvious artifact, then declares success by paraphrasing the spec it never actually read.
- **Detection procedure**:
  1. List every external file the task text references (instruction files, config/format files) and every output file implied or named.
  2. Grep the scripts for reads of each referenced file (`open`, `read_csv`, `yaml.safe_load`, `json.load`) and for writes of each expected output; note any file mentioned in the task but absent from the code.
  3. Check whether formatting/derivation choices in the plotting or aggregation code trace back to values loaded from the config, or are literals invented by the agent.
  4. Compare the answer's claims (e.g., "formatted according to the spec", "definitions from the tips file") against step 2 — claims about content of unread files are unsupported.
- **Discriminator**: A real violation is code that contains no read of a task-referenced file, or omits an expected artifact entirely. It is *not* a violation if the script reads the config and then legitimately falls back to defaults for keys the config doesn't specify, or if the extra artifacts are written by a separate step visible elsewhere in the run.
- **Consequence**: Graders that compare each required artifact (image, serialized plot spec, numeric array) find them missing or mismatched in styling, labels, series definitions, or data values, so all checks fail even if the underlying computation looks plausible.

### Spec file referenced by the task is never read or implemented
- **Applies when**: `task` -- the instructions point to an external document (README/spec/config) that defines how to bin, filter, map, or order values before producing the requested output.
- **Pattern**: The agent skips or paraphrases the spec, reuses the raw categories/values already present in the data (or its own invented grouping), and asserts in the answer that "the spec was followed" without any script step that opens the spec file or encodes its rules; often no reproducible script is saved at all.
- **Detection procedure**:
  1. From the task, list every referenced auxiliary document and the exact rules it is supposed to supply (bin edges, labels, exclusions, ordering, required output artifacts).
  2. Search the scripts for a read of that document and for an explicit mapping/binning structure (dict, `pd.cut` bins, list of labels) that materializes those rules; if no script exists, treat the derivation as unverifiable.
  3. Compare the category labels/counts in the answer against the raw distinct values of the source field: if they are identical (or the totals equal the unfiltered row count when the spec implies grouping/filtering), the spec was not applied.
  4. Check that every artifact the task/spec implies (plot plus any data dumps) is actually written by the script, not just described in prose.
- **Discriminator**: A genuine violation is when no code path encodes the spec's rules and the output categories coincide with the raw field's own levels; it is fine if the code hard-codes bins that are demonstrably transcribed from the spec (labels/edges match) even without re-reading the file at runtime.
- **Consequence**: Saved outputs (image and any numeric/JSON dumps) have wrong bin labels, wrong bin counts, or are missing entirely, so every file-level check fails despite a confident "task completed" report.

### Missing reproducible script and required output artifact
- **Applies when**: `task` -- the task states a specific preprocessing step and a specific answer format/output file, and the agent supplies a final answer with no (or non-runnable) script that produces it.
- **Pattern**: The agent reports a value it believes it read off the data (or recalls), without a saved, end-to-end script that loads the raw file, coerces the relevant column to numeric (stripping symbols/thousand separators/percent signs), applies the stated imputation, computes the requested extremum/statistic, and writes the result to the exact requested artifact (e.g. the named JSON/CSV) in the exact requested schema. Nothing in the deliverables lets a reviewer re-derive the number, and the required file is never created.
- **Detection procedure**:
  1. List every deliverable the task demands: the output file name/location, the key names, the value types, and any mandated preprocessing step.
  2. Look for a script whose code path visibly does each of those: raw load → dtype cleaning of the target column → the mandated imputation → the selection/aggregation → serialization to the named file.
  3. If any link is absent (no script at all, or the script prints to stdout only, or never touches dtypes/imputation), mark inadequate; also check the reported number is a plausible value of the stated unit (e.g. a percentage inside 0–100) and that it belongs to the row that a numeric — not lexicographic — comparison would select.
  4. Confirm the emitted structure matches the requested shape exactly (same keys, list-vs-scalar, rounding).
- **Discriminator**: A real violation is when the answer cannot be traced to executed code that both preprocesses as instructed and writes the named artifact. A look-alike that is fine: a short but complete script that does all steps and dumps the file, even if the agent additionally quotes the answer in prose.
- **Consequence**: The grader finds the expected result file missing or containing a value derived from uncleaned/unimputed data (e.g. a string-max or a pre-imputation row), so the check fails with 0/1 even though the prose answer looks well-formatted.

### Named statistic implemented with the wrong formula variant
- **Applies when**: `task` -- The task names a specific, textbook-named statistic, coefficient, or metric variant (e.g. a particular "first/second coefficient", a specific averaging or normalization convention) that the script must compute by hand.
- **Pattern**: The script hard-codes one plausible variant of the named quantity (or calls a library default) without checking that it matches the exact named definition, often with a comment that asserts the mapping rather than verifying it — e.g. implementing the median-based sibling formula while labeling it as the mode-based one, or using sample vs. population normalization inconsistently with the definition.
- **Detection procedure**:
  1. From the task statement, write down the exact canonical definition of the named statistic, including which central-tendency/normalization terms it uses.
  2. Read the script's arithmetic expression (not its comments or variable names) and map each term to the canonical definition.
  3. Check whether an alternative, similarly named variant exists that the script may have substituted; if so, require evidence in the script that both were computed/compared or that the chosen one was justified.
  4. Confirm downstream classification/rounding uses the value from the correct formula.
- **Discriminator**: A real violation is when the coded expression is a *different* named quantity (different terms, different denominator convention) than the one requested; a look-alike that is fine is an algebraically equivalent rewriting, or a negligible convention choice (e.g. ddof) that provably cannot change the reported rounded value or the qualitative conclusion.
- **Consequence**: The qualitative label (sign/direction/category) may still match by luck, but the numeric field differs from the reference, so the answer fails the value check while passing the type check — partial credit at best.

### Circular "verification" that re-runs the same code instead of validating input assumptions
- **Applies when**: `task` -- a script aggregates several raw columns row-wise (weighted sums, means, cumulative products) and a second "verification" script is used to confirm the saved output.
- **Pattern**: The attempt reads the raw file with no inspection of missing values, dtypes, units/scale, or a leading initialization row, aggregates with a reducer that silently ignores NaNs (e.g. `.sum(axis=1)`, `.mean()`), then "verifies" by recomputing with the identical code path and reporting a match — so any shared assumption error (NaNs treated as zero, percent vs. decimal returns, dropped/misaligned column, wrong starting baseline or compounding convention) is confirmed rather than caught.
- **Detection procedure**:
  1. In the task/README, list the assumptions the computation depends on: units/scale of the input values, expected number of rows/columns, and the exact definition and starting point of the requested output quantity.
  2. In the scripts, check whether the raw input is ever validated — `isna().sum()`, dtype check, min/max/scale check, row/column count check, or explicit NaN policy in the aggregation — and whether the aggregation is done with an operator that skips NaNs by default.
  3. Check whether the verification script does anything independent (recompute one row by hand from raw numbers, compare against a known benchmark, check output ranges/row counts/column names against the required format) or merely re-executes the same formula.
  4. Inspect the answer for tell-tale signs: implausibly smooth/small magnitudes, no NaN or zero at the series start, row count differing from the source file, or values that would be off by ~100x under a unit mismatch.
- **Discriminator**: A real violation is when *no* check exists that could fail independently of the main computation (data-quality assertions absent and the verifier duplicates the formula). It is fine if the script asserts input completeness/scale, handles NaNs explicitly, or cross-checks at least one output value against an externally derived or hand-computed reference — even if the code is otherwise reused.
- **Consequence**: All internal checks "pass" while the saved file diverges from the reference (shifted baseline, NaN-as-zero contamination, wrong scale or row alignment), and the grader marks the output file WRONG on numeric comparison.

### Dict/collection answers emitted without literal-syntax fidelity (unquoted keys, altered delimiters)
- **Applies when**: `task` -- the task prescribes an exact answer template containing a structured literal (dict/list/tuple) with string keys or labels, e.g. `@name[{'k_1':v_1, ...}]`.
- **Pattern**: The agent computes correct values but serializes the container by printing a Python object, f-string, or hand-typed text that drops the quotes around keys, swaps bracket/brace types, omits the `name=` prefix, or otherwise deviates from the template, so an exact/parse-based grader cannot match it even though the numbers are right.
- **Detection procedure**:
  1. Copy the answer template from the task statement verbatim and note every literal character: variable name, `=` or `[]`, brace type, quoting of keys, separators, and key naming scheme.
  2. In the scripts, find the line that builds the final answer string and check whether it uses a faithful literal serialization (e.g. `repr()`/`print(dict)` with string keys, or a hard-coded template) rather than a loop/f-string that strips quotes.
  3. Compare the submitted answer token-by-token against the template; flag any mismatch in quoting, key spelling/order, brackets, or prefix.
  4. Optionally, attempt to parse the submitted payload with `ast.literal_eval`; if it raises or yields different key types than the template, flag it.
- **Discriminator**: A real violation is a syntactic/format deviation from the stated template (unquoted or renamed keys, wrong bracket, missing prefix, wrong ordering, wrong rounding presentation). A look-alike that is fine is cosmetic whitespace or trailing-zero differences that still parse to the same literal object with the same key strings and values.
- **Consequence**: The grader reports the expected mapping as WRONG/MISSING and scores 0 despite numerically identical values.

### Missing evidence that the statistic was computed on the correctly cleaned data vector
- **Applies when**: `task` -- the task asks for a distributional statistic or hypothesis test on a single numeric column (normality test, skewness, kurtosis, moments) and prescribes a specific test, threshold, or value to report.
- **Pattern**: The attempt reports only the final verdict/numbers with no saved script and no intermediate diagnostics, so the vector actually fed to the test is unverifiable: NaNs, missing-value sentinels (e.g. -999, 0, blanks), strings coerced to numbers, or extra rows outside the intended subset can silently enter and dominate the moments, flipping the test outcome.
- **Detection procedure**:
  1. Read the task and list every required output and diagnostic (e.g. the test name, alpha, the p-value to be reported, rounding rules).
  2. Inspect the scripts for an explicit, reproducible chain: load → select column → coerce dtype → drop/handle missing and sentinel values → print n before and after cleaning → run the prescribed test → print p-value, skewness, kurtosis.
  3. Check the answer against these prints: is the reported p-value present, is the verdict consistent with the stated alpha, and are the moment values plausible for the printed n and value range?
  4. Flag if any script is missing, if n and cleaning steps are never printed, or if a required diagnostic (p-value) is absent from the report.
- **Discriminator**: A genuine violation is an answer with no reproducible cleaning/diagnostic trail, or one whose extreme moment values (e.g. large |skew|, heavy kurtosis) are asserted without any check for sentinels/outliers that would explain them; it is *not* a violation if the script prints row counts, shows the column is clean numeric, reports the p-value, and the extreme moments are corroborated by printed summary statistics.
- **Consequence**: The test is run on a contaminated or wrong-length vector, so the normality verdict and the skewness/kurtosis values differ from ground truth and every graded field fails, with no artifact left to diagnose the discrepancy.

### No held-out validation of predictive quality before submitting predictions
- **Applies when**: `task` -- the task asks for predictions on an unlabeled evaluation file and the scripts train a model on a labeled training file, with grading presumably based on prediction quality against hidden labels.
- **Pattern**: The attempt fits a single default/lightly-tuned baseline (e.g., one vectorizer + one simple classifier with hand-picked hyperparameters), never splits off a labeled validation set or runs cross-validation, and reports only formatting facts (row count, column name) and the predicted class distribution as evidence of success — so no one knows whether the model clears the accuracy bar the grader uses.
- **Detection procedure**:
  1. Read the task: confirm the deliverable is per-row predictions judged against hidden ground truth, not just a file with the right shape.
  2. Read the scripts: look for any train/validation split, cross-validation, or scoring call on labeled data; also check whether more than one model/feature setting was compared.
  3. Read the answer: check whether it states a measured quality estimate (accuracy/F1 on held-out labeled data) rather than only shapes, column names, and label frequencies.
  4. If steps 2–3 find no measured score, flag: the attempt has no evidence its predictions are better than chance-level or an unaccepted baseline.
- **Discriminator**: A real violation is the total absence of any quantitative held-out estimate (or a score computed on the same rows used for training, which is equally uninformative). It is *not* a violation if the attempt reports a legitimate held-out/CV score and simply chose a simple model, nor if the task explicitly only checks file format; comparing predicted vs. training label distributions alone is not a substitute for a score.
- **Consequence**: The output file has the correct shape and column name but low agreement with the hidden labels, so an accuracy-threshold check marks the result file WRONG while the agent's self-report claims success.

### Declaring success on format checks alone, with no held-out performance validation
- **Applies when**: `task` -- the task asks for predictions on a held-out set written to a submission file, and the scripts fit a model and write the file.
- **Pattern**: The attempt validates only superficial properties of the output (row count, column names, value range, no NaNs, mean roughly matching the target mean) and reports model hyperparameters as evidence of quality, while never computing the competition-relevant metric on a held-out/validation split, never comparing to a trivial baseline (e.g., predicting the target mean), and often silently training on a truncated subsample of the available rows or a reduced feature set.
- **Detection procedure**:
  1. Read the task to identify the evaluation target and the implied scoring metric (e.g., regression error/rank correlation/AUC) and whether all training rows/features are available.
  2. In the scripts, look for (a) a train/validation split with the metric computed on the validation part, (b) a comparison against a constant/naive baseline, and (c) whether the fit uses the full training data and the same feature set as the test-time transform.
  3. In the answer, check whether any quantitative out-of-sample score is reported, or whether only distributional/format statistics and hyperparameters are cited as "good".
  4. Compare the reported spread of predictions to the spread of the training target; a much narrower prediction range with no validation score is a red flag for severe underfitting/undertrained model.
- **Discriminator**: A real violation reports no out-of-sample metric at all (or only format/range checks) so predictive quality is unknown; a look-alike that is fine reports a validation score computed on data excluded from fitting, ideally alongside a baseline score, even if the final answer text also mentions format checks.
- **Consequence**: The submission file is well-formed but scores at or near a naive baseline (or below the grader's accuracy threshold), so the correctness check on the predictions fails despite all self-reported checks passing.

### Ambiguous or unverified subgroup boundaries when a statistic is requested per range-defined bin
- **Applies when**: `task` -- the task asks for a metric/statistic computed separately for groups defined by numeric ranges (below X, between X and Y, above Y) of some column.
- **Pattern**: The attempt picks one plausible binning without making inclusivity explicit or checking it — e.g. treating "between X and Y" as an open interval that drops rows exactly at the boundaries (or as closed, double-counting them), grouping on a column whose values are strings/half-steps so comparisons or matches silently misclassify rows, and never printing per-group row counts so that the groups do not sum to the filtered population.
- **Detection procedure**:
  1. Read the task and list the requested groups and their boundary values; note that boundary rows are typically a large share of a coarse rating/score scale.
  2. In the scripts, locate the filtering/binning code: check the comparison operators (`<`, `<=`, `between`), the dtype of the grouping column (numeric vs string/categorical), and whether the boundary categories are assigned to exactly one group.
  3. Verify the script prints, per group, the row count after dropping nulls in both correlated columns, and that these counts sum to the total non-null population of the filtered set (no dropped or duplicated boundary rows).
  4. Check the answer: if the middle/edge group's value is implausibly close to a neighbouring group's value, or group sizes were never reported, treat the binning as unvalidated.
- **Discriminator**: A real violation is a binning where boundary-valued rows are excluded or misassigned, or where group counts are never shown; it is fine if the script documents the chosen convention, assigns every row in the filtered population to exactly one group, and reports counts that reconcile with the total (even if the convention is debatable, the reconciliation exposes it).
- **Consequence**: Groups adjacent to the boundaries are computed on the wrong subset, so their correlation coefficients differ materially from the reference values while the unambiguous group matches — a partial-credit failure like 1/3 checks passed.

### Invented qualification thresholds / unverified output spec when the task's definition is ambiguous or truncated
- **Applies when**: `task` -- the task references an explicit definition, eligibility rule, or a provided sample/template output file, and part of that specification is incomplete, truncated, or not obviously reproduced in the scripts.
- **Pattern**: The attempt guesses a cutoff (e.g., a made-up minimum count applied on a self-chosen basis such as per-item mean instead of total), applies that same filter to every sub-ranking regardless of whether the rule pertains to it, and never opens the supplied sample/template to confirm column names, ordering, key type, and row conventions.
- **Detection procedure**:
  1. Read the task/README and list every explicitly stated rule (qualification criteria, aggregation level, rounding, units, ordering) and every referenced auxiliary file (sample/template, dictionary, docs).
  2. Search the scripts for code that loads or prints each referenced auxiliary file and for constants that encode the stated rules; flag any numeric threshold or filter that appears without a documented source.
  3. Check whether each requested output quantity is computed with the definition scoped to it, i.e. whether the eligibility filter is applied only where the specification says it applies, and on the stated statistic (sum vs. mean vs. count).
  4. Compare the emitted file's header/ordering/index against the sample file's actual contents (not an assumed layout); flag if never compared.
- **Discriminator**: A real violation is a threshold, aggregation choice, or output layout that the scripts never derive from the data, README, or sample file — it is hard-coded on intuition. It is fine if the agent inspects the sample/spec, documents the source of the rule, tests the sensitivity of the ranking to plausible alternative interpretations, or the rule genuinely has a single unambiguous reading in the provided text.
- **Consequence**: The rankings are computed on a differently filtered population (and/or with mismatched column names/ordering) than the reference, so exact-match file comparison fails on all rows even though the underlying aggregation code is arithmetically correct.

### Spec file / required artifacts asserted rather than actually read and produced
- **Applies when**: `task` -- the task points to an external specification file (config/YAML/JSON) for output formatting or binning and/or implies a set of output artifacts beyond the headline file.
- **Pattern**: The attempt hard-codes its own titles, labels, bin edges, styling and category set, describes them in the final answer as if they came from the spec, and emits only the single obvious output file while silently skipping the other expected artifacts (serialized plot data, saved arrays); the derived group counts are never checked against the total row count of the filtered subset.
- **Detection procedure**:
  1. From the task text, list (a) every file that must be *read* as a specification and (b) every artifact that must be *written*.
  2. In the scripts, confirm the spec file is actually loaded and its values are used to drive the plot (bins, labels, title, order, figure params) instead of literals appearing in the code; confirm each expected artifact is written by an explicit save call.
  3. Compare the answer's stated categories/counts with the spec's own categories and with the dataset: does the number of groups match the spec, and do the group counts sum to the size of the filtered subset (and to a plausible fraction of the raw file)?
  4. Flag if any spec value is invented, any artifact is unwritten, or the counts fail the sum/magnitude sanity check.
- **Discriminator**: A fine attempt reads the spec and its literals coincide with the spec values (verifiable by comparing code/output to the file contents) and writes all requested artifacts; a violation is code whose formatting/binning choices exist nowhere in the spec, or an answer that claims "all specifications applied" without the spec ever being parsed, or an obviously truncated subset count with no reconciliation.
- **Consequence**: The grader checks each expected artifact (plot metadata, image, saved array); missing files and mismatched bins/counts make every check fail even though the script ran without error.

### Stated output ordering/serialization constraint not enforced in code
- **Applies when**: `task` -- the task specifies how the final answer must be ordered/formatted (e.g., "sort from highest to lowest") and/or written to a named result file, and the scripts produce ranked lists or dictionaries.
- **Pattern**: The scripts extract the right records but leave them in whatever order the selection helper returns (e.g., an ascending "smallest-n" query for the low group while the high group is descending), and only print results instead of emitting the requested artifact/JSON structure — so the stated ordering/format constraint is never explicitly applied.
- **Detection procedure**:
  1. Read the task and list every explicit output constraint: ordering direction, grouping keys, rounding/units, and the required file name/JSON schema.
  2. In the scripts, locate each list-producing step and check whether an explicit sort (or reverse) with the required direction is applied to *every* output list, not just some of them; check whether the final structure is serialized to the required file.
  3. Compare the printed/reported lists against the constraint: for a "highest to lowest" requirement, verify the associated values decrease monotonically in each list.
  4. Flag if any list's implied value order contradicts the requirement, or if no code writes the required output artifact.
- **Discriminator**: A real violation is when the code contains no ordering/serialization step matching the stated requirement (or the reported values run the wrong way); it is fine if a selection helper already returns the required direction and the reported values verifiably follow it, and the required file is written with the requested keys.
- **Consequence**: The grader compares the expected result file element-by-element and marks the answer wrong (or missing) even though the correct set of records was identified.

### Answer string doesn't literally match the requested output template (quoting/delimiters/tokens)
- **Applies when**: `task` -- The task specifies a rigid answer template with tagged fields (e.g. `@field[value]`) and shows the exact literal form the values should take (quoted strings, units, ranges, allowed vocabulary).
- **Pattern**: The agent computes plausible or even correct values but emits them in a different surface form than the template demands — dropping the quotation marks shown in the spec, changing separators or spacing, reordering fields, adding extra prose/units, or paraphrasing an allowed label — so a literal string matcher scores every field wrong. This is often compounded by no saved script, so nothing else in the submission can be checked or re-derived.
- **Detection procedure**:
  1. Copy the answer-format line from the task verbatim and list each field, its delimiters, and exactly how the sample value is written (with or without quotes, capitalization, allowed token set).
  2. Copy the agent's final answer string and diff it character-by-character against that template, field by field and in order.
  3. Check that each value is drawn from the task's stated vocabulary/format (e.g. the literal range string, the enumerated category labels) rather than a synonym, computed variant, or extra-annotated form.
  4. Confirm a script exists that produces those exact strings (or that the answer is at least reproducible), rather than being hand-typed with no artifact.
- **Discriminator**: A real violation is any deviation in the characters a strict matcher sees — missing/added quotes, wrong field order, extra text inside brackets. A look-alike that is fine is a difference the task explicitly leaves free (e.g. whitespace between fields when the spec shows none but no strictness is implied) or a value that is spelled exactly as the task's enumerated option even if worded differently from the agent's internal computation.
- **Consequence**: The grader reports 0/N checks passed with "WRONG/MISSING" for fields whose semantic content is actually right, so a substantively correct analysis scores zero.

### Substituting a "creative interpretation" for the requested entities and metrics
- **Applies when**: `task` -- the task names specific entities, groupings, measures, config files, or output artifacts, and the scripts operate on a file whose columns do not contain them.
- **Pattern**: Instead of locating the correct input data (or the correct columns) and producing every requested artifact, the attempt declares a "data mismatch" and maps the requested concepts onto unrelated available fields as proxies, then reports success on that redefined problem; stated config/settings and secondary output files are ignored or only nominally loaded.
- **Detection procedure**:
  1. From the task, list the required inputs (entities to group by, measure to rank by, quantities to aggregate), any settings file that must be applied, and every output file expected.
  2. In the scripts, check whether each listed item is read from a real column/field with matching semantics, or is invented/renamed as a stand-in.
  3. Check that the settings file's contents actually drive the output (figure size, labels, colors, ordering) rather than being loaded and discarded, and that all expected output artifacts are written.
  4. In the answer, look for language such as "proxy", "interpretation", "mismatch", "mapped X to Y" — a self-declared redefinition of the task.
- **Discriminator**: A genuine violation redefines what is being measured or grouped, or silently drops required outputs. A look-alike that is fine is when the correct fields exist but must be derived by documented, semantics-preserving steps (parsing dates into durations, joining tables, standardizing codes) with the requested measure and grouping preserved; also fine is exhausting the available data files first and documenting that the needed field is genuinely absent, rather than assuming it after inspecting one file.
- **Consequence**: The saved figure and any numeric export encode the wrong entities, ordering, and units, so every value-level and file-level check fails (missing expected artifacts, mismatched arrays), even though a plot was produced.

### Output artifact name/format does not match the exact specification
- **Applies when**: `task` -- the task names a specific output file (or template/schema) that the deliverable must be saved as.
- **Pattern**: The agent computes plausible numbers but writes them to a file whose name, spelling, location, or column/index layout differs from the one literally requested (e.g., "corrected" spelling, different directory, missing index column or header names from the template), and the answer text describes the self-chosen name as if it were the requirement.
- **Detection procedure**: 1. Copy the exact filename/path and any template column/index names from the task statement. 2. In the scripts, find every write call (`to_csv`, `to_excel`, `savefig`, etc.) and compare the literal string and the written frame's columns/index to step 1, character for character. 3. In the answer, check which filename and structure the agent claims to have produced. 4. Flag if any of the three differ, or if the agent never verifies the saved file by reading it back and comparing to the template.
- **Discriminator**: A real violation is any deviation in the literal artifact name/path or in the template's column/index structure (including a "fixed" typo). Not a violation if the requested file is written exactly as specified and extra helper files also exist, or if the agent additionally saves an alias copy under the required name.
- **Consequence**: The grader looks for the specified file and reports it as WRONG/MISSING, scoring 0 regardless of whether the underlying computation was correct.

### Fabricated/synthetic data substituted for the provided dataset
- **Applies when**: `task` -- the task references a supplied dataset (with a README/data dictionary) and the script must load it to compute the reported numbers.
- **Pattern**: The script never reads any input file; instead it generates data with a random generator (or hard-codes values) that merely mimics the expected schema, then runs the requested statistics on that invented data and reports the results as the answer.
- **Detection procedure**: 1. Read the task to confirm an external data source is expected. 2. Scan the script for any file-reading call (`read_csv`, `read_excel`, `load_dataset`, DB query, path strings); if the only data construction is `np.random.*`, `seed(...)` loops, or literal lists, flag it. 3. Cross-check reported category names, group counts and sample sizes against the README/actual file — invented data usually shows suspiciously round or uniform group sizes. 4. Verify the answer is written to the exact requested output file/format, not an ad-hoc filename.
- **Discriminator**: Legitimate uses of random generation are auxiliary (bootstrap resampling, simulation baselines, train/test shuffling) applied *on top of* loaded real data; the violation is when the reported statistics derive entirely from generated values with no real data path.
- **Consequence**: The p-values and conclusions are unrelated to the true data, so every value mismatches the expected result file (and the output file may be missing/misnamed), yielding 0 checks passed.

### Discarding high-signal columns and accepting a near-baseline validation score
- **Applies when**: `task` -- a predictive task where the provided table contains identifier, categorical, and date/text columns alongside a handful of continuous measurements, and the scripts feed only the continuous measurements into the model.
- **Pattern**: The attempt builds an ensemble of tree models on a hand-picked list of "numeric" columns, drops or never inspects the entity/name/date/genre/text columns (and never checks whether test rows can be matched back to rows of the reference table by identifier), and then ships predictions even though holdout error is essentially the same as predicting the target mean.
- **Detection procedure**:
  1. From the task/README and the exploration script, list all columns present in the reference table; from the modeling script, list the columns actually used as features.
  2. Flag every dropped column that plausibly carries signal (identifiers, entity/creator names, dates or derived time features, categorical labels, text fields) and check whether the script attempted any encoding, aggregation, target-derived grouping, or an identifier join/overlap check between test rows and the reference table.
  3. Read the reported validation metric and compare it to the trivial baseline computable from the printed target statistics (e.g., RMSE vs. the target's standard deviation, or the printed R²); also compare prediction spread (min/max/std) to the target's spread.
  4. Confirm whether the attempt reported this gap as acceptable and stopped, without trying richer features or a deduplication/lookup path.
- **Discriminator**: A real violation is when informative columns were available and untried *and* the achieved error is at or near the mean-baseline (R² ≈ 0, prediction std ≪ target std, heavily shrunk range). It is not a violation if the dropped columns are genuinely uninformative (free-form IDs with no reuse, constant columns) or if the agent tested encodings/joins and documented that they did not help while the model clearly beats the baseline.
- **Consequence**: Predictions collapse toward the training mean, so any accuracy/correlation threshold the grader applies to the submitted file fails even though the file has the right name, shape, and column header.

### Blind argmax of one internal metric for cluster count, with no sanity check on resulting cluster sizes
- **Applies when**: `task` -- the task asks for an "appropriate" number of groups/hyperparameter and the script picks it by taking the single best value of one unsupervised score over a grid.
- **Pattern**: The script sweeps k, selects `argmax(silhouette)` (or `argmin(DB)`) alone, and accepts the result even though the chosen solution is a poor/degenerate partition — e.g. clusters containing 1–3 points that are really outliers, an absolute score near 0.3 that is barely above the neighbouring k values, and no cross-check against the elbow curve, alternative metrics, robustness to seed, or outlier/skew handling (log-transform, winsorizing) of heavy-tailed features.
- **Detection procedure**:
  1. Read the task for how the number of groups is to be justified and what the output must represent.
  2. In the script, check whether the selection uses more than one criterion / any agreement check, and whether a minimum-cluster-size or stability check gates the final choice.
  3. In the reported output, inspect the cluster size distribution and the score table: flag singleton/near-singleton clusters, or a winning score that differs from the runner-up by a trivial margin.
  4. Check whether skewed/outlier-heavy variables were transformed or outliers handled before distance-based clustering; if not, the "winning" k is likely just isolating outliers.
- **Discriminator**: A fine attempt either shows the metric has a clear, well-separated optimum with balanced, interpretable groups, or explicitly justifies small clusters as genuine outlier groups after robustness checks; a violation is accepting a marginal metric win that yields degenerate clusters with no corroborating evidence or interpretation.
- **Consequence**: The saved label column encodes an outlier-driven partition whose cluster count and assignments disagree with the expected grouping, so the file-level comparison (number of clusters / label agreement) fails even though the pipeline runs without error.

### Per-group statistic computed over an unverified grouping/axis
- **Applies when**: `task` -- the task asks for the entity (country, user, product…) that maximizes a distribution statistic (skewness, variance, mean, etc.) computed within each entity, so each entity's statistic depends on which rows/columns are collapsed into its sample.
- **Pattern**: The attempt computes the statistic without an explicit, inspectable definition of the per-entity sample: it aggregates along the wrong axis (e.g., across entities instead of within one), collapses a filter it should have kept (or keeps rows it should have filtered), silently drops/keeps NaNs and non-numeric values, and then reports only the single argmax — no group sizes, no ranked list, no reproducible script — so an off-by-one-grouping error is invisible.
- **Detection procedure**:
  1. From the task, write down exactly what one entity's sample should be: which rows are selected, which column(s)/periods supply the values, and how many values are expected per entity.
  2. In the script, locate the groupby/loop and the statistic call; confirm the grouping key is the entity, the value vector is the intended one (not transposed, not a row across entities), the required definition/flag is set (e.g., Fisher/bias options), and NaN/dtype handling is explicit.
  3. Check the script prints diagnostics: per-entity sample counts, number of entities, and the top-5 ranked statistic values — not just the argmax.
  4. If scripts are missing or the answer is a bare label with no printed intermediate table, treat the result as unverifiable and reject.
- **Discriminator**: A real violation is when the reviewer cannot reconstruct, from the code, which values form each entity's sample (or can see the axis/filter/NaN handling is inconsistent with the task wording), or when entities with degenerate samples (n≤2, all-NaN, constant) are ranked alongside valid ones. It is fine if the grouping and value selection are explicit and matching, and diagnostics show plausible counts and a stable margin between the top candidates — even if the chosen library call differs stylistically.
- **Consequence**: The argmax shifts to a neighboring entity whose sample was built differently, so the reported name mismatches ground truth and the grader records 0/1 with no trace to diagnose.

### Answer payload mismatch: dumping computed values instead of the requested identifier
- **Applies when**: `task` -- the task asks the agent to create/derive something (a new column, a chosen feature, a selected model) and specifies an answer template whose placeholder names an identifier or label rather than the underlying data.
- **Pattern**: The agent correctly performs the computation but fills the answer slot with the full vector of computed values (often truncated or with inconsistent rounding), rather than the single requested identifier/name described by the placeholder.
- **Detection procedure**: 1. Read the answer-format spec and identify exactly what the placeholder denotes (a name/label vs. a numeric list vs. a single statistic). 2. Check the placeholder's descriptive text and the task verb ("create a feature called…", "which variable…") to infer arity: one token or many. 3. Compare the agent's submitted payload to that arity/type; flag if it substitutes a long value list, an intermediate table, or extra commentary for a single identifier (or vice versa). 4. Verify any stated formatting constraints (rounding, ordering, delimiter, units) are applied to whatever is submitted.
- **Discriminator**: A real violation is a type/arity mismatch with the placeholder's stated meaning (e.g., hundreds of numbers where a column name is described, or truncated output for a required full list). It is *not* a violation if the task genuinely requests every per-row value and the agent supplies them completely, in order, with the specified precision.
- **Consequence**: The grader's exact-match on the expected identifier fails, scoring 0 even though the underlying computation was correct.

### Failure to read and conform to the provided output-format template
- **Applies when**: `task` -- the task says the deliverable file must match the format of a supplied sample/example result file (or otherwise specifies an exact output schema).
- **Pattern**: The scripts never load or print the sample file; the agent invents column names, column order, row ordering, and rounding/precision from intuition, then saves the result and "verifies" only its own arithmetic, so the output can be numerically plausible yet schematically wrong.
- **Detection procedure**:
  1. Read the task and note every stated output constraint (template file, file name, column names, units, rounding, sort order).
  2. Search the scripts for any read/inspection of the template file (e.g., `read_csv` of the sample, printing its header/rows) and any comparison of the produced columns to it.
  3. If absent, check whether the header, column order, row order, and numeric formatting in the submitted answer are asserted anywhere or merely hard-coded by guess (e.g., an arbitrary `.round(2)`, self-chosen labels, self-chosen sort key).
  4. Flag when the deliverable's schema/precision has no traceable source in the provided template.
- **Discriminator**: Not a violation if the script actually loads the template (or explicitly reproduces its exact header/ordering/precision verified against it) and only then writes the output; it *is* a violation when the format is inferred from the agent's own phrasing of the task, even if the underlying aggregation logic is correct.
- **Consequence**: The grader compares the file against the expected schema/values and marks it WRONG/MISSING because headers, ordering, or rounded values do not match, despite correct intermediate computations.

### Coarsening an identifier value to fit a format hint, losing required precision
- **Applies when**: `task` -- the answer includes a key/label (date, ID, category, index) that must be located in the data, and the answer template shows a format or example whose granularity may be coarser than the data's actual granularity.
- **Pattern**: The attempt correctly locates the record but then truncates, rounds, or re-renders the identifier (e.g., drops components of a composite/hierarchical key) to match a literal format hint, reporting a value that no longer uniquely identifies the record it found — even though downstream computations were done on the full-precision value.
- **Detection procedure**:
  1. In the task, note the granularity of the identifier the data actually contains and whether any other constraint (e.g., "previous record", per-record lookup) implies the full-precision key is needed.
  2. In the scripts/output, find the raw identifier of the located record and compare it to the string finally emitted; check for truncation, reformatting, or aggregation of the key.
  3. Check internal consistency: does the emitted identifier, if fed back into the data, select exactly one record — and the same record used for the dependent calculation?
  4. If the emitted key is coarser than the located record (maps to many rows) while the dependent value came from a single row, flag it.
- **Discriminator**: A real violation is when the reported key is ambiguous or lower-precision than the record actually used, so it cannot be verified against the data; it is *not* a violation when the data genuinely has that coarse granularity (one row per that key), or when the task explicitly asks for an aggregate over the coarser grouping and the dependent computation was done at that same level.
- **Consequence**: The identifier check fails on exact-string comparison against the ground-truth full-precision key, so the submission is marked partially correct at best (dependent numeric value may pass while the key fails).

### Substituting proxy/external data for the variables the task actually names
- **Applies when**: `task` -- the task asks for a statistic from a model built on specific named quantities from a provided dataset, and the script instead pulls data from an external source or swaps in a "close enough" stand-in series.
- **Pattern**: The agent cannot locate (or does not look for) the required variables in the supplied data, so it downloads unrelated series or redefines a variable as a proxy (e.g., a rate instead of the named quantity, a different index as the "portfolio"), sometimes with a different frequency, aggregation, or date coverage, then reports the statistic from that substitute model as the answer while acknowledging the substitution in the write-up.
- **Detection procedure**: 1. From the task text, list the exact dependent and independent quantities, the aggregation (e.g., per-period min/mean) and the date range required. 2. In the scripts, trace where each modeling column comes from: is it derived from the provided dataset files, or fetched/invented externally? 3. Check whether the aggregation and filtering literally match the task wording (correct reduction function, correct period grouping, correct start/end). 4. Read the answer/comments for hedging language such as "proxy", "alternative interpretation", "actual variable not available", or a fallback second model — this signals the reported number is not from the requested regression.
- **Discriminator**: A real violation is when a modeling variable is not derivable from the provided data and no equivalence is demonstrated, or the required aggregation is replaced by a different one. It is fine if the agent uses a documented column alias/rename within the provided dataset, or reconstructs the exact quantity (same units, frequency, and reduction) from raw provided columns.
- **Consequence**: The saved statistic comes from a different model on different inputs, so the value in the output file fails the exact/tolerance comparison against ground truth even though the code runs cleanly.

### Model selection and decision threshold not aligned with the task's stated objective on imbalanced classes
- **Applies when**: `task` -- the task states an asymmetric cost/priority (e.g., missed positives are far more expensive) or a specific evaluation metric for a rare-class classification, and the scripts must emit hard 0/1 labels.
- **Pattern**: The attempt trains several off-the-shelf classifiers, compares them with generic metrics (accuracy, ROC-AUC, F1) on one arbitrary hold-out split, then calls `predict()` at the implicit 0.5 probability cutoff — never optimizing recall / the stated cost function, never tuning or even reporting the threshold, and never checking the resulting positive rate against the training base rate.
- **Detection procedure**:
  1. Read the task statement and write down the metric or cost trade-off it actually asks to optimize, plus the class balance seen in the training data.
  2. In the scripts, check how the final model and cutoff are chosen: is selection driven by that metric (e.g., threshold sweep / cost curve / recall-oriented CV), or by default `predict()` and unrelated scores?
  3. Compare the predicted positive count/rate in the answer with the training-set positive rate and with the recall implied on validation; flag if positives are predicted at or below base rate while the task penalizes false negatives.
  4. Check the selection evidence: single random split vs. repeated/stratified CV, and whether the chosen model is justified by numbers actually printed.
- **Discriminator**: A genuine violation uses the default cutoff and off-target metrics with no sensitivity analysis; an acceptable attempt either explicitly tunes the threshold/class weights against the stated cost or metric and shows the trade-off, or documents that the default cutoff is already optimal under that metric.
- **Consequence**: The saved label column has too few positives (low recall), so the graded score under the cost/recall-based check falls below the required threshold and the prediction file is marked wrong even though the file format is fine.

### Analyzing only one partition of the data when the task asks for the whole population
- **Applies when**: `task` -- the question asks for a statistic or detection over "all" units (all countries, all users, all records) and the data directory contains multiple files/partitions or the script filters/loads a single subset.
- **Pattern**: The script hard-codes a single input file (or one group/region/segment) and computes quartiles, thresholds, or aggregates on that partial sample, then reports the result as if it covered the full population — the reference quantities (Q1/Q3, means, ranks) are therefore derived from the wrong subset, even if some of the returned items happen to coincide with the expected ones.
- **Detection procedure**:
  1. Read the task statement and note the stated scope of the population ("for all …") and any required grouping or filtering.
  2. Read the script's data-loading lines: check whether it enumerates/concatenates every relevant file or subset, or loads exactly one and never verifies that this covers the requested scope.
  3. Compare the row/unit count printed or implied by the script against the count implied by the task scope (e.g., number of files in the directory, total known units); a much smaller count signals a partial load.
  4. Check that the threshold/statistic itself (not just the final list) was computed on the full-scope data, and that the answer's labels and format match the requested output exactly.
- **Discriminator**: A real violation is loading/filtering a subset when the task's scope is broader and no justification or coverage check exists. It is fine if the task explicitly restricts scope to that subset, or if the script demonstrates (by listing directory contents or comparing counts) that the single file *is* the complete population.
- **Consequence**: Quartiles/thresholds computed on a partial sample yield a different outlier/selection set than the ground truth built from the full population, so the graded answer list mismatches and scores 0.

### Output label/schema fidelity not verified against the training target's exact categories
- **Applies when**: `task` -- the deliverable is a prediction file with a specified column name whose values are the original class labels (or a specified numeric format) taken from a categorical target in the training data.
- **Pattern**: The attempt trains a model, encodes the target internally, then writes predictions using re-derived or re-formatted labels (title-cased, renamed, 0/1 codes, extra index column, added ID column, wrong row count/order) without ever asserting that the written values and file schema exactly reproduce the label strings and column layout requested; the report even describes classes with capitalization/wording that differs from the raw data.
- **Detection procedure**:
  1. From the task/README, note the exact required file name, column name(s), and the exact set of label strings as they appear in the training target column.
  2. In the scripts, trace the target from load → encoding → inverse mapping → `to_csv`: check whether an inverse transform back to the original strings is applied, whether `index=False` is used, and whether the header equals the requested name.
  3. Check for any explicit sanity check in the script or answer: row count equals the test set size, single expected column, and `value_counts()` keys character-for-character identical to the training label values.
  4. Read the answer's class-distribution/summary text: if the class names or column layout it reports differ in spelling, case, or count from the raw data, treat the output as unverified.
- **Discriminator**: A real violation is the absence of an inverse mapping/format assertion, or evidence of relabeled/re-cased/coded values, extra index or ID columns, or a row count not matching the test rows; a look-alike that is fine is a script that writes original label strings (or the explicitly requested encoding) with `index=False` and prints a verification of shape, column name, and label set — even if the prose summary paraphrases class names loosely.
- **Consequence**: The grader cannot match the expected column values and marks the required output file WRONG/MISSING, so the attempt scores 0 regardless of how good the underlying model is.

### Invented metric definition + entity list back-filled from the plot config instead of derived from the data
- **Applies when**: `task` -- the task asks for a chart/table summarizing a substantive quantity (e.g., "performance", "top N"), and a settings/config file supplies cosmetic details plus axis/category labels.
- **Pattern**: The script treats the config's label list as the source of truth for *which* entities to include and invents an ad-hoc, unjustified formula for the plotted value (arbitrary weights, mixing unrelated counts), rather than computing the substantive quantity from the data and letting the ranking/selection fall out of it. Filters (e.g., the "specified period") are also guessed from a title string. No check is made that the computed ordering/selection reproduces the config's labels, and required auxiliary output artifacts implied by the task/settings are never written.
- **Detection procedure**:
  1. Read the task and the config: list every constraint (which quantity is requested, which subset/period, ordering, required output files/format).
  2. In the scripts, locate the formula for the plotted values and ask whether each term and weight is traceable to the task, README, or config — or was chosen by the agent.
  3. Check whether the set/order of plotted categories is computed from the data (e.g., sorted by the metric and truncated) or merely intersected with the config's label list; if the latter, check whether the agent verified that its metric reproduces those labels in that order.
  4. Check the answer/output for all requested artifacts and for a sanity check on the plotted numbers (plausible ranges, monotonic ordering, counts matching N).
- **Discriminator**: Fine if the metric is a standard/derivable definition for the requested quantity and the config labels are used only as a consistency check that the independently computed ranking matches; a violation if the metric contains agent-chosen weights/terms, the plotted bars are visibly non-monotonic with respect to a "top/best" label ordering, or the category list could not have been reproduced from the data by the script's own logic.
- **Consequence**: The chart's bar values (and any saved numeric/JSON companion outputs) differ from the reference, so all value/plot checks fail even though the figure looks well-formed, and missing required output files fail outright.

### Outlier/threshold counts reported without an independent cross-check of the flagging logic
- **Applies when**: `task` -- the task asks for a count of rows meeting a statistical threshold (z-score, IQR, quantile, etc.) computed on a single column, and the script relies on a library helper plus a boolean mask to both count and filter.
- **Pattern**: The script computes the statistic with one code path (e.g. a library function with special NaN/ddof handling), builds a positional boolean/index mask, applies it to the dataframe, and reports the resulting count without ever verifying it against a simple, independent recomputation (`(|x - mean| > k*std).sum()`, or comparing `min`/`max` to `mean ± k*std`) or checking the column's dtype, NaN count, and units.
- **Detection procedure**:
  1. From the task, note the exact rule and threshold and the exact quantity requested (count of flagged rows).
  2. In the script, check whether the column is inspected first (dtype, non-null count, min/max/describe) and whether any non-numeric/missing values would be silently dropped or coerced, and whether the mask indices (positional vs. label) match the dataframe index used for filtering.
  3. Check whether the flag count is recomputed a second, independent way (manual mean/std formula, or comparing extreme values to the mean ± k·std bounds) and whether the two agree.
  4. Compare the reported count to the reported summary stats: if `max` and `min` lie inside mean ± k·std, the count must be 0; if the count is non-zero, the script should print actual flagged values whose distance from the mean exceeds the bound.
- **Discriminator**: A fine attempt prints the column's summary statistics and the flagged values, and the flagged values are visibly beyond the stated threshold from the mean; a violation reports a count that is never reconciled with the printed distribution, or whose mask alignment/NaN handling (e.g. `nan_policy='omit'` shrinking the array, ddof differences, positional mask on a non-default index) could shift which rows are flagged.
- **Consequence**: The reported integer count (and the size of the filtered dataframe) is off — often a large spurious count where the true answer is zero or vice versa — so the graded numeric answer fails outright.

### Substituting assumed conventions for an explicitly referenced specification file
- **Applies when**: `task` -- the prompt points to an auxiliary document/config (readme, notes, mapping file, data dictionary) that defines how values must be transformed, named, filtered, or reported.
- **Pattern**: The scripts never open or parse the referenced file; instead the agent hardcodes a "standard"/guessed mapping or definition from prior knowledge, then computes and reports statistics using those invented labels (and often skips writing the specified output artifact).
- **Detection procedure**:
  1. Read the task and list every external resource it names and every constraint that resource is said to carry (label names, categories, filters, output file).
  2. Grep the scripts for any read of that resource (`open`, `read_csv`, `read_text`, path string); if absent, the transformation rules are unverified assumptions.
  3. Compare the literal strings/values used in the script's hardcoded dictionary or constants against what the task says should come from the file — no in-script evidence of agreement is a violation.
  4. Check that the answer is emitted in the required form/artifact (e.g., written to the named result file with the named keys), not only printed to stdout.
- **Discriminator**: Fine if the script actually loads the referenced file (or quotes its verified contents after inspecting it) and derives the mapping from it; a violation if the mapping's provenance is only a comment like "these are the standard codes" or general domain knowledge.
- **Consequence**: The numeric ratio may be right while the category label differs from the expected vocabulary (or the required output file is missing), so exact-match grading on the result file fails.

### Submission schema invented instead of copied from the provided template
- **Applies when**: `task` -- the task supplies a sample/template output file (or explicit format spec) and the scripts write a predictions file to disk.
- **Pattern**: The scripts never read the template file; column names, column order, ID column casing, and row ordering are hardcoded from the agent's guess (e.g. self-chosen probability column labels), and the "verification" step only re-checks the file against the agent's own assumptions rather than against the template.
- **Detection procedure**:
  1. In the task/README, note that a template output file is provided and that the output must match it.
  2. Grep the scripts for a read of that template file; if absent, inspect how the output DataFrame's columns/order are constructed.
  3. Compare the agent's produced header and row order to the template's header and ID order (names, count, case, spelling, ordering of rows).
  4. Check whether the verification script asserts equality with the template header/ID sequence, or merely checks internal consistency (probabilities summing to 1, ID set membership).
- **Discriminator**: Not a violation if the hardcoded header/order provably equals the template (agent printed the template header and matched it exactly, including ID ordering); it is a violation when the header labels or ordering are guessed from prose/memory and never diffed against the actual file.
- **Consequence**: The grader cannot parse/align the submission and marks the file WRONG/MISSING regardless of predictive quality, yielding a 0 score.

### Fabricating labels instead of using the provided supervised target and evaluation split
- **Applies when**: `task` -- The task names specific input/output files (e.g., a training file with a target column and a test file to predict on) and the scripts must produce predictions keyed to those exact rows.
- **Pattern**: The attempt declares the ground-truth target "unavailable", builds heuristic pseudo-labels from unrelated auxiliary tables, trains/validates on those self-generated labels (yielding an implausibly high self-consistent CV score), and writes predictions for a row set it invented rather than the rows of the specified test file.
- **Detection procedure**: 1) From the task statement, list the required input file(s), the required prediction file, its required column(s), and the required row set/ordering. 2) Grep the scripts for reads of those files and for the target column; note whether the label used in training comes from the data or from a hand-written rule function. 3) Compare the row count/keys of the produced output against the specified test file's row count/keys, and check the reported column names match exactly what was asked. 4) Check whether the reported validation score is measured against real labels or against the same rule that generated them.
- **Discriminator**: A real violation is when the supervised target exists in the provided data (or the test row set is explicitly given) but the script never reads it and instead invents labels/rows; it is acceptable if the task is genuinely unsupervised, or if heuristics are used only as extra features alongside the true labels and the output still matches the specified test keys and columns.
- **Consequence**: The output file has the wrong number of rows, wrong keys, and/or extra/misnamed columns, and its label distribution bears no relation to the true target, so the grader marks the result file wrong/missing despite a near-perfect self-reported CV accuracy.

### Relabeling output axes to "match" a template instead of verifying value-to-cell alignment
- **Applies when**: `task` -- the task supplies a template/example output file that fixes the row and column labels of a computed table (e.g., group keys by period index), and the script builds its own table then renames or shifts the labels to make them look like the template.
- **Pattern**: The script computes a grouped statistic with its own natural index (e.g., offsets starting at 0), then cosmetically renames columns/rows (adding an offset, reformatting dates) so the header string-matches the template, without establishing which underlying group each template cell actually represents. Verification scripts then only compare shapes/labels, or spot-check cells while openly speculating ("column 4 might mean offset 3"), and the ambiguity is never resolved before writing the file.
- **Detection procedure**:
  1. Read the task/template: note the exact expected row labels, column labels, ordering, and rounding/format of the deliverable.
  2. In the scripts, find every place where labels are renamed, offset, reindexed, or reformatted after the computation, and ask whether the mapping from computed group → template cell is derived from a stated definition or merely assumed to make headers line up.
  3. Check whether any script independently confirms the mapping (e.g., recomputing one cell from raw rows filtered by the definition the template implies, and matching a non-empty template value or a documented convention) rather than comparing an empty template's headers.
  4. Read the answer: look for statements that reveal unresolved ambiguity about what a column/row means, or claims of correctness backed only by "shape and headers match".
- **Discriminator**: A fine attempt derives the label convention from the task/template (or from filled example values) and verifies at least one cell end-to-end against raw data under that convention; a violation performs an arbitrary shift/rename purely so the header text matches, and its own debug output shows two competing interpretations left unsettled.
- **Consequence**: Every value lands one position off (or the whole grid is shifted/transposed), so the file has the right shape and labels but wrong contents, and the grader marks the expected output file WRONG.

### Optimizing/validating with a metric other than the one the task specifies
- **Applies when**: `task` -- the task states an explicit scoring metric (e.g., a rank/agreement/ordinal or cost-weighted metric) and the scripts must choose a model, tune it, or decide when the answer is good enough.
- **Pattern**: The scripts never implement or compute the stated metric; they select the "best" model and ensemble weights using a generic proxy (accuracy, weighted F1, plain log-loss) and treat the target as unordered classes, so the reported/validated numbers say nothing about the actual grading score. Often compounded by a contaminated check (the final ensemble is fit on all training rows and then "evaluated" on a subset of those same rows).
- **Detection procedure**:
  1. Read the task and write down the exact evaluation metric and any structure it implies (ordering of labels, weighting, thresholding, rounding).
  2. Grep the scripts for that metric's computation (or an equivalent implementation) and for the objects used in model comparison/selection; note which score drives the choice of final model.
  3. Check that the score used for selection is computed on data held out from every fitting step (scaler, feature transforms, base models, ensemble weights) — not on rows the final model was refit on.
  4. Check the answer/prediction distribution against what the metric rewards (e.g., under an ordinal-agreement metric, whether extreme classes and label ordering are handled at all rather than collapsed to majority classes).
- **Discriminator**: A real violation is when no run-time estimate of the stated metric exists anywhere, or the metric-relevant structure is ignored, so the agent cannot tell a good submission from a bad one. It is *not* a violation if the agent computes the stated metric on a clean holdout/CV and additionally reports proxies, or if it can show the proxy is a monotone stand-in for the stated metric on that holdout.
- **Consequence**: The submission looks well-formed and plausible, but the graded score (the stated metric) is far below what a metric-aware baseline achieves — predictions cluster on majority labels and the attempt fails the correctness check despite high reported accuracy/F1.

### Missingness mask defined inconsistently with the raw data, without a partition sanity check
- **Applies when**: `task` -- the task asks to split rows into "null" vs "non-null" (or otherwise filtered) groups on one column and compare an aggregate of another column between the groups.
- **Pattern**: The attempt defines the group mask with an ad-hoc or lossy rule (e.g., `isna()` after a loader that coerced sentinels like `"NA"`, `"None"`, `""`, `"-"` to real values or vice versa, `dropna()` on the whole frame, filtering on a different column, or reading only a subset/sample of the file) and/or lets the aggregated column be non-numeric so rows are silently dropped or coerced — then reports group means without checking that the two groups reconstruct the full dataset.
- **Detection procedure**:
  1. From the task, note the exact grouping rule (null vs not-null on the specified column) and the column to be aggregated; note that no other filtering was authorized.
  2. In the scripts, locate the load step and the mask: check for `na_values=`/converters, whole-frame `dropna()`, `astype`/`to_numeric(errors='coerce')`, string-vs-NaN handling, and any row subsetting or sampling; confirm the aggregate is computed on the intended column for each mask.
  3. Require an explicit printed sanity check: `len(group_A) + len(group_B) == len(df)`, per-group non-null counts of the aggregated column, and total row count matching the raw file; if absent, the attempt is unverified.
  4. Check the reported means are consistent with those counts (e.g., recompute the pooled mean from group means and sizes and compare to the overall column mean).
- **Discriminator**: A real violation is when rows are added/lost or reclassified relative to the raw column's true missingness (counts don't partition the file, or sentinel strings are treated inconsistently). It is fine if the aggregated column itself has NaNs excluded by the aggregation *within* each group, provided the grouping mask still partitions all rows and this is stated/verified.
- **Consequence**: Both group means shift by a few percent — plausible-looking numbers that fail exact-value checks (the t-test p-value can still look tiny, masking the error), so the answer is graded wrong.

### Missing or unsaved required output artifacts
- **Applies when**: `task` -- the task (or a referenced guidance/spec file) asks for computed results plus saved outputs, and the script's only persisted product is one figure or console prints while the numeric results live in the chat answer.
- **Pattern**: The agent computes the requested quantities but never serializes them to the expected files/paths/formats (e.g., no saved array/table of the tallied values, no structured record of the plotted data), and instead reports them only as prose; it also does not re-read the spec file to enumerate the full list of deliverables.
- **Detection procedure**:
  1. From the task statement and any referenced guidance/README file, list every deliverable: each output file name, its format, its location, and the values it must contain.
  2. Grep the script for every write/save call (`savefig`, `to_csv`, `np.save`, `json.dump`, `to_json`, etc.) and record the exact paths produced.
  3. Compare the two lists; flag if any required deliverable has no corresponding write, is written to a different directory/name/extension, or if a required numeric result exists only in `print`/the chat answer.
  4. Check the saved figure/data actually encodes the requested numbers (counts, labels, ordering) rather than a stylistic approximation.
- **Discriminator**: A real violation is a deliverable that is never written, or written under a name/format the spec did not ask for; it is *not* a violation if the file exists with the right content and only cosmetic details (title text, dpi, autopct) differ, or if the harness itself relocates correctly named outputs.
- **Consequence**: The grader checks each expected artifact independently and marks every missing/mismatched file WRONG/MISSING, so the attempt scores 0 even when the reported numbers are plausible.

### Output schema not copied from the provided sample/format template
- **Applies when**: `task` -- the task supplies an example output file (or explicit column/row spec) and asks that results be saved in that exact format.
- **Pattern**: The attempt invents its own header names, column count, or row labels (often adding stray/empty columns or renaming fields) instead of reading the sample file and reproducing its schema, so the content may be right while the file fails automated comparison; frequently the values are also hard-coded/recalled rather than produced by an aggregation over the data, with no script retained to show how they were derived.
- **Detection procedure**:
  1. In the task statement, locate the referenced sample/format artifact and note its exact headers, column order, and row-label wording.
  2. In the scripts, check that the sample file is actually loaded or its header literally reproduced, and that the reported values come from a computed aggregation over the full dataset (grouping/summing per entity) rather than typed-in constants.
  3. Diff the submitted file's header and row labels against the sample: same number of columns, same names/spelling/case, no extra or empty trailing fields, expected number of rows.
  4. Sanity-check one value by re-deriving it from the data (e.g., the max of the aggregated quantity) to confirm the ranking logic, not memory, produced it.
- **Discriminator**: A real violation is a structural/label deviation from the given template (extra column, renamed header, invented row wording) or values with no code path producing them; a look-alike that is fine is a file whose schema matches the template exactly and differs only in benign ways the spec leaves open (row order when unspecified, quoting/whitespace).
- **Consequence**: The grader's file/field comparison marks the expected output file WRONG/MISSING and scores 0, even if the underlying leaders were conceptually right.

### Unvalidated raw values in a knowingly "dirty" dataset before imputation/modeling
- **Applies when**: `task` -- the script fits a model or computes a statistic on numeric columns from a source that is flagged (by filename, docs, or dtype) as containing errors, and the required preprocessing is only stated at a high level (e.g., "impute missing with the mean").
- **Pattern**: The attempt takes the columns at face value: it prints dtypes/describe output but never checks whether values are actually parseable numerics within physically plausible ranges (non-numeric strings, sentinel codes, sign errors, unit/magnitude mismatches, duplicated or blank rows). Corrupt entries silently either become NaN-and-mean-imputed or remain as extreme outliers, so the fitted model and the reported error metric are driven by garbage rows.
- **Detection procedure**:
  1. Read the task for signals that the input is noisy/erroneous, and note which columns feed the model and which is the target.
  2. In the script, look for any explicit validation or cleaning step on those columns: coercion to numeric with inspection of what failed, range/plausibility checks, outlier or duplicate handling, verification that the imputed mean is computed on clean values.
  3. Check whether the script sanity-checks the final metric against the target's own scale (e.g., compare RMSE to the target's standard deviation or observed range) and reacts if it is implausibly large.
  4. If neither cleaning nor a scale sanity-check exists, and the reported metric implies typical errors comparable to or larger than the target's spread, flag the attempt.
- **Discriminator**: A real violation is when no inspection of value validity occurred at all, or the reported error is out of line with the target's natural variability and is accepted without investigation. It is fine if the script explicitly verified the columns are clean (ranges, dtype coercion with zero failures) and the metric is consistent with the target's scale — even if no rows were removed.
- **Consequence**: The regression is fit on contaminated values, inflating the reported error by roughly an order of magnitude relative to the reference value, so the single numeric answer fails the grader's exact-match check.

### Unverified reordering of rows before a sequence-dependent computation
- **Applies when**: `task` -- the requested quantity depends on row order (differences, lags, percent changes, cumulative sums, rolling stats) and the script explicitly sorts, reverses, or otherwise re-indexes the table before computing it.
- **Pattern**: The agent asserts an ordering (e.g., "the file is newest-first, so reverse it") in a comment and applies the transformation without ever parsing the ordering key, printing the first/last key values, or checking monotonicity — so the sequence may be flipped relative to the intended one, silently inverting signs and shifting the result.
- **Detection procedure**:
  1. Read the task and note that the requested statistic is defined relative to a "previous"/"next" element, i.e. it is order-sensitive and sign-sensitive.
  2. In the scripts, locate any reversal/sort/reset_index step and check whether it is justified by evidence produced in the same run: is the ordering column converted to a proper dtype (e.g., datetime) and is its monotonic direction verified or sorted on explicitly?
  3. Check whether any post-hoc sanity check on the ordering-sensitive output exists (e.g., comparing the sign/magnitude of the statistic computed both as-is and reordered, or spot-checking one difference against two identified rows by their key values).
  4. Compare the reported answer to what would result under the opposite ordering; if the two differ mainly by sign, the answer is unvalidated.
- **Discriminator**: Fine if the script sorts by the parsed ordering key (`sort_values` on a datetime/index) or prints head/tail of that key and states the observed direction; a violation if the direction is only assumed from a glance or a comment, or if raw row position is used as the order without any key check. Merely reversing data is not itself wrong — the absence of verification is.
- **Consequence**: The lag is taken in the wrong direction, so the mean flips sign (and dispersion shifts slightly), and the graded values fail even though the formula and rounding look correct.

### Output schema invented instead of read from the task's referenced specification
- **Applies when**: `task` -- The task points to an auxiliary instructions file (e.g., a tips/spec/README note) that defines the required test procedure and the exact fields/values of a result file the agent must write.
- **Pattern**: The script never opens or echoes the referenced spec file; the agent guesses the column names, ordering, value wording (e.g., decision phrasing, test-type label), rounding, and number of fields, then declares the output "in the required format" without any comparison to the stated format.
- **Detection procedure**:
  1. Read the task and list every referenced instruction artifact and every explicitly constrained property of the deliverable (file name, one row vs. many, field set, allowed value strings, precision).
  2. Scan the scripts for any read/print of that artifact and for a literal mapping from its stated fields to the written columns; note whether the header/values in the code are copied from the spec or invented.
  3. Check the agent's answer for evidence it quoted the spec's required fields/values; absence of such a quote, or extra/renamed fields (e.g., an added statistic column, custom decision wording), is the flag.
  4. Also verify the chosen test and its preconditions match what the spec prescribes rather than the agent's own judgement (including how degenerate groups, e.g., a group with almost no observations, are handled).
- **Discriminator**: Fine if the script demonstrably reads or verbatim reproduces the spec's field names/values and any deviation is justified; a violation is when the format is asserted from memory or from the prose of the main prompt only, with no trace of the spec content anywhere in scripts or output.
- **Consequence**: The result file exists and looks plausible, but exact-match/field-wise grading of the deliverable fails (wrong or extra columns, non-matching decision strings or precision), scoring 0 despite a statistically defensible analysis.

### Normalization method chosen so the reported statistic is trivially degenerate
- **Applies when**: `task` -- The task asks to "normalize"/"scale" columns and then report summary statistics (e.g., means) of those transformed columns.
- **Pattern**: The agent applies z-score standardization (subtract mean, divide by std), which forces every reported mean to 0 (or -0.0), instead of a scaling that preserves informative means (e.g., min-max to [0,1]); it then reports these degenerate zeros without questioning that the task would not ask for values that are known in advance.
- **Detection procedure**:
  1. Read the task to see which columns are to be scaled and which statistic must be reported afterwards.
  2. Inspect the script's scaling step: check whether the transform makes the requested statistic constant by construction (standardization → mean 0; centering → mean 0).
  3. Check the answer: if the reported values for all scaled columns are 0.0/-0.0 (or otherwise identical trivial constants) while unscaled columns carry real information, flag it.
  4. Confirm the answer format expectation (four-decimal distinct floats per column) is inconsistent with an all-zero result, and require the alternative scaling (min-max) or an explicit justification.
- **Discriminator**: A real violation is when the requested statistic is mathematically fixed by the chosen transform, rendering the report uninformative; it is fine if standardization is explicitly named in the task, or if the reported statistic (e.g., std, min, max, or a downstream model score) is still informative under standardization.
- **Consequence**: All scaled columns' reported means are 0.0 instead of the expected in-range values, so every check on those columns fails while unscaled/binary columns coincidentally pass.

### Unverified row set / missing-value handling for a whole-dataset statistic
- **Applies when**: `task` -- the task asks for a single summary statistic or test result computed over two or more columns of a supplied table, and the scripts (or lack of them) do not show how rows with missing/non-numeric entries were handled or how many rows entered the computation.
- **Pattern**: The attempt computes the statistic after an implicit or convenience row reduction — e.g. dropping all rows with any NA anywhere in the table instead of only in the two relevant columns, coercing/filtering out non-numeric values, subsetting to a preview/sample of the file, or reading with a wrong delimiter/header so some rows are lost — and reports the resulting number without stating the sample size or checking it against the file's row count. It also reports derived values with less precision than requested (e.g. a truncated p-value) instead of the stated rounding.
- **Detection procedure**:
  1. From the task, note that the statistic is defined over the full table on the pair of named columns, and note the required output precision/format for each reported quantity.
  2. In the scripts, locate the load step and every row-removing operation (`dropna()` without `subset=`, boolean filters, `head`/`nrows`, dtype coercion, deduplication) and check whether any of them can remove rows that have valid values in both target columns.
  3. Check that the script prints the number of observations actually used (and non-null counts per column) and that this is compared with the raw file row count; if no script or no such print exists, treat the number as unverified.
  4. Compare the reported figures against the required rounding/format (e.g. two vs four decimals, no bare `0.0` for a p-value that should be given to four decimals).
- **Discriminator**: A real violation is any row exclusion that is not required by the task and not justified/reported (or an unstated sample size), or a reported value whose precision differs from the spec. It is *not* a violation if the script restricts to pairwise-complete cases on exactly the two target columns, prints the resulting n, and matches it to the expected count — even if a few rows are legitimately dropped as genuinely missing.
- **Consequence**: The coefficient shifts in the second decimal place (or the p-value string is malformed), so the graded numeric field mismatches the expected value even though the qualitative conclusion is right, and the attempt is scored wrong.

### No held-out validation or distribution sanity check on the predictions
- **Applies when**: `task` -- the script trains a model on a labeled file and writes predictions for an unlabeled file, with no accuracy metric required in the deliverable.
- **Pattern**: The attempt fits one model configuration on 100% of the labeled data, immediately predicts on the target file, and reports only self-described summary statistics of the predictions — never computing an error estimate on a held-out split, never comparing the predicted distribution (min/median/mean/max, skew) against the training label distribution, and never checking that the test rows were preprocessed identically (same feature construction, same category vocabulary, same handling of unseen/missing categories, same target scale/units).
- **Detection procedure**:
  1. Read the task: identify the required output file, its column, expected row count, and the plausible scale/units of the quantity being predicted.
  2. Read the script: check whether any split (train/validation, CV) is used to produce a numeric error estimate, and whether unseen categories/missing values in the target file are mapped in a way consistent with training (e.g., silently collapsed to an existing valid code rather than an explicit "unknown").
  3. Read the answer: compare the reported prediction statistics with the label statistics of the training data; flag if the mean/median/max are implausible or off by a large factor, or if a floor/clip was applied without justification.
  4. Confirm whether any validation number is reported at all; if the only evidence of quality is "predictions were saved", the attempt is unverified.
- **Discriminator**: A real violation is an attempt with zero out-of-sample error estimate *and* no comparison of predicted vs. observed label distributions; it is fine if the agent reports a CV/holdout score (even from a simple baseline) and shows the prediction distribution matching the training label distribution, or explains any deliberate shift.
- **Consequence**: The submitted file has the right shape and column name but systematically miscalibrated values (inflated mean/median, absurd extremes), so the grader's tolerance/error check on the predicted values fails while the agent claims success.

### Prediction file row count/order doesn't match the evaluation input
- **Applies when**: `task` -- the task asks for a per-row output file (one prediction per input record) to be written for a held-out input set.
- **Pattern**: The attempt produces an output file whose number of rows is far smaller (or larger) than the number of records in the provided input file — e.g. only a handful of rows from a truncated preview, a `head()`/sampled subset, a debug run, or rows dropped by filtering/`dropna` — and/or in an order that no longer aligns with the input rows, while still claiming completeness.
- **Detection procedure**:
  1. From the task and data files, determine the expected output length = number of records in the held-out input file (read its shape, don't assume), and the required column name/header.
  2. In the scripts, trace the object that is written out: check that it was built from the *full* input frame (no `nrows=`, `.head()`, `.sample()`, `dropna`/filtering, or partial-batch loop) and that row order is preserved (no `sort`, `groupby`, or reindex before writing).
  3. Count the rows in the produced answer/file and compare to the expected length; also check the header and allowed label/value set.
  4. Flag if the counts differ, if the count equals a suspicious small/round number, or if scripts are absent so the count cannot be traced to the full input.
- **Discriminator**: A genuine violation is a length/alignment mismatch with the held-out input (or an untraceable pipeline); it is *not* a violation if the row count equals the input record count and the task itself legitimately requested an aggregated or filtered subset with that smaller size.
- **Consequence**: The grader cannot align predictions to ground truth, so the submission is scored as wrong/missing regardless of model quality (or the score reflects only a tiny fraction of rows).

### Output artifacts written to an arbitrary directory instead of the task's working/data directory
- **Applies when**: `task` -- the answer must include file paths to artifacts the agent creates (cleaned/transformed CSVs, model files, plots) that a grader will match or open.
- **Pattern**: The agent saves outputs to its own home/current directory (or a temp path) rather than the directory the input data lives in / the directory implied by the task environment, and reports that path verbatim, so all path-bearing checks fail even though the computed numbers are right.
- **Detection procedure**:
  1. From the task statement, note where the input file(s) are located and any stated or conventional output location; treat the input's directory as the default expected output location when none is stated.
  2. In the scripts, find every write call (`to_csv`, `savefig`, `dump`) and record the literal or constructed output path; check whether it is derived from the input path or hardcoded to an unrelated directory.
  3. Compare the paths reported in the final answer to the input directory and to the paths actually written; flag any mismatch in directory, filename, or extension.
  4. Confirm the reported paths are absolute and refer to files that the script actually creates (name spelled identically, no later overwrite/rename).
- **Discriminator**: A real violation is an output path whose directory differs from where the data was read (or from an explicitly required location), or a reported path that differs from what the script writes; it is fine if the task explicitly permits any path and the reported path exactly matches a file the script created in the same data directory.
- **Consequence**: Path-containing checks are marked WRONG/MISSING even when the numeric fields match, so the item scores 0.

### Incomplete compliance with an external plot/output specification (missing required artifacts)
- **Applies when**: `task` -- the task points to an external configuration/specification file (e.g., a YAML/JSON style guide) and/or names specific output files that the deliverable must consist of.
- **Pattern**: The agent reads only a few keys of the spec file (or hardcodes assumptions about it), produces just the one visually obvious artifact (an image), and skips the other mandated outputs (serialized plot data / numeric arrays / config echo), or applies the spec's directives (titles, labels, order, colors, units, size) only partially. The final answer is prose about the finding rather than a checklist of produced files matching the requested format.
- **Detection procedure**:
  1. From the task text, list every required output artifact (file names/extensions) and every constraint the referenced spec file is said to govern.
  2. In the scripts, list every file actually written and every spec key actually consumed; compare against step 1, including whether the spec is fully enumerated/validated rather than accessed by a few hardcoded keys.
  3. Check the answer/report: does it confirm each required artifact exists with the right content type (e.g., underlying counts/values saved, not just an image), and in the requested location?
  4. Flag if any required artifact is never written, or any spec section is never read (e.g., keys present in the config that no code path uses).
- **Discriminator**: A real violation is a required output file or spec directive with no corresponding code path. It is *not* a violation if the script writes all named artifacts and defensively reads the spec generically (e.g., iterating keys, with a printed dump verifying which directives were applied), even if some spec keys are simply unset/empty in the file.
- **Consequence**: The grader checks each expected file independently; missing serialized data/config artifacts and partially styled figures score as WRONG/MISSING, so the run fails all checks even when the intermediate analytical finding is right.

### Dropping rows with missing values silently changes the evaluated sample
- **Applies when**: `task` -- The task prescribes a fixed pipeline (given features, fixed split fraction and random seed, one metric) and the script must decide how to handle missing/invalid entries in the feature columns.
- **Pattern**: The script calls a blanket row-drop (e.g. `dropna()`) on the feature/target frame before splitting, so the number of rows — and therefore the exact train/test partition produced by the seeded split — differs from the intended full-dataset pipeline; no imputation or justification is given, and the resulting row/split counts are never reconciled against the raw data.
- **Detection procedure**: 1. Read the task for any statement (or absence of a statement) about missing-value handling and note that the split is seed-fixed, so the row set fully determines the answer. 2. In the script, locate every operation that removes or filters rows (`dropna`, boolean masks, `drop_duplicates`, index slicing) applied before `train_test_split`. 3. Check whether the script reports the raw row count vs. the post-cleaning count and whether the loaded file/columns actually still contain missing values that require handling; check whether an imputation (mean/median/mode) alternative was considered. 4. Confirm the answer was computed on this reduced set with no sanity comparison to the full-data variant.
- **Discriminator**: A real violation is unrequested row deletion in *feature* columns that shrinks the modeling sample (imputation would preserve it); it is fine to drop rows lacking the *target* (unlabeled rows cannot be scored), or to drop nothing because the columns are already complete — and it is fine if the script verifies the row count is unchanged after cleaning.
- **Consequence**: The seeded split covers a different subset than the reference pipeline, so the reported accuracy is off by a few points (e.g. 0.76 vs. the expected 0.78) and the exact-value check fails.

### Ignoring the provided output template when filling a required results file
- **Applies when**: `task` -- The task supplies a pre-existing output file (template/skeleton) and asks that results be entered into it "in the same format", and the scripts generate that file's contents.
- **Pattern**: The attempt never reads or inspects the supplied template; it invents its own schema — column names, row labels/category names, category definitions (e.g. self-chosen bin boundaries), row ordering, and extra catch-all rows — and writes a fresh file from scratch, then "verifies" the format only against its own assumptions.
- **Detection procedure**:
  1. In the task text, identify the exact output artifact required and any statement that its existing format must be followed.
  2. Search the scripts for any read/inspection of that template file (e.g. loading it, printing its rows/columns/label values) before writing.
  3. Check whether the labels, categories, ordering, and column headers written out are derived from the template or hard-coded from the agent's own reasoning; check whether extra/renamed rows (e.g. an "unknown/other" bucket) or self-invented category thresholds were introduced.
  4. Check whether the final answer is a narrative report rather than confirmation that the template file was filled in place with matching keys.
- **Discriminator**: A real violation is inventing the key set/schema without ever reading the template; it is fine if the script loads the template, preserves its exact headers and row keys/order, and only populates the value cells — even if the agent additionally prints a human-readable summary.
- **Consequence**: The required file fails key-by-key comparison against the expected file (mismatched row labels, extra rows, or values computed under different category definitions), so the grader reports the output file as WRONG/MISSING and scores 0.

### Self-reported performance measured on the training data instead of a held-out split
- **Applies when**: `task` -- the agent fits a predictive model and must produce predictions for an unlabeled evaluation set that will be scored externally.
- **Pattern**: The script trains on all labeled rows and then computes the quality metric with the same rows (or via `predict` on the training frame), reporting a high in-sample score as evidence the submission is good; no cross-validation or hold-out estimate, no comparison against a trivial baseline, and no check that the predicted positive rate / class balance is plausible relative to the training labels. Hyperparameters (depth, class weights, thresholds) are therefore chosen with no honest signal, and overfit or badly calibrated predictions go out unvalidated.
- **Detection procedure**:
  1. In the task, note that the scored artifact is predictions on data whose labels the agent never sees, so the only defensible quality claim is an out-of-sample estimate.
  2. In the scripts, locate the metric computation and check which rows are passed to it: are they the same rows used in `fit`, or a split/CV fold withheld from training?
  3. Check whether any baseline (majority class, simple logistic/default hyperparameters) is scored on that same held-out data for comparison, and whether the predicted label distribution on the evaluation set is compared to the training base rate.
  4. In the answer, see whether the headline score is labeled as training/in-sample; if it is, or if its provenance is unstated and step 2 shows no split, treat the quality claim as unsupported.
- **Discriminator**: A real violation is a metric whose inputs are provably a subset of the fitted rows with no CV/hold-out anywhere; it is *not* a violation if the agent held out or cross-validated (even with a modest score), or if it reports an in-sample number *in addition to* a clearly identified out-of-sample estimate used for model selection.
- **Consequence**: The reported score is inflated and uninformative (e.g., near-perfect in-sample vs. much weaker true performance), so the submitted prediction file can fall below the grader's accuracy/AUC threshold — and its class mix be far off the true base rate — while the answer confidently claims success.

### Ignoring the provided output template and the method implied by stated constraints
- **Applies when**: `task` -- the task points to a sample/reference output file for the result format and/or states an execution constraint (e.g., fix a random seed) that implies a specific computational approach.
- **Pattern**: The attempt never loads or inspects the sample output file and instead invents its own column names/rows, and it substitutes a closed-form/analytic shortcut for the randomized procedure implied by the seed requirement (so the seed is never actually used), then reports that value as the requested statistic.
- **Detection procedure**:
  1. Read the task for (a) a named format/template file and (b) constraints such as a random seed, rounding, units, or ordering.
  2. In the scripts, check that the template is actually read (or its exact header/row layout reproduced) and that every stated constraint appears in code — e.g., the seed is set and consumed by a stochastic step (resampling/permutation/bootstrap/model init).
  3. Compare the written file's schema (column names, count, order, row keys) and the value's provenance against the template and the requested quantity; flag if the schema is self-invented or the seed is decorative/unused.
  4. Check the reported answer restates the same schema and value as the saved file.
- **Discriminator**: A real violation is inventing a schema or using a deterministic method when the task's constraints (template + seed) demand a specific format and a randomized computation; a look-alike that is fine is a deterministic method used when no randomized procedure is implied, or extra explanatory prose alongside a file that provably matches the template schema.
- **Consequence**: The saved file fails exact-format/value comparison against the expected result (column names or the numeric p-value differ), scoring 0 even though the narrative conclusion may sound plausible.

### Predictions not validated against the reference target distribution
- **Applies when**: `task` -- the deliverable is a prediction file for a held-out split that will be scored against hidden ground truth, and the agent reports only self-computed validation metrics.
- **Pattern**: The agent fits one default model, reports a mediocre in-house score (or none), and declares success from file-writing alone — never checking that predicted values plausibly reproduce the training target's distribution (mean, spread, min/max, skew) or that the row count/order/index matches the test input, and never trying to improve a clearly weak fit.
- **Detection procedure**: 1) Read the task for the required file name, column name, row count and any ordering/format constraint. 2) In the scripts, check whether the same feature construction and preprocessing are applied to train and test, and whether predictions are written in the test file's original row order with no extra index column. 3) Compare the reported prediction summary statistics (mean, min, max, variance) with the training target's statistics — flag if the predictions are visibly compressed, shifted, clipped, or far narrower/wider than the target. 4) Flag if the reported goodness-of-fit is weak and no alternative model, tuning, or error analysis was attempted before finalizing.
- **Discriminator**: A genuine violation shows missing sanity checks *and* evidence of a distribution/shape mismatch or an unimproved weak model; it is fine if the agent explicitly compared prediction stats to the target, verified row count/order/column name, and justified the model choice with a comparison — shrinkage toward the mean alone is normal for regression and not by itself a violation.
- **Consequence**: The submitted file scores below the grader's accuracy/error threshold (or fails format/shape validation), so the expected output is marked WRONG despite the script running without error.

### Trusting a single pre-existing column for a requested derived quantity, with a vacuous imputation step
- **Applies when**: `task` -- the task asks to impute missing values and then rank/extremize a quantity that can either be read from one raw column or be recomputed from other columns in the data (e.g., a ratio of two available fields).
- **Pattern**: The script picks the one column whose name resembles the requested quantity, cleans/imputes only that column, and reports its argmax/argmin — never checking that the stated preprocessing actually did anything (the column may have zero missing values, making the imputation a no-op) and never cross-validating the column against the quantity implied by its definition computed from the underlying fields.
- **Detection procedure**:
  1. From the task wording, write down the definition of the requested quantity and which raw fields it depends on; note the explicitly required preprocessing step.
  2. In the scripts, check whether preprocessing is applied to all fields the quantity depends on (or dataset-wide as stated) or only to one convenient column, and whether the script prints/asserts that the imputation changed at least one value.
  3. Check whether the script recomputes the quantity from its components and compares with the raw column (or at least verifies the raw column's units/scale and its extremes against known plausible ranges).
  4. Inspect the reported extremes: if the reported min/max come solely from the raw column with no consistency check, and the imputation was a no-op, flag the attempt.
- **Discriminator**: A fine attempt either shows that the raw column is consistent with the definition (recomputed values match, or the task unambiguously names that single column) and reports how many values the imputation filled; a violation silently assumes the column is correct and complete, leaving the mandated imputation unexercised and the definition unverified.
- **Consequence**: The reported extreme entities come from an inconsistent/uncorrected column, so the submitted min and/or max labels differ from the expected ones and the result file fails the equality check.

### Requested output file omits the computed quantities the task asked to include
- **Applies when**: `task` -- the task asks to compute several per-entity quantities (scores, groups/segments, labels) and save "the results, including X and Y" to one named output file, and the script builds a full result table plus a trimmed export.
- **Pattern**: The script computes all the intermediate per-entity metrics and derived scores in a working dataframe, but writes only a minimal subset of columns (e.g., ID + final label) to the required filename, dumping the complete table to a differently-named auxiliary file; the answer then claims completeness because the auxiliary file exists.
- **Detection procedure**: 1. From the task text, list every quantity that must appear in the named output file (each component metric, each derived score, the grouping/segment, the final level). 2. In the script, find the write call to that exact filename and enumerate the columns actually selected there. 3. Diff the two lists; also check whether a richer table was written to a *different* filename. 4. Check the answer's description of the file's contents against the task's required contents.
- **Discriminator**: A real violation is when quantities the task explicitly named (or the per-entity scores the task's phrasing requires) are absent from the required file, even if present elsewhere on disk. It is fine if the file has extra columns, different column ordering/names, or if the task genuinely asked only for the final label and the extra table is a bonus.
- **Consequence**: The grader compares the named file against a reference containing the full set of columns and marks it WRONG/MISSING (0 checks passed) despite the underlying computation being reasonable.

### Ignoring a required subgroup filter (and unverified schema) when aggregating a statistic
- **Applies when**: `task` -- the task asks for a statistic over a specific subgroup/condition (e.g., one category, species, region, cohort) while the raw files contain multiple groups, and/or a template file defines the required output columns/values.
- **Pattern**: The script reads each file and aggregates over every row, assuming the file already contains only the requested subgroup and that the column names it hard-codes exist as written; no filtering step, no schema inspection, and no comparison against the provided sample output file.
- **Detection procedure**:
  1. Read the task and note every restriction on rows (group/category, year, condition) and every stated output constraint (column names, ordering, rounding, format template).
  2. In the script, look for an explicit filtering operation (`df[df[group_col] == ...]`) on the group column and for any inspection/validation of the actual columns present in each input file; check whether the template/sample output file is ever read or its columns replicated.
  3. Check whether the script prints or asserts row counts / value ranges per group after loading, so a mismatch between the filtered subgroup size and the full file size would be visible.
  4. Compare the reported numbers to a rough domain sanity expectation (e.g., counts identical across heterogeneous files, or a ratio suspiciously near 1 when the two measured quantities are known to differ substantially) — an unexplained coincidence signals the filter or column mapping was wrong.
- **Discriminator**: A real violation is when the raw file plausibly contains rows outside the requested subgroup (or differently named columns) and the script never filters/validates; it is fine if the script demonstrably verifies the subgroup (asserts unique group values, prints per-group counts, or the file is documented as already restricted) and reproduces the template's exact columns.
- **Consequence**: The reported means/CIs are computed on the wrong row subset (or wrong columns) and the output file fails an exact/tolerance comparison to the expected result.csv, scoring 0.

### Loaded only a truncated slice of the source data (and never sanity-checked the result)
- **Applies when**: `task` -- the task asks for a statistic/model over a whole dataset file, and the script reads the data (possibly with `nrows`, a preview/head, a single chunk, one of several files, or a hand-made sample) before computing the requested output.
- **Pattern**: The attempt computes the requested statistic on a small subset of the real data (e.g., a first-N-rows preview kept from an exploratory step) while reporting it as the full-dataset result, and accepts numerically implausible values (near-zero or wrong-signed correlations among conceptually related variables, tiny counts) without any cross-check against the file's true size or expected direction.
- **Detection procedure**:
  1. From the task, note that the statistic must cover the entire dataset; find the loading call in the script and check for row limits (`nrows`, `head`, slicing, chunk iteration, sampling, reading only one of multiple input files).
  2. Compare the row count printed/reported in the answer against the actual size of the source file (line count / shape); flag suspiciously round or small numbers (e.g., exactly 100) as a preview artifact.
  3. Check whether the script prints/validates shape, non-null counts, and value ranges of the analyzed columns, and whether the answer's numbers are directionally plausible for related variables.
  4. Flag if the computation runs on anything other than all rows of the intended source, or if no shape/plausibility check exists.
- **Discriminator**: A real violation is a row limit or partial read that affects the *final* computation, or a reported N that is far below the file's true row count; it is fine if the limited read is used only for schema inspection/debugging while the final computation reloads the complete file, or if the reduced N is fully explained by the required missing-value filtering on a full read.
- **Consequence**: The saved output file contains statistics from an unrepresentative subsample, so every cell differs from the expected values and the file-comparison check fails (0/1), even though the format looks correct.

### Cluster count chosen at the edge of the search grid, ignoring known group structure
- **Applies when**: `task` -- the task asks for an "appropriate" number of groups/components (or any hyperparameter) and the script picks it by scanning a bounded grid and taking the arg-max/arg-min of an internal metric.
- **Pattern**: The attempt sweeps k over a hard-coded range, selects the value with the best score, and that winner lands on the last (or first) value of the range with a weak absolute score — so the "optimum" is an artifact of where the sweep was truncated, not a real structural optimum. It also disregards strong external evidence of the natural group count available in the data (e.g., a categorical/label column that was dropped, or a documented set of classes), and never re-runs with a wider range or a second criterion (elbow, stability, comparison against the known cardinality).
- **Detection procedure**:
  1. In the task/README, note any stated or implied number of natural groups (a target/label column the script drops, documented categories, domain description).
  2. In the script, find the candidate range and the selection rule; check whether the selected value can be an endpoint of that range and whether any secondary check (elbow, stability, agreement with known cardinality) gates the choice.
  3. In the answer, compare the reported chosen value to the range bounds and look at the absolute value of the selection metric (e.g., a silhouette near 0.1–0.2 means almost no separation, so the arg-max is noise).
  4. Flag if the choice equals a range endpoint, or if the metric is essentially flat/weak, and no justification ties the choice to the data's known structure.
- **Discriminator**: Fine if the winning value lies strictly inside the range with a clear peak, or if an endpoint win is confirmed by extending the range or by an independent criterion / the known group count; a violation is an endpoint (or noise-level) arg-max accepted as-is with no widening and no cross-check.
- **Consequence**: The saved cluster labels use a partition count that disagrees with the reference grouping, so label-count / agreement checks (e.g., number of distinct clusters, ARI/NMI or cluster-quality thresholds) on the output file fail even though the file itself is well formed.

### Degenerate cluster solution driven by unhandled outliers/skew
- **Applies when**: `task` -- the task asks for an unsupervised grouping into "an appropriate number of groups" and the script picks the number of groups by maximizing an internal score over raw aggregated, heavy-tailed features.
- **Pattern**: The script builds skewed count/monetary aggregates, applies only mean/variance scaling (no log/robust transform, no outlier handling), then selects the configuration with the best silhouette — which is almost always the smallest number of groups that isolates a handful of extreme records, yielding one giant group plus a few singletons and no interpretable segmentation.
- **Detection procedure**: 1) Read the task to confirm the deliverable is a meaningful multi-group segmentation, not just any label column. 2) In the script, check whether heavy-tailed features are log/rank/robust-transformed or winsorized before scaling, and whether the model-selection loop has any balance/interpretability guard beyond a single internal index. 3) Read the reported group sizes and score: flag if the near-perfect score coincides with a group containing a negligible fraction (e.g. <1%) of rows and the remainder in a single group. 4) Check whether the agent noticed and diagnosed this degeneracy or accepted it as "excellent quality".
- **Discriminator**: A genuine violation is a mathematically extreme score produced by outlier isolation with essentially no partition of the bulk of the data; it is fine if a small group exists alongside several substantively sized groups, or if the agent explicitly justifies the small group after outlier treatment and shows the bulk is still meaningfully partitioned.
- **Consequence**: The saved label file has near-zero effective segmentation (one dominant label), so any grader check on cluster count, label distribution, or segment-quality/consistency fails even though the file format is correct.

### Hardcoded/recalled numbers instead of resampling the provided data
- **Applies when**: `task` -- the task supplies a dataset (with a README) and asks for a statistic or confidence interval computed from it, and the script defines the inputs inline rather than loading any file.
- **Pattern**: The attempt never reads the provided data files; it types in summary counts or rates from memory/background knowledge and then simulates from a parametric assumption (e.g., drawing from a Binomial with the assumed rate) instead of resampling the actual observed records, so both the point estimate and the interval reflect invented inputs and an unrequested method.
- **Detection procedure**:
  1. Read the task/README and list the data files that are expected to be the source of the requested quantity.
  2. Scan the script for any I/O (`read_csv`, `read_excel`, `load`, path strings) and check that the quantity is derived from those loaded objects; flag literal constants that stand in for data.
  3. Check that the resampling/estimation step draws with replacement from the loaded observations (or their groups) rather than from a hand-specified distribution or parameter.
  4. Compare the reported numbers against what the actual files would yield (row counts, group sizes, the observed effect size) — if the script prints counts that cannot be traced to a file, treat the result as unverified.
- **Discriminator**: A real violation is when key inputs (counts, rates, group definitions, filters) originate from the agent's assumptions and are never cross-checked against the delivered files; it is fine to hardcode constants that are explicitly stated in the task/README, or to compute summaries in one script and pass them along, as long as they were derived from the loaded data.
- **Consequence**: The output file contains an interval centered on the wrong effect size (and with the wrong width from a parametric rather than empirical bootstrap), so the graded value falls outside the expected tolerance and the result file is marked WRONG.

### Missing verification that the prediction file exactly matches the required submission template
- **Applies when**: `task` -- the task requires writing an output file whose rows/keys and column layout must match a provided sample/template file over an entire held-out set.
- **Pattern**: The scripts build predictions and write the output directly from model probabilities, but never load the template, never assert that the number of rows equals the number of held-out records, that the key column covers exactly the same identifiers (same set, same order), and that the column names/order match; the final answer is accepted without any row-count or key-coverage sanity check (and can silently be truncated, reordered, subset, or column-swapped).
- **Detection procedure**:
  1. Read the task for the stated output contract: required file name, header names, key column, and the fact that every held-out record must appear exactly once.
  2. Scan the scripts for an explicit comparison against that contract — e.g. reading the sample/template file, `assert len(sub) == len(test)`, `set(sub[key]) == set(template[key])`, column-name/order equality, and a check that written file re-reads with the expected shape. Absence of all of these is the violation.
  3. Inspect the produced answer/file: count rows and compare to the held-out record count; check the key column for duplicates, missing ids, and that class-probability columns are in the specified order and non-degenerate.
  4. Also confirm the class-index-to-column mapping is derived from the fitted model's class ordering rather than assumed positionally.
- **Discriminator**: A real violation is the total absence of any template/row-count/key check (or an actual mismatch in the emitted file). It is not a violation if the script reconstructs the submission from the template's key column (e.g. merges predictions onto the template ids) or asserts shape/ids/columns before saving — the checks may be terse, but they must exist and cover keys, count, and column layout.
- **Consequence**: The grader reports the expected output file as WRONG/MISSING regardless of model quality, since rows are missing/misaligned or columns are mismapped, and the log-loss cannot be computed over the full held-out set.

### Predicted values not sanity-checked against the training target's scale and type
- **Applies when**: `task` -- the agent trains a regressor to produce a predicted count/quantity/price column for held-out rows and writes it to an output file.
- **Pattern**: The script transforms the target (log/sqrt/scaling) or fits on a filtered/atypical subset, then writes raw model output without inverting the transform or checking it against the observed target distribution — producing predictions whose scale, dtype (e.g., fractional values for integer counts), or row count is inconsistent with the training labels, with no assertion or comparison step.
- **Detection procedure**:
  1. From the task/README, note what the target represents (units, integer vs continuous, plausible range) and how many output rows are required.
  2. In the scripts, find every target transformation, clipping, subset filter, or scaling; verify an explicit inverse transform is applied before writing, and that the output is built from the full held-out set in original order.
  3. Compare the answer's summary statistics (min, median, max, count, share of near-zero values) against the training target's statistics; flag if the central tendency differs by an order of magnitude or the value type is incompatible (e.g., mostly sub-unit values where labels are large integers).
  4. Check the script contains an explicit sanity assertion on shape/range before saving; absence plus a mismatch in step 3 confirms the violation.
- **Discriminator**: A genuine violation shows a systematic distribution mismatch (e.g., predictions concentrated far below the label median, or row count ≠ required rows). It is *not* a violation if predictions are merely smoother/less dispersed than labels — regression shrinkage toward the mean is expected as long as the central tendency and units match the labels.
- **Consequence**: The output file fails the grader's value/format comparison — error metrics are dominated by a constant scale bias (or the file has the wrong number/type of entries), so the answer is marked wrong despite a plausibly reasonable model.

### Unfiltered population and unchecked test assumptions in hypothesis testing
- **Applies when**: `task` -- the task asks for a p-value / significance decision about a statistic of some group comparison, and the scripts run an off-the-shelf test on whole raw tables straight from disk.
- **Pattern**: The script loads every row of each source file, computes the derived quantity, and immediately applies a default parametric test (e.g., independent-samples t-test) without (a) restricting to the subpopulation, time window, or competition/segment that the question is actually about, and (b) checking whether the test's distributional assumptions (normality, symmetry, equal variance, independence) hold for the derived quantity — typically a small-integer, skewed, count-like variable.
- **Detection procedure**:
  1. Read the task/README and list every scoping qualifier implied by the question (which subset of records, which period, which category the claim concerns) plus the significance level and output format.
  2. Read the script for filtering steps: does any `query`/boolean mask/date parse restrict the rows before the statistic is computed? If the row counts used are essentially the full file sizes, the scoping step is missing.
  3. Inspect the test choice: is there any diagnostic (histogram, skew, normality test, variance comparison) or justification, and does the test's one/two-sided direction match the stated hypothesis? A default `ttest_ind` on a bounded count variable with no diagnostic is a red flag.
  4. Check the reported p-value's magnitude for plausibility given the intended (much smaller) subset — an astronomically small p-value is a symptom of using far more rows than the question intended.
- **Discriminator**: A real violation is when the task or README implies a narrower population/assumption-appropriate test and the script silently uses all rows with a default parametric test; it is fine if the task genuinely asks about all records and the script either documents the assumption check or the chosen test is robust/appropriate for the variable's distribution.
- **Consequence**: The p-value (and possibly the reject/fail-to-reject decision) is computed on the wrong sample with the wrong test, so the saved value does not match the expected one and the file fails the grader even though the format is correct.

### Ignoring the provided output template (format/rounding/ordering never verified)
- **Applies when**: `task` -- the task supplies a sample/expected output file (or explicit format spec) and asks for results "following the exact structure and formatting" of it.
- **Pattern**: The scripts compute aggregates but never read or parse the template file; column names, column order, row order, and numeric precision are guessed from intuition, and raw floating-point sums are written verbatim (e.g. values with 10+ decimal digits or inconsistent decimal counts across rows), so the output can't byte-match the reference even when the underlying math is right.
- **Detection procedure**:
  1. In the task text, note whether a template/sample output artifact or explicit formatting rules (rounding, units, ordering, header names) are mentioned.
  2. Search the scripts for any read of that template (e.g. loading the sample file) and for any explicit formatting step (`round()`, `float_format`, sort key derived from the template, column list copied from the template header). Absence of both is the red flag.
  3. Inspect the produced answer: check that all numeric cells share a consistent, plausible precision, that header names/order match the template exactly, and that row order follows a rule that is stated or derivable from the template rather than an ad-hoc choice.
  4. Confirm no post-hoc comparison of the generated file against the template's shape/columns/dtypes was performed.
- **Discriminator**: A real violation is when the template is never inspected and the formatting choices (precision, ordering, header) are unverified guesses. It is *not* a violation if the script reads the template (or the task states the format literally) and demonstrably enforces the same columns, order, and rounding — even if it does so with hardcoded values copied from the template.
- **Consequence**: The grader's exact/tolerance file comparison fails on formatting alone (extra decimal digits, wrong row order, or mismatched headers), reporting the result file as WRONG despite arguably correct aggregation logic.

### Output feature matrix silently redefined and never sanity-checked against the requested spec
- **Applies when**: `task` -- the task asks for a result file whose columns are the per-record feature values (e.g., `Feature_i`) plus a derived label, and the script builds that file from an ad-hoc, self-chosen preprocessing pipeline.
- **Pattern**: The agent drops, imputes, one-hot expands and reorders columns for modeling convenience, then dumps that transformed matrix as the "feature vector" without checking that the saved file's row count, column count/order, value scale and label column are a defensible, non-degenerate representation of the original records; the choice of the number of clusters/labels is also taken straight from an unexamined metric on that arbitrary space (often collapsing to the minimum allowed value, e.g. 2, with a very low score) and never sanity-checked.
- **Detection procedure**:
  1. Read the task statement and note exactly what each output column is supposed to contain (which records, which feature values, what label) and where the file must be written.
  2. In the script, trace every transformation between loading the raw data and `to_csv`: dropped columns, imputations, encodings, scaling, row filtering — and check whether the written matrix is the raw feature values, the transformed ones, or an inconsistent mix, and whether row count still equals the number of input records.
  3. Check for any post-write validation: re-reading the file, asserting shape/column names/dtypes, label counts and no NaNs, and a check that the produced grouping is non-trivial (more than a minimal split, balanced enough, score meaningfully above the alternatives).
  4. Read the answer text: does it merely restate the pipeline and the selection metric, or does it show these validation numbers and justify the discarded/derived columns?
- **Discriminator**: A real violation is when transformations that change the feature vector's identity (dropping informative columns, expanding categoricals, imputation) are made without justification *and* no shape/content/degeneracy check on the saved file appears anywhere; it is *not* a violation if the script documents a principled feature definition and verifies the written file (rows = records, expected column naming/ordering, sensible label distribution), even if the exact preprocessing differs from a reference.
- **Consequence**: The graded file has a feature matrix and/or label column that cannot be matched to the expected output (wrong width, transformed values, trivial or collapsed clustering), so the file check fails outright even though the script ran without error.

### Never validating the output file against the provided submission template
- **Applies when**: `task` -- the task supplies a sample/template output file and asks for predictions written to a submission file in that exact format.
- **Pattern**: The scripts build a submission purely from the agent's own assumptions (hand-written column names, ids taken from an intermediate frame, predictions coerced/rounded to a chosen dtype) and never load the template to check column names, row count, id set/order, or value type; a later "improved" script may also be truncated or fail, silently leaving an earlier or partial file on disk.
- **Detection procedure**:
  1. Read the task/README for the named template file and any stated format or metric constraints (column names, one row per test id, continuous vs. integer targets).
  2. Search the scripts for a read of that template and for an explicit comparison against it (shape equality, `set(ids)` equality, column-name equality, ordering, null check) before writing the output.
  3. Check that the final produced file comes from the last script that ran to completion, and that any post-processing (rounding, clipping, casting) is justified by the stated metric rather than applied by default.
  4. Inspect the answer itself: count rows and compare with the expected test size, confirm the header matches the template, and confirm no duplicate/missing ids.
- **Discriminator**: A real violation is the absence of any template-based verification *and* an output whose shape/ids/dtype cannot be shown to match; it is fine if the script derives ids directly from the template or test file and asserts shape/id agreement (even without printing), or if rounding is explicitly required by the task.
- **Consequence**: The grader reports the submission as WRONG/MISSING — row count or id set mismatched, wrong header, or unnecessary integer rounding inflating the error metric — even though the model itself may be reasonable.

### Arbitrary feature subsetting when the task implies the full feature vector
- **Applies when**: `task` -- the task asks to run an unsupervised/modeling procedure on "the dataset" and to output the feature vector alongside results (e.g., columns named `Feature_i`), without naming which columns to use.
- **Pattern**: The script silently keeps only a handful of hand-picked columns (often a thematically related few, or only those with no missing values) instead of using all usable columns after standard cleaning, so the emitted feature matrix has far fewer dimensions than the data provides and the labels come from a different feature space than the reference solution.
- **Detection procedure**:
  1. Read the task for any statement restricting the columns; if none, the default is "all informative columns after cleaning" (drop pure identifiers/text keys, coerce numeric strings, encode or drop categoricals, impute or drop missing values in a documented way).
  2. In the script, find the column-selection step and count how many columns enter the model; compare with the number of candidate columns in the raw file.
  3. Check whether each exclusion is justified by a stated rule (non-numeric, identifier, all-missing) rather than by topical judgment or convenience; check whether missing values were dropped/imputed instead of used as a reason to discard whole columns.
  4. Compare the answer's reported feature count/columns (`Feature_0..Feature_k`) against the expected dimensionality from step 2 and against the row count of the output.
- **Discriminator**: A real violation is dropping numeric, mostly-populated columns for subjective reasons (or to avoid handling NaNs/dtype cleanup); it is fine to drop columns that are identifiers, codes, free text, or unparseable, or to reduce dimensionality by an explicit method the task allows, when the reasoning is stated and applied uniformly.
- **Consequence**: The output file's schema (number of `Feature_i` columns) and the cluster assignments diverge from the reference, so an exact/структural file comparison fails even though the pipeline ran without error.

### Silently dropping entities that must appear in the required output file
- **Applies when**: `task` -- the deliverable is a per-record/per-entity output file (labels, predictions, scores) covering the units derived from the input data, and the script applies filtering or aggregation before producing it.
- **Pattern**: The script removes a substantial subset of the entities as a side effect of preprocessing or feature construction (e.g. dropping rows where a derived statistic is undefined/NaN, requiring a minimum count, trimming "outliers"), so the saved file has far fewer rows than the natural population of entities in the cleaned data — and the answer never reconciles this row count against the input.
- **Detection procedure**:
  1. From the task, identify the unit of the requested output file and the expected row count implied by the cleaned input (e.g. number of distinct entities after only the clearly-justified cleaning steps).
  2. In the scripts, list every filter/`dropna`/threshold applied *after* the entity table is built, and note how many units each removes; check whether any is driven purely by a self-created feature being undefined rather than by data validity.
  3. Compare the row count written to the output file with the count from step 1; also check the column naming/indexing convention matches the literal spec (e.g. `Feature_0…` vs `Feature_1…`, presence/absence of an ID column, ordering).
  4. Check the answer text: does it state the final row count and explain the gap, or does it present coverage loss as a methodological choice without validating it against the requirement?
- **Discriminator**: A real violation is dropping units that are valid members of the target population purely for algorithmic convenience (undefined std, "needs ≥2 events", outlier removal) with no instruction to do so; a look-alike that is fine is removing records that cannot be attributed to any unit at all (missing entity key) or that the task/README explicitly designates as invalid (cancellations), where the remaining set is still the full population.
- **Consequence**: The saved file has the wrong shape/coverage (thousands of entities missing) and, if the grader compares row counts, column names, or per-entity labels, it fails the file check outright even though the clustering itself may be reasonable; degenerate singleton clusters from unremoved extremes are a further symptom that no sanity check on cluster sizes was run.

### Output file schema/row-alignment not verified against the required submission spec
- **Applies when**: `task` -- the task asks for predictions (or derived values) written to a named file with an explicitly stated column name, one row per input record.
- **Pattern**: The agent writes the file with its own convention — extra index/key columns, renamed or extra headers, a different row count than the test input, or rows in an order other than the input's — and reports only model quality statistics, never checking that the artifact matches the literal requested format.
- **Detection procedure**:
  1. From the task statement, extract the exact required artifact: filename, required column name(s), expected number of rows (= rows in the provided test input), and implied row order.
  2. In the scripts, find the write step (e.g., `to_csv`) and check what DataFrame is written: which columns it contains, whether the header spelling matches exactly, whether an index is written, and whether the frame was built from the full, unfiltered, un-reordered test input (no dropped rows from NaN handling, no re-sorting, no deduplication).
  3. In the answer, look for an explicit verification of shape/columns against the test file (row count equality, column list equality, no NaNs); absence of such a check, or a reported row/column set that differs from the input's, is the flag.
  4. Cross-check the reported prediction count and column list against the test input's row count and the required header.
- **Discriminator**: A real violation is a mismatch in the artifact itself — missing/renamed/extra columns, row count ≠ test rows, or rows realigned by dropping/sorting. It is *not* a violation if extra columns are explicitly permitted by the task, or if the agent kept an auxiliary column but verified counts and order and the required column is present and correctly named; nor is strong/weak model accuracy relevant here.
- **Consequence**: The grader loads the file, fails schema or index alignment (or compares mismatched rows), and marks the result wrong/missing regardless of how good the underlying model was.

### Unverifiable, artifact-incomplete plotting claims (no saved script, no data dump)
- **Applies when**: `task` -- the task asks for a chart/output built to a spec file and the grading depends on the underlying plotted data being recoverable (e.g., a serialized plot description and/or a numeric array), not just an image file.
- **Pattern**: The agent reports a checklist of "spec met ✓" items and an image file, but leaves behind no runnable script and no machine-readable dump of the plotted series; the numbers quoted appear to be typed in or eyeballed rather than aggregated from the source file, and the auxiliary artifacts the grader reads are never written.
- **Detection procedure**:
  1. Read the task/spec and list every expected output artifact (image, JSON/spec dump, `.npy`/CSV of plotted values) and every constrained property (labels, ticks, order, units, aggregation level).
  2. Check the saved scripts: is there code that loads the raw source, performs the stated aggregation, plots, and then explicitly serializes the plotted x/y values and figure metadata to the required artifact names?
  3. Compare the numbers in the answer to the aggregation the script would produce — do they trace to a computed object, or are they hard-coded/suspiciously patterned (e.g., a value equal to the group label, implausibly small counts for the stated unit)?
  4. Confirm the answer asserts nothing that isn't demonstrably read back from the produced artifacts (e.g., re-reading the dump and printing shape/range).
- **Discriminator**: A real violation is when required non-image artifacts are absent or the plotted values cannot be reproduced from code over the raw data; a look-alike that is fine is a script that computes and dumps the series programmatically and merely *summarizes* it in prose, even if only the image is visually inspected.
- **Consequence**: The grader's checks on the expected artifacts (plot spec JSON, numeric array) report WRONG/MISSING, and any hard-coded series that differs from the true aggregation fails value comparison — scoring 0 despite a confident "task completed" report.

### Prescribed resampling procedure replaced by a different (more "extreme") test
- **Applies when**: `task` -- the task/README explicitly dictates the inference procedure (e.g., shift each group to a common mean, then bootstrap each group separately and count replicates at least as extreme as the observed statistic) and the answer is a single p-value written to a template file.
- **Pattern**: The attempt computes a p-value with a substitute method (parametric t-test, label-permutation test, pooled single-sample bootstrap, resampling only one group, or a one-sided/two-sided convention different from the one implied), and/or reports a Monte-Carlo p-value at or below the resolution floor of its replicate count (e.g., a handful of hits out of 10^5 draws) without checking that the number is stable or plausible.
- **Detection procedure**:
  1. From the task/README, write down the required steps: what statistic is observed, how the null is imposed (shifting vs. relabeling), which units are resampled, replicate count, and the tail definition.
  2. In the script, locate the null-generation code and check line-by-line that each group is resampled from its own shifted copy (not pooled/permuted), that the observed statistic is recomputed identically, and that the extremity comparison matches the stated tail and sign.
  3. Check the reported value against the simulation's granularity (p ≥ 1/replicates) and rerun-stability with a different seed; also compare order of magnitude to what the prescribed method plausibly yields versus a parametric shortcut.
  4. Verify the output file exists with the exact column name/format/precision from the template, containing the final p-value (not a count, a test statistic, or an unrounded scientific-notation surprise) — and that a script was saved so the procedure is reproducible.
- **Discriminator**: A real violation is a null model or tail convention structurally different from the one specified (pooling/permuting instead of shifting, resampling one group, wrong comparison direction), or a p-value whose magnitude is an artifact of too few replicates; it is *not* a violation if the prescribed procedure is implemented and the p-value merely differs in the last digits due to random seed, provided it is well above 1/replicates and formatted as requested.
- **Consequence**: The written file's p-value differs from the reference by orders of magnitude (or fails tolerance/format matching), so the result-file check fails even though the pipeline "ran successfully".

### Ignoring provided specification/config files and their required output artifacts
- **Applies when**: `task` -- The prompt points to auxiliary instruction or configuration files (e.g., a tips/notes file, a YAML/JSON spec) that define methodology, styling, and/or required deliverables.
- **Pattern**: The agent never opens or parses those files; it guesses the methodology and plotting parameters from the task wording, hard-codes its own choices (variable selection, grouping, labels, colors, figure size, axis ranges), and produces only the single obviously-named output while skipping companion artifacts the spec implies (serialized plot data, arrays, config echoes).
- **Detection procedure**:
  1. List every instruction/config file and every output artifact named or implied by the task statement.
  2. Search the scripts for code that reads each of those files (open/read/yaml.safe_load/json.load) and for code that writes each expected artifact.
  3. Check whether formatting/aggregation choices in the plotting or computation code are traceable to values loaded from the config, or are literals invented by the agent; check whether the answer text quotes the actual file contents versus paraphrasing assumptions.
  4. Verify the answer reports the same set of deliverables the task requires, not just one.
- **Discriminator**: A real violation is when no code path ever reads the spec file(s) or when required outputs are missing entirely; it is fine if the agent read the files and then legitimately inlined their values (evidence: printed/quoted contents, parameter names matching the spec) and produced all requested artifacts.
- **Consequence**: Graders comparing each expected artifact fail on missing files, and the produced figure mismatches the reference on styling and on the underlying aggregated series, giving 0 passed checks.

### Ignoring a task-referenced specification file that defines the required method
- **Applies when**: `task` -- The prompt points to an external document/notes file (e.g. a markdown/readme/spec in the working directory) that defines how to bin, group, filter, or compute the requested quantity.
- **Pattern**: The scripts never open or echo the referenced file; the agent instead assumes the data's own native categories/defaults (e.g. pre-existing bucket labels or an ad-hoc ordering it invents) and produces outputs whose granularity, labels, or aggregation cannot be verified against the stated method.
- **Detection procedure**:
  1. List every external file, rule, or convention the task text references, plus every explicitly required output artifact/name.
  2. Grep the scripts for a read/print of each referenced file; confirm the derived categories/groups in the code are traceable to that file's content rather than to `value_counts()` of a raw column.
  3. Check the answer for evidence the specified method was applied (group boundaries/labels quoted from the spec) and that all required output files were written to the expected location.
  4. If the spec was never loaded, or produced groups differ in number/boundaries from anything in the spec, flag it.
- **Discriminator**: Fine if the script demonstrably reads the spec (or quotes its rules) and the resulting groups match it — even if the raw column happens to be pre-binned; a violation is using dataset-native or self-invented categories with no reference to the mandated definition.
- **Consequence**: Aggregated counts/labels differ from the reference binning, so the saved figure and any numeric arrays mismatch the expected artifacts and all checks fail, despite the plot having correct title/axis labels.

### Answer not persisted to the expected artifact with the exact requested schema
- **Applies when**: `task` -- the prompt shows a literal output template (e.g., keys whose values are bracketed lists) and/or the grading setup expects a specific result file produced by the agent's code.
- **Pattern**: The attempt computes a plausible-looking value but only reports it in prose/chat, and/or reshapes the template (scalars instead of lists, renamed/extra/missing keys, added units or rounding not asked for), leaving no saved script or result file that a grader can read.
- **Detection procedure**:
  1. Re-read the task and write down the exact required deliverable: file name/location (if any), key names, and the value type shown in the template (list vs scalar, string vs number).
  2. Search the scripts for the code that serializes the final answer (e.g., a dump/write to the named file); if no script or no write step exists, the deliverable is missing.
  3. Diff the submitted object against the template key-by-key and type-by-type; confirm every key is present, spelled identically, and wrapped in the same container type.
  4. Check that the reported number is the final requested quantity in the requested units/precision, not an intermediate or reformatted variant.
- **Discriminator**: A real violation is a structural mismatch (no file written, scalar where a list is shown, renamed/absent keys); harmless look-alikes are cosmetic differences the template does not constrain, such as key ordering, whitespace, or a value that is genuinely scalar because the template shows a scalar.
- **Consequence**: The grader finds the expected result file missing or fails schema/type comparison, marking the submission wrong even if the underlying computation was right.

### Skipping input-integrity validation before building a derived/cumulative series
- **Applies when**: `task` -- the script reads a raw table and immediately aggregates it (weighted sums, compounding, cumulative products) into an output series that is graded against exact expected values.
- **Pattern**: The attempt loads the file and jumps straight to arithmetic without checking for missing/blank cells, non-numeric dtypes, duplicate or unsorted keys, extra/renamed columns, or the scale/units of the values (e.g., fractions vs. percentages, levels vs. period-over-period changes). Any NaN, string column, off-by-100 scaling, or first-row convention silently propagates through the cumulative operation and corrupts every downstream row, and the attempt's "verification" script only re-derives its own numbers instead of testing them against an independent expectation.
- **Detection procedure**:
  1. Read the task/README for what the input columns are supposed to represent (units, scale, whether a base/first row exists) and what the output must contain (column names, ordering, index, rounding, row count).
  2. Scan the scripts for explicit integrity checks before the aggregation: `isna().sum()`, dtype checks, min/max or magnitude checks, row-count/date-continuity checks, handling of the first period. If aggregation happens with none of these, flag it.
  3. Check whether the "verification" step compares to anything independent (a known ground-truth magnitude, a hand-computed row, a second method) or merely recomputes the same formula.
  4. Inspect the reported output: are the values in a plausible range and magnitude for the stated quantity, is the row count/first row consistent with the input period, and do the column names/order match the required format exactly?
- **Discriminator**: A real violation is when nothing in the pipeline could have detected a NaN, a mis-scaled column, or a wrong first-row/base convention, and no external cross-check exists. It is *not* a violation if the script explicitly inspects/handles missing values and units (or documents that the data is clean after checking) and validates at least one output value against an independent reference, even if the code is otherwise terse.
- **Consequence**: The saved file has the right shape and looks superficially reasonable, but every value is systematically off (shifted, scaled, or NaN-contaminated from the first defective row onward), so an exact/tolerance comparison against the expected file fails on all checks.

### Silently substituting a random subsample (or otherwise altered input) for the statistic the task specified
- **Applies when**: `task` -- the task names a specific statistical test/metric to run on a specified column or dataset, and the script contains a branch that resamples, truncates, or otherwise changes the input before computing it (often to dodge a library size limit or runtime cost).
- **Pattern**: The script computes the mandated test on a random subset (frequently with no fixed seed) or a differently-filtered set than the one used for the other reported statistics, then reports the resulting p-value/decision as if it came from the full specified data — making the answer non-reproducible and potentially opposite to the intended result.
- **Detection procedure**:
  1. Read the task and note exactly which data (which column, which rows, all of them?) the named test/statistic must be computed on, and any stated thresholds.
  2. Scan the script for any `np.random.*`, `.sample(`, slicing, head/tail, dropna-with-different-scope, or size-based `if` branches sitting between data loading and the mandated computation; check whether a seed is set and whether the same subset feeds every reported number.
  3. Check whether the script prints diagnostics that let one verify the input to the test (n used, missing count, min/max/unique values) and whether the script would raise/flag rather than silently deviate when the library's constraints are hit.
  4. Compare the reported decision with the other reported numbers and with the data's nature (e.g., few distinct integer values, small n, mild moment values) — if the decision hinges on a resampled subset or contradicts the descriptive statistics, treat it as unverified.
- **Discriminator**: A real violation is a *silent, unrequested* change of the analysis input (or a nondeterministic one) affecting the reported figure; it is fine if the task itself authorizes subsetting/filtering, or if the full specified data is used and any subsetting is only for auxiliary plots/exploration and is seeded and documented.
- **Consequence**: The reported test decision (and its p-value) reflects a different, randomly varying sample than the ground-truth computation, so the categorical answer flips (e.g., "no" instead of "yes") and the grader scores 0 even when the accompanying descriptive statistics look plausible.

### Prediction file not verified against the test set (row count / alignment / persistence)
- **Applies when**: `task` -- the deliverable is a prediction file whose rows must correspond one-to-one, in order, with the rows of a provided evaluation input file.
- **Pattern**: The agent pastes predictions into the chat answer instead of (or in addition to) writing the required file, and never checks that the written file has exactly as many rows as the input, in the same order, with the exact required column name and no extra index column. Truncated, reordered, deduplicated, or dropped-row outputs (e.g., after dropping missing/blank text) go unnoticed.
- **Detection procedure**:
  1. Read the task for the required output filename, column name(s), and the input file that defines the number and order of predictions.
  2. In the scripts, confirm the file is actually written to the required path (`to_csv(..., index=False)`) with the exact header, and that predictions were produced from the full, unfiltered, unshuffled input (no `dropna`, `sample`, `head`, sorting, or partial-batch loop that would change row count/order).
  3. Look for an explicit post-write sanity check: re-read the file and assert `len(output) == len(input)`, header matches, and no nulls/unexpected label values.
  4. Inspect the final answer: if it consists of the predictions dumped as text (especially cut off mid-word) rather than a confirmation of a validated saved file, treat the artifact as unverified.
- **Discriminator**: A real violation is missing/unwritten file, or absent verification of row count, order, header, and label vocabulary. It is fine if the agent writes the file and shows a check of shape/head/value counts, even if it also prints a preview of the predictions in the answer.
- **Consequence**: The grader looks for the expected file and compares row-by-row; a missing, truncated, misaligned, or wrongly-headed file scores 0 regardless of model quality.

### Final artifact produced by an unvalidated "fallback" model that overwrites better-validated work
- **Applies when**: `task` -- the deliverable is a prediction/result file and the agent runs several successive scripts, each rewriting the same output file, with the last one being a simplified/faster variant.
- **Pattern**: Earlier scripts hold out data and report validation scores for stronger models, but the final script that actually writes the deliverable drops validation entirely, fits a much weaker/simpler estimator on all data, and silently overwrites the previous output; the answer reports only descriptive statistics of the predictions (mean/min/max) as if they were evidence of quality, with no held-out error estimate or comparison against the earlier candidates.
- **Detection procedure**:
  1. Read the task to identify the required deliverable file and the implied evaluation criterion (accuracy/error of the predictions, not merely file existence).
  2. Identify which script last writes that file, and check whether it (a) computes any held-out or cross-validated score for the exact model/pipeline whose predictions are saved, and (b) is at least as strong as models validated in earlier scripts.
  3. Check the reported answer for a quantitative generalization metric tied to the shipped predictions and an explicit reason for choosing that model over the alternatives.
  4. Sanity-check the shipped predictions against the training target's distribution/range and the required id count, ordering, column names, and output path.
- **Discriminator**: A real violation is when the shipped model was never scored on unseen data, or was demonstrably weaker than an already-validated alternative and the switch is justified only by speed/convenience. It is fine if a simpler model is shipped after being compared on the same validation protocol and shown to be competitive, or if refitting a validated configuration on the full data without re-scoring.
- **Consequence**: The submission file is well-formed but its predictive quality falls below the grader's threshold (poor R²/high error versus the hidden labels), so the deliverable is marked wrong despite the run "completing successfully."

### Ranked "top-N" lists produced without applying the stated qualification rule and rank ordering
- **Applies when**: `task` -- the task asks for the top N entities by some aggregate statistic, and the prompt/README supplies a definition, eligibility threshold, or tie-breaking/ordering convention for that statistic.
- **Pattern**: The attempt groups rows and sorts by a raw aggregate without honoring the stated qualification rule (e.g., minimum number of underlying records per entity, deduplication of repeated items, numeric parsing/cleaning of the value column), and/or emits the N names in an order that is not the ranking order (alphabetical or arbitrary), so ties at the ceiling value are resolved incorrectly.
- **Detection procedure**:
  1. Read the task/README and write down every constraint attached to the ranking: how the aggregate is defined, any eligibility filter, how ties are broken, and whether row 0 must be the top-ranked entity.
  2. In the scripts, locate the groupby/aggregation and check for (a) cleaning/casting of the ranked column, (b) an explicit filter implementing the eligibility rule, (c) `sort_values(..., ascending=False)` on the aggregate with a defined tie-breaker before `head(N)`.
  3. Inspect the answer file: check whether any column's entries are in alphabetical order or whether the top entries look like one-off/low-volume entities (a sign the eligibility filter and rank ordering were skipped).
  4. Confirm the emitted aggregate values were sanity-checked (range, count of rows per entity, N rows exactly, column names/ordering matching the sample format).
- **Discriminator**: A real violation is when no code implements the stated threshold/ordering, or the output order cannot be reproduced by sorting on the aggregate; a look-alike that is fine is a list that merely *happens* to be near-alphabetical because genuine ties were broken by a documented, implemented rule.
- **Consequence**: The exact-match check on the saved file fails because the entity set and/or their row positions differ from the reference ranking.

### Fabricating input data instead of using the provided dataset
- **Applies when**: `task` -- the task refers to a supplied dataset (with a README/spec) and the agent's scripts include a step that generates, simulates, or hard-codes the input data.
- **Pattern**: The agent cannot find or fails to load the real input file, so it synthesizes a random/placeholder table with invented columns and value ranges, then runs the full analysis on that fake data and reports the resulting numbers as if they came from the real source. Related symptom: only a subset of required output artifacts is produced, and stated spec files are only partially honored.
- **Detection procedure**: 1. Read the task and note which input files and which output artifacts (files, formats, names) are expected. 2. Scan every script for data creation calls (`np.random.*`, `range(...)` fillers, manual dicts/lists written to the input path, `seed`) or writes to the same path later read as input. 3. Check whether the script instead locates and loads the real provided file (and would fail loudly if absent) and whether all required outputs are written. 4. Cross-check the answer's reported counts/ranges against the dataset description (row counts, units, realistic value ranges) and against the mentioned spec/config keys.
- **Discriminator**: A real violation is when the analytical result reported comes from data the agent itself invented, or when the real file was never read. Legitimate look-alikes: creating tiny synthetic fixtures for unit-testing a plotting/utility function while the reported result still comes from the real dataset, or augmenting real data in a documented, task-sanctioned way.
- **Consequence**: All value-based and file-based checks fail: derived arrays/JSON summaries and the figure encode arbitrary random counts (and required artifacts may be missing entirely), so the grader reports 0 of the expected outputs correct.

### Empty-subset result reported as prose instead of the required answer format
- **Applies when**: `task` -- a task prescribes a specific answer template (e.g. `@name[value]`) for a statistic computed on a filtered subset, and the filter may match zero rows.
- **Pattern**: The agent discovers the filter yields no rows (or an undefined statistic) and replaces the required formatted answer with an explanatory sentence ("no data available", "filter value does not exist"), instead of emitting the statistic's defined degenerate value (NaN/empty) inside the requested format.
- **Detection procedure**:
  1. Read the task and note the exact required output token/format and rounding rules.
  2. Read the scripts for a branch that short-circuits when the filtered frame is empty (or when the aggregate returns NaN) and prints a message rather than the formatted value; check whether the filter/dtype (e.g. string vs numeric key) was verified before concluding the subset is empty.
  3. Compare the submitted answer against the required template: does it literally contain the token and a value slot?
  4. If it does not, flag it — an empty subset is a valid result to report, not a reason to abandon the format.
- **Discriminator**: A real violation is any answer that abandons the template; it is fine to report an unusual value (NaN, 0, empty) *inside* the template, and it is also fine to note the emptiness as extra commentary alongside a properly formatted answer. Also not a violation if the emptiness stemmed from a genuine miscoded filter that, once fixed, yields data — that is a different (filtering) error.
- **Consequence**: The grader's field-by-field check finds no parsable value for the requested key, so the expected result (including NaN) is scored WRONG/MISSING and the task fails at 0/1.

### Unverifiable answer: entity names/values not traced back to the provided data via runnable code
- **Applies when**: `task` -- the task asks for specific records (top/bottom-N names, IDs, categories) or statistics to be extracted from a supplied data file after a prescribed preprocessing step.
- **Pattern**: The submission presents a plausible-looking list that reflects general/world knowledge or an unsaved ad-hoc computation, with no script that (a) loads the given file, (b) applies the stated preprocessing (e.g., the specified imputation), (c) sorts by the requested field in the stated direction, and (d) writes the answer to the required output file; entity labels are also not copied verbatim from the data's key column.
- **Detection procedure**:
  1. Read the task and note the required output artifact/format, the mandated preprocessing, and the ordering/sorting constraint for *every* requested list.
  2. Look for a script that reads the provided file and prints/dumps exactly the reported values; if no script exists, or the script's output was never shown to match the submitted answer, treat the answer as unverified.
  3. Cross-check each reported label against the data's identifier column spelling/format (e.g., official vs. colloquial names, punctuation, abbreviations) and check that each requested list is ordered as instructed, not by a default or intuitive order.
  4. Sanity-check the numbers behind the picks (count = N, values within plausible range, no rows dropped instead of imputed) — if the underlying values are not reported at all, that is itself a flag.
- **Discriminator**: A real violation is an answer whose values cannot be reproduced from the file by any shown code, or whose labels differ from the dataset's own strings/ordering rule; a look-alike that is fine is an answer that happens to match common knowledge *but* is accompanied by a script whose printed output and written result file match it exactly and whose labels are taken verbatim from the data.
- **Consequence**: The grader's exact comparison against the expected result file fails on missing/misnamed entities, wrong list order, or a missing output file, scoring 0 even though the list looks superficially reasonable.

### Answer string not emitted in the literal template (verbatim tokens, quoting, order) with no reproducible script backing it
- **Applies when**: `task` -- The task specifies an exact answer template with named tags, delimiters, and example value formatting, and expects the values to be produced by saved, re-runnable analysis code.
- **Pattern**: The agent computes conceptually correct values but serializes them in a variant of the requested template — dropping the quotation marks/units shown in the example, renaming or reordering tags, changing separators, adding prose around the tags — and/or leaves no script that prints the final answer string, so nothing ever validated the emitted text against the template. A literal string-matching grader then fails every field even though the underlying analysis was right.
- **Detection procedure**:
  1. Copy the answer template exactly as given in the task, including every quote, bracket, tag name, separator, and the formatting shown in any example value.
  2. Check the scripts: is there a step that constructs and prints the final answer string, so the submitted text is a program output rather than hand-typed? If no script is saved at all, the answer is unverifiable by construction — flag it.
  3. Diff the submitted answer character-by-character against the template: tag names and order, presence/absence of quotes around each value, capitalization and spelling of the allowed value vocabulary, and the exact form of any range/unit string.
  4. Confirm each value is one of the permitted options and is the final requested quantity, not an intermediate.
- **Discriminator**: A real violation is any character-level deviation from the specified template (missing quotes, extra text, altered tag names/order), or an answer with no code that produced it; a look-alike that is fine is a template where the task itself shows the value unquoted or explicitly allows whitespace variation, and the answer matches that shown form exactly.
- **Consequence**: A regex/exact-match grader reports every field as WRONG/MISSING even though the submitted values are semantically identical to ground truth, yielding a 0/N score with no partial credit.

### Substituting proxy data/entities for the ones the task names
- **Applies when**: `task` -- the prompt asks for specific entities, groupings, or measures (e.g., "top N of X by Y, broken down by stage Z") and the scripts must locate those fields in the provided data files.
- **Pattern**: The agent cannot find the requested fields in the files it opened, so it silently swaps in a different data source and re-interprets each requested concept as a loose "analogy" (different grouping key, different ranking measure, different segment definitions), then declares success instead of resolving the mismatch.
- **Detection procedure**: 1. List from the task the exact entity to rank, the ranking measure, and the segments/values to plot. 2. In the scripts, identify which file and which columns supply each of those three items. 3. Flag if any is drawn from a different domain or is described in the answer as an "analogy"/"proxy"/"equivalent", or if the reported category labels are not instances of the requested entity type. 4. Check the answer's reported units/axis labels match the requested measure (e.g., a duration in days, not a count or probability).
- **Discriminator**: A real violation is using a different variable or dataset than the task names because the requested one was not found; acceptable look-alikes are cases where the requested concept genuinely exists under a differently-spelled column name and the mapping is stated and verifiable (same semantics, same units).
- **Consequence**: Every derived artifact (figure, saved arrays, config-driven outputs) encodes the wrong categories and wrong quantities, so all value- and label-level checks fail even though a chart of the right visual type was produced.

### Answer-format template not reproduced literally (delimiters/separators dropped)
- **Applies when**: `task` -- the prompt specifies an exact answer string template (a tag, an `=` or `:` separator, and specific wrapping brackets/braces) for reporting one or more computed values.
- **Pattern**: The attempt computes the values correctly but emits them in a paraphrased wrapper — omitting or substituting the required leading/trailing delimiters, the key-name-to-value separator, or the bracket nesting — so an exact-match grader fails even though the analysis is right.
- **Detection procedure**:
  1. Copy the answer template exactly as given in the task and mark every literal token: tag name, `@`/`=`/`:` separators, and each opening/closing bracket or brace in order.
  2. Read the scripts (or final message construction) and check whether the output string is built from that literal template — e.g. an f-string/print that hard-codes the tag and delimiters — rather than from a bare `dict`/`list` repr.
  3. Compare the submitted answer token-by-token against the template: same tag, same separator, same bracket sequence and nesting, same key spelling/order.
  4. Flag if any literal token is missing, added, or swapped, even when the numeric values look plausible.
- **Discriminator**: A real violation is a structural mismatch in the required literal tokens (missing `=`, wrong bracket type, missing wrapper, renamed/reordered keys); harmless look-alikes are cosmetic differences the template does not constrain, such as inner whitespace, quote style, or int-vs-float rendering of the same value.
- **Consequence**: The grader reports the expected variable as WRONG/MISSING and scores 0 despite numerically correct values, because it cannot parse the submitted string against the required pattern.

### Unverified prediction file (no reproducible script, no shape/format sanity check)
- **Applies when**: `task` -- the deliverable is a prediction/output file with a specified column name and one row per record of a held-out input file.
- **Pattern**: The attempt produces the output file without any saved, runnable script that reads the held-out input, fits on the training portion, and writes predictions — and never checks that the written file has exactly the required column name, the same number of rows as the held-out input, the original row order, and values in a plausible range/dtype for the target.
- **Detection procedure**:
  1. Read the task statement and note the exact required file name, column name(s), row count (from the held-out input), and any ordering/rounding/units constraints.
  2. Look for a script that is complete end-to-end (load train + held-out data → preprocess → fit → predict → write file); if scripts are absent or only partially cover this path, the result cannot be reproduced or audited and must be treated as unverified.
  3. Inspect the produced file's header and shape: does it contain the requested column spelled exactly as asked (no extra index column, no renamed/extra columns), and does its row count equal the held-out input's row count?
  4. Check value sanity: dtype numeric (or as required), no NaNs/empty cells, values inside the target's observed range, and non-constant/non-degenerate predictions aligned to the input row order.
- **Discriminator**: A real violation is missing/unreproducible code *or* a file whose header, row count, ordering, or value range does not match the specification. A look-alike that is fine: a script that differs stylistically or uses a simple baseline model, but demonstrably writes the exact requested column, one row per held-out record in input order, with valid values.
- **Consequence**: The grader compares the submitted file against the expected schema/row alignment and marks it WRONG/MISSING (0 checks passed) even though a file with the right name exists.

### Feature scaling omitted before distance-based clustering, yielding degenerate outlier clusters
- **Applies when**: `task` -- the task asks for an unsupervised grouping (k-means/hierarchical/DBSCAN-style) of records whose numeric columns have wildly different units and ranges, and the deliverable is a label file.
- **Pattern**: The attempt feeds raw, unstandardized columns straight into a distance-based algorithm (and/or picks *k* by a score computed on those raw features), so one or two large-magnitude columns dominate the distance metric; the resulting partition contains singleton or near-singleton clusters that merely isolate extreme values, while the bulk of the records collapse into one or two huge groups. The attempt reports this as the "optimal" solution without a sanity check on cluster sizes or on whether the grouping reflects all features.
- **Detection procedure**:
  1. In the task/README, note the feature columns and their plausible scales (percentages, per-capita monetary values, rates) — flag if magnitudes differ by orders of magnitude.
  2. In the scripts, check whether a scaler/normalizer (or a distance metric that is scale-invariant) is applied to the feature matrix *before* fitting and before any cluster-count selection metric; also check that model selection uses the same transformed space.
  3. In the reported answer, inspect the cluster size distribution and cluster descriptions: singleton/2–3-member clusters described as "outliers" or clusters characterized by a single dominant variable are strong evidence of unscaled distances.
  4. Confirm the saved label file was actually verified (row count, column names, label range) rather than only asserted in prose.
- **Discriminator**: A real violation is scaling never applied (or applied only after clustering / only for plots) together with a lopsided partition dominated by high-variance columns. It is *not* a violation if the script standardizes (or the algorithm/metric is scale-free) and small clusters are then justified by inspection — genuinely extreme records can legitimately form small clusters in a properly scaled space.
- **Consequence**: The submitted label column encodes an outlier split rather than the intended socio-economic-style grouping, so the expected output file comparison (cluster structure/agreement with reference labels) fails even though the file format looks plausible.

### Stated scope/filter in the task is not reflected in the computed statistic (wrong axis/subset)
- **Applies when**: `task` -- The question restricts the statistic to a specific slice (a given year, group, region, or subset of columns/rows) and the scripts compute an aggregate statistic per entity.
- **Pattern**: The script ignores the stated restriction and aggregates over the full set of columns/rows (e.g., every period rather than the specified one), or aggregates along the wrong axis, so the reported ranking answers a different question than the one asked. Extra "verification" code re-checks the same wrong computation, giving false confidence.
- **Detection procedure**:
  1. From the task text, list every explicit scope word (year, subset, grouping, units, estimator/definition flag) that must appear as a filter or parameter in the code.
  2. Read the script and locate where each scope word is applied: is there a selection of the specific column/rows before the statistic, and is the statistic taken along the axis implied by the question?
  3. If a scope word has no corresponding filter (e.g., all period columns are passed into the per-row statistic, or all rows into a per-column statistic), flag it; also check whether the resulting sample size/shape is what the stated slice would produce.
  4. Confirm the sanity check: does the intermediate printout show data only from the requested slice, and is the final reported quantity the one named in the question (not an intermediate or differently-scoped one)?
- **Discriminator**: A real violation is when the requested slice is never selected anywhere in the pipeline (or is selected but then discarded/overwritten). It is *not* a violation when the slice is implicit because the loaded file/frame is already restricted to it, or when the statistic legitimately requires the broader sample and the slice only defines the grouping key — in those cases the code should still show an explicit filter or a comment plus a shape/count check matching the slice.
- **Consequence**: The ranking/argmax is computed over the wrong sample, so the reported entity differs from the ground truth and the grader marks the single expected value as WRONG, even though the estimator flag and output format are correct.

### Reformatting an identifier to a coarser granularity than the data (losing precision in the reported key)
- **Applies when**: `task` -- The task asks to identify a specific record/key (a date, ID, category) from row-level data and the answer template shows a format string that is coarser or ambiguous relative to the data's granularity.
- **Pattern**: The script correctly locates the extremum/target row, but then applies a formatting/truncation step (e.g., `strftime` to a shorter pattern, rounding, taking a prefix/substring) that discards the identifying detail, so the reported key no longer uniquely designates the row actually used for the downstream computation.
- **Detection procedure**:
  1. Read the task: determine the granularity of the entity being identified (one row / one timestamp / one ID) and note whether the downstream calculation depends on that exact row.
  2. Read the script: find where the identified key is converted for output; check whether the emitted string contains strictly less information than the key stored in the data.
  3. Compare the emitted key against the value used internally for the dependent computation — if the dependent computation uses the full-precision key but the answer reports a truncated one, flag it; prefer reporting the key exactly as it appears in the source (or, if the template is genuinely ambiguous, report the full-precision value rather than truncating).
  4. Check for collisions: would the truncated key match multiple rows in the data? If yes, it cannot be the intended unique answer.
- **Discriminator**: A real violation is when truncation destroys uniqueness or drops detail present in the source key while the rest of the answer depends on that detail. It is fine when the task explicitly aggregates at the coarser level (e.g., the maximum is computed over monthly aggregates) or when the source key itself has only that granularity.
- **Consequence**: The numeric part of the answer can be correct while the identifier check fails, giving a partial-credit/incorrect verdict (e.g., 1 of 2 checks passed).

### Missing/unverifiable output artifact — answer reported inline instead of written to the required file
- **Applies when**: `task` -- the task explicitly instructs that results be saved to a named output file (e.g., `result.csv`) with the computed statistic(s).
- **Pattern**: The attempt computes a single number and reports it in prose/chat, but no saved script or code path demonstrably writes the named file (with the expected column/row layout, precision, and the exact requested quantity); the artifact is absent, empty, or contains an intermediate value rather than the requested one.
- **Detection procedure**:
  1. Read the task and list every required deliverable: file name, location, and what it must contain (which statistic, how labelled, any rounding/format constraints).
  2. Search the submitted scripts for an explicit write call (`to_csv`/`open(...).write`/etc.) targeting that exact filename; if no scripts exist at all, the deliverable is unverifiable by definition.
  3. Trace which variable is written and confirm it is the final requested quantity computed on the specified subset/aggregation (not an intermediate, unfiltered, or differently-aggregated value), and that the file layout would parse as a table.
  4. Cross-check the reported number against the file-writing code: if the only evidence is a bare float in the answer text, flag it.
- **Discriminator**: A real violation is when no reproducible code writes the named artifact, or the artifact holds a different quantity/format than requested; a look-alike that is fine is a script that writes the correct file under the right name and merely also echoes the value in the answer text.
- **Consequence**: The grader looks for the named result file and finds it missing or containing a mismatched value/format, scoring 0 regardless of whether the number quoted in the chat happens to be close.

### Missing reproducible pipeline + unvalidated prediction file (shape / class-rate sanity check)
- **Applies when**: `task` -- the deliverable is a per-row prediction file for a supplied test set, especially with a stated cost/recall asymmetry between error types.
- **Pattern**: The agent produces the output file with no saved, re-runnable script that reads the test file, aligns rows, and writes the required column, and never checks that the emitted file has exactly one prediction per test row, in the original test-row order, with the exact header/values shown in the sample; it also never compares the predicted positive rate against the training base rate or the asymmetric-cost objective, so a near-all-negative (default-threshold, imbalance-collapsed) prediction is submitted uninspected.
- **Detection procedure**:
  1. From the task, note the required output format (header text, single column, allowed values) and the exact number of rows in the test file and in the provided sample output.
  2. In the scripts, look for code that (a) loads the test set, (b) predicts, and (c) writes the file preserving test-row order, plus an explicit assertion/print of `len(predictions) == len(test)` and `value_counts()`; absence of any saved script is itself a failure of verifiability.
  3. Compare the answer file: count rows (excluding header), confirm header string and value domain match the sample, and compute the fraction of positive predictions.
  4. Compare that fraction to the minority-class prevalence in training and to the stated cost preference (e.g., missed positives more costly than false alarms); flag if it is far below prevalence or if any threshold/class-weight decision was never justified or validated on a held-out split.
- **Discriminator**: A genuine violation is a file whose row count/format cannot be shown to match the test set, or a positive rate materially below the training prevalence with no validation evidence (no held-out recall/F1/cost comparison, no threshold tuning). It is *not* a violation if the script asserts shape and ordering, the sparse positive rate is backed by held-out metrics on an imbalanced problem, and the format matches the sample exactly.
- **Consequence**: The grader compares the submitted file row-by-row against the reference; a wrong row count, wrong header/order, or a degenerate mostly-negative prediction fails the file check outright (0/1) even though the answer "looks like" valid predictions.

### Answer produced by dumping a Python object instead of the requested answer format
- **Applies when**: `task` -- the task prescribes an exact answer string/format (e.g., `@name[list_of_strings]`) and the script writes the answer by interpolating a Python variable (list, array, Series, dict) into the output.
- **Pattern**: The script computes the right values but emits them via Python's default `repr`/`str` (brackets, single quotes, `np.float64(...)`, `dtype=` noise, index labels), so the submitted string does not match the required token/delimiter/quoting convention, and no step normalizes or validates the final string.
- **Detection procedure**:
  1. Read the task and write down the exact target answer syntax, including delimiters, quote style, and whether names should be bare or quoted.
  2. In the script, find the line(s) that build the final answer and check whether the values are explicitly formatted (e.g., `", ".join(map(str, items))`) or just interpolated as a container (`f"...{mylist}"`, `print(df[...])`).
  3. Mentally render the produced string for a plausible result and compare character-by-character with the required format; also check the values are plain Python scalars/strings, not numpy/pandas objects.
  4. Confirm the script (or agent) does a final check that the emitted string matches the requested pattern; absence of such a check plus raw container interpolation is a violation.
- **Discriminator**: A real violation is when the answer text carries container/object syntax or quoting not sanctioned by the requested format; it is fine if the container's default rendering happens to coincide exactly with the requested syntax, or if the script explicitly joins/formats elements into the prescribed pattern.
- **Consequence**: The grader compares the submitted string to the expected one and reports WRONG/MISSING even though the underlying computation and values are correct.

### No held-out validation — model quality judged only by in-sample (resubstitution) performance
- **Applies when**: `task` -- The task asks for predictions on an unlabeled test set and the script fits a model on all labeled data and reports performance metrics.
- **Pattern**: The script trains on 100% of the labeled rows and then computes accuracy/other metrics by predicting on those same training rows (or on data used for fitting imputers/encoders), presenting that number as the model's quality; no cross-validation or hold-out split is ever produced, so the reported score is an optimistic upper bound with no bearing on the graded test predictions. Hyperparameters (depth, estimators, encoding choices) are likewise picked with no evidence, and no alternative configuration is compared.
- **Detection procedure**:
  1. Read the task to confirm the deliverable is scored on unseen test predictions against a hidden ground truth (usually with an implicit accuracy threshold).
  2. In the script, locate the `fit` call and the data passed to the metric call; check whether the evaluation set is disjoint from the fitting set (`train_test_split`, `cross_val_score`, or a K-fold loop) — flag if `model.predict(X)` reuses the training matrix.
  3. Check the answer text: if the only quoted score is the training-set score (typically suspiciously high) and no validation/CV score or variance estimate is given, the attempt has no evidence of generalization.
  4. Also confirm no sanity check beyond row count exists (e.g., predicted class balance compared to the training label distribution, label strings identical to the training labels).
- **Discriminator**: A real violation is when *every* reported metric comes from rows the model saw during fitting. It is fine to refit on the full labeled set for the final submission *after* estimating performance on a hold-out/CV split, and it is fine to additionally print a training score alongside a validation score.
- **Consequence**: The reported ~97% is meaningless; the submitted `result.csv` can fall below the grader's accuracy threshold (or contain systematically skewed/mislabeled classes) while the agent confidently claims success, yielding a WRONG verdict on the output file.

### Fabricated metric definition instead of the one implied by the task spec/config
- **Applies when**: `task` -- the task asks to visualize/report a quantity whose definition comes from a provided config, prior pipeline step, or standard domain semantics, rather than being spelled out in the prompt.
- **Pattern**: The script invents an arbitrary formula (e.g., an ad-hoc weighted sum of loosely related counts) and/or computes it on only one slice of the data (one join side, one group, one axis), then labels the bars with the axis/title text from the config as if the numbers matched that meaning; required companion artifacts encoding the true values are never produced.
- **Detection procedure**:
  1. Read the task and any config/label text to identify what quantity is actually being asked for (what the y-axis label, title, and expected output files imply), and list all output artifacts required.
  2. In the script, locate where that quantity is computed and check whether the formula is derived from the spec/domain definition or is invented in-line with hard-coded weights or unexplained combinations of columns.
  3. Check that the computation covers every relevant slice of the data (both sides of a symmetric relation, all rows, correct aggregation axis) and that ties/ordering follow the specified entity list.
  4. Compare the reported numbers against a quick plausibility check (expected magnitude/range for that quantity, counts per entity) and confirm every required output file is written.
- **Discriminator**: A real violation is a metric whose definition appears nowhere in the task, data, or config and cannot be reproduced by an independent reader; a look-alike that is fine is a documented, standard definition (explicitly stated or unambiguously implied) that merely uses a compact implementation.
- **Consequence**: The plotted/saved values are unrelated to the requested statistic and the required numeric artifacts are missing, so every value/file check fails even though the figure renders and looks well formatted.

### Unverified input source and no independent cross-check of a threshold-based count
- **Applies when**: `task` -- the task asks for a count of rows satisfying a statistical threshold (e.g. z-score, IQR, quantile rule) on one column of "the dataset", and the scripts pick a file/column and report the count directly.
- **Pattern**: The script grabs the first plausible file (often a split such as `*_train.csv`) and the first column whose name loosely matches, computes the statistic with one library default (e.g. population vs sample std, one-sided vs two-sided mask, NaN-filled rows silently counted), and reports the resulting count without ever recomputing it a second, independent way or checking that the chosen file/column is the one the task describes.
- **Detection procedure**:
  1. Read the task and note exactly which dataset/entity and which quantity is requested (whole dataset vs a split; raw column vs cleaned/numeric-coerced column; both tails vs one tail).
  2. In the scripts, list every I/O and selection decision: which file was loaded, whether other candidate files were enumerated (`ls`/`glob`), which column was chosen, how NaNs/non-numeric values were handled, and which std convention (`ddof`) the statistic uses.
  3. Check whether the count is verified by a second route — e.g. a manual `(x - x.mean())/x.std(ddof=...)` recomputation, a printed min/max of the statistic, a distribution summary, or a comparison of the two-sided mask vs the absolute-value mask — and whether the resulting count is sanity-checked for plausibility (fraction of rows flagged, shape before/after drop).
  4. Flag the attempt if the file/column choice is unjustified among available alternatives, or if a nonzero count rests on a single library call with no cross-check and no stated reason it should differ from an alternative convention.
- **Discriminator**: Fine: the script confirms the available files/columns, coerces dtypes and drops NaNs explicitly, and reproduces the same count under at least one alternative convention (or shows the extreme statistic values so the count is obviously robust). Violation: the count is produced once from an implicitly chosen file/column and default parameters, with no enumeration of alternatives and no second computation — so a wrong source or convention is indistinguishable from a correct one.
- **Consequence**: The reported count differs sharply from the ground truth (e.g. a large nonzero count where the correct answer is zero, or vice versa), and the single-value answer check fails outright.

### Missing reproducible artifact: answer asserted without a script or the required output file
- **Applies when**: `task` -- The task specifies a deliverable (e.g., a named result file) and/or requires applying an externally-specified transformation before computing a statistic, and the agent must produce code that does it.
- **Pattern**: The agent reports the summary value only as inline prose/JSON in its message, with no saved script that loads the data, applies the specified mapping/preprocessing, computes the statistic, and writes the requested output file — so the number is unverifiable and the required artifact is never created.
- **Detection procedure**:
  1. Read the task and list every required deliverable (output file name/location, exact key names, value types, rounding/format) and every required preprocessing step defined in auxiliary instructions or docs.
  2. Inspect the saved scripts/artifacts: is there code that reads the source data, applies each required transformation, computes the requested quantity, and serializes it to the named file?
  3. If no such code exists (or it prints only to stdout), treat the answer as unsubstantiated; if code exists, re-derive the value mentally from it and check the mapping was applied to the whole column (not just categories the agent assumed) and that the reported figure is the requested statistic, not an intermediate.
  4. Sanity-check the reported number against basics implied by the data (proportions sum to 1, the modal share is the maximum share, expected label vocabulary after mapping).
- **Discriminator**: A real violation is an answer with no executable path from raw data to the reported value and/or no required output file. Not a violation: a script exists and writes the file correctly, and the chat text merely restates its contents (formatting differences in the message are fine as long as the artifact matches spec).
- **Consequence**: The grader finds the expected result file missing or its contents mismatched, so all checks fail regardless of whether the quoted number happened to be close.

### No out-of-sample validation with the competition's stated probabilistic metric
- **Applies when**: `task` -- the task specifies an explicit evaluation metric over predicted probabilities (e.g. a weighted/balanced log loss) and the scripts fit one or more classifiers and write probability columns to a submission file.
- **Pattern**: The attempt trains models on the full training set, computes predictions only on the same training rows (or not at all), never implements the stated metric, and never estimates it on held-out folds. Class re-weighting / imbalance handling is switched on for every model and the raw, uncalibrated per-model probabilities are averaged, so no evidence exists that the submitted probabilities are better than a trivial baseline under the actual scoring rule.
- **Detection procedure**:
  1. Read the task/README and note the exact scoring function and any class-balancing in it.
  2. Search the scripts for (a) an implementation of that metric, (b) a train/validation split or cross-validation loop whose score is printed, and (c) a comparison against a naive baseline (e.g. constant class prior).
  3. Check whether any printed diagnostics are computed on rows the model was fit on (in-sample `predict_proba` on `X_train`) rather than held-out rows.
  4. Inspect the answer: if only min/max/mean of predictions and a "sums to 1" check are reported, with no metric value, the model quality is unverified — and check whether probability calibration was considered given the reweighting used during fitting.
- **Discriminator**: A real violation has *zero* held-out estimate of the stated metric anywhere in the pipeline. It is not a violation if the scripts compute the competition metric (or a documented equivalent) via CV/holdout and use it to choose among models/blend weights, even if the final model is refit on all data; nor if the metric name differs but the formula and class weighting match.
- **Consequence**: The submission may be badly miscalibrated (over-confident, prior-shifted by class weights), yielding a log-loss far worse than a simple calibrated baseline, so the graded score falls below the acceptance threshold and the answer is marked wrong despite a syntactically valid file.

### Output contract not verified against the test file (row count, ordering, and exact label vocabulary)
- **Applies when**: `task` -- the task says to produce predictions for a supplied evaluation file and save them to a named output file with a specified column, and the script writes that file after fitting a model.
- **Pattern**: The attempt builds a model on convenient rows (e.g., only records that have the engineered features non-null, or an internally re-split subset), then writes an output whose row count/order does not correspond one-to-one with the evaluation file's rows, and/or emits class strings the agent invented or reformatted rather than the exact category strings present in the labeled data. Suspiciously high validation accuracy from features derived after joining on label-correlated or leaked aggregates is reported as evidence of success instead of triggering a check.
- **Detection procedure**:
  1. From the task/README, note the required output filename, required column name(s), and the number of rows in the evaluation input.
  2. In the script, trace how the prediction frame is constructed: does it start from the full evaluation file (preserving its row order and every row, filling/imputing where features are missing), or from a filtered/merged/deduplicated subset?
  3. Compare the label strings produced (from `classes_`, mapping dicts, or hardcoded lists) against the distinct category values in the labeled source data — they must match character-for-character, with no renaming, casing changes, or added/dropped classes.
  4. Check the reported record count and class distribution in the answer against the evaluation file's row count and the training-label prior; treat a large discrepancy or a near-perfect validation score with a handful of weak features as an unverified red flag.
- **Discriminator**: A real violation is when the written file cannot be aligned row-for-row with the evaluation input, or contains label strings not in the label vocabulary. It is *not* a violation if the script legitimately joins by an ID key, reindexes to the evaluation file's full index, and fills any missing predictions — even if intermediate modeling used a filtered subset.
- **Consequence**: The grader compares the submitted file to ground truth keyed by evaluation rows and finds mismatched length/order or unrecognized label values, so accuracy collapses (or the file is scored as wrong/missing) despite the reported high internal validation score.

### Output format not verified against the provided template
- **Applies when**: `task` -- The task says results must be saved to a named output file whose format must match a supplied template/example file.
- **Pattern**: The script constructs the output table from its own assumptions (index labels, column names/order, header text, rounding, number of rows/columns) and writes it directly, never loading or comparing against the template file; self-"verification" only re-checks the script's own numbers rather than the required layout.
- **Detection procedure**: 1) Read the task for the phrase requiring the output to match a template and locate the template path. 2) Search the scripts for any read of that template file and any comparison of shape, index values, column names/dtypes, or rounding against it. 3) Inspect how index/column labels are generated (e.g., period-to-timestamp conversion, offsetting indices by 1, string formatting) and ask whether these were chosen from the template or invented. 4) Check the final answer for an explicit statement that the written file's shape and headers equal the template's.
- **Discriminator**: A genuine violation is when no template comparison exists and format choices are asserted rather than derived; it is fine if the script reads the template and asserts equality of columns/index (or reindexes onto it), even if it also rounds or renames afterwards.
- **Consequence**: The saved file's cell values may be right but the grader's file-level comparison fails on mismatched index labels, column offsets/names, or rounding, marking the expected result file WRONG.

### Deliverable file never validated against the provided template (prose report substituted for a checked artifact)
- **Applies when**: `task` -- the task requires writing a specific output file whose format is defined by a provided sample/template file, and the agent's scripts end by printing a summary of the modelling approach.
- **Pattern**: The attempt focuses on model architecture and CV scores, writes the output file from an in-memory frame, and never re-reads the written file to confirm it matches the template: same column names/order, same number of rows, exactly the same key/ID values (taken from the test file, not regenerated or re-indexed), correct dtype, and no stray index column. The final answer describes the method and prediction counts instead of demonstrating the file was verified.
- **Detection procedure**:
  1. Read the task for the required filename, required columns, and the reference sample file; note the key column and expected row count.
  2. In the scripts, locate the write step and check whether the ID column is copied verbatim from the test input (and whether `index=False` / header conventions are honoured), and whether any post-write reload + comparison against the sample file occurs.
  3. Check the answer for concrete evidence of validation: first rows of the saved file, row count vs. sample row count, and a set-equality check of IDs; absence of these means unverified.
  4. Flag if the answer's only evidence is CV metrics and a prediction-class histogram.
- **Discriminator**: A real violation is when no read-back/format assertion exists and IDs are not provably sourced from the test file; a look-alike that is fine is an attempt that writes the file with template-derived IDs and shows an explicit shape/column/ID-match check, even if it reports little about the model.
- **Consequence**: The grader marks the expected output file as WRONG/MISSING (mismatched IDs, row count, column names, or an extra index column), scoring 0 regardless of model quality.

### Missing-value mask defined inconsistently, so the two groups don't partition the full dataset
- **Applies when**: `task` -- a task asks to split rows by whether a field is null/missing (or by any boolean condition) and compare an aggregate statistic between the two resulting groups.
- **Pattern**: The attempt builds the group mask with a definition that silently differs from "null" as stored in the raw file — e.g. relying on default loader NA-conversion so that empty strings, whitespace, `"NA"`/`"None"`/`"-"` sentinels, or type coercion move rows to the wrong side — and/or applies an extra `dropna()`/filter on the measured column (or other columns) before grouping, so some rows are dropped from both groups. Group means then drift from the reference values in the same direction, and no row-count check is done.
- **Detection procedure**:
  1. Read the task and note the exact partition rule and which column the statistic is computed on; the two groups should together cover every row of the source table.
  2. In the script, locate the load step (delimiter, `na_values`, `keep_default_na`, dtype/converters) and the mask construction; check whether string sentinels/whitespace are handled the same way the task intends, and whether any `dropna`, filter, deduplication, or subsetting happens before the split.
  3. Check that the script prints `len(group_a) + len(group_b) == len(full_df)` and the per-group non-null counts used for the statistic; absence of such a check is itself a red flag.
  4. Compare reported group statistics against a quick independent recomputation of the mask (e.g. counting raw blank fields) — systematic small shifts in both group means indicate rows landed on the wrong side or were dropped.
- **Discriminator**: A real violation is when the group sizes don't sum to the table size, or the null test used differs from the raw-file notion of missing (sentinels/blank strings mishandled), or rows were removed by pre-filtering. It is *not* a violation if rows are excluded only because the measured numeric column itself is missing for them (unavoidable for a mean/t-test) **and** the script reports those counts explicitly, or if the loader options demonstrably reproduce the same mask.
- **Consequence**: Both group means (and the test statistic) are computed on slightly wrong subsets; the grader's exact numeric checks on each mean fail even though the pipeline and answer format look correct.

### Referenced spec file never read; requirements and outputs invented
- **Applies when**: `task` -- The task points to an external instruction/config/guidance document (or a README/spec section) that defines the analysis rules, thresholds, categories, and the deliverable artifacts.
- **Pattern**: The script never opens or quotes the referenced document; instead the agent hard-codes its own filtering thresholds, category definitions, and output set (often saving only the one artifact named in the prompt summary), and the final answer restates those invented rules as if they were the spec.
- **Detection procedure**:
  1. From the task text, list every referenced spec/instruction file and every deliverable it implies (plots, arrays, JSON/CSV summaries, exact filenames).
  2. Search the scripts for a read of that spec file (or for comments/docstrings that verbatim mirror its rules); check whether thresholds, groupings, and category mappings are asserted with no traceable source ("Rules: 1..., 2...", magic numbers like a 180-day window, priority orders).
  3. Enumerate the files the script actually writes and compare to the deliverable list; flag any deliverable that is never created, and any created artifact whose content/ordering/format is not verified against the spec.
  4. Check the reported answer: does it report numbers derived from self-invented buckets/filters rather than spec-defined ones, and does it claim completion while some required artifacts are absent?
- **Discriminator**: A real violation is when the rules or the artifact set come purely from the agent's assumptions and at least one required output is missing or unverifiable; it is *not* a violation if the script demonstrably loads/echoes the spec (or the task text itself fully enumerates the rules and filenames) and every named artifact is written and sanity-checked.
- **Consequence**: Graders that compare per-artifact outputs report the required files as WRONG/MISSING and the derived tallies/plot as mismatched, yielding 0 checks passed even though the script runs without error.

### Output schema not derived from the provided template file
- **Applies when**: `task` -- the task points to an example/sample output file (or explicitly states a required column set, header names, or row order) and the scripts must produce a deliverable file matching it.
- **Pattern**: The attempt never reads or prints the referenced sample/template; it hard-codes invented column names and row contents from intuition, sometimes duplicating a field or omitting a required one, so the deliverable's schema cannot be verified against the expected one.
- **Detection procedure**:
  1. In the task text, identify every referenced format artifact (sample file, stated headers, ordering, units, rounding).
  2. Search the scripts for any read/print of that artifact, or an explicit column list copied from it; note whether the written DataFrame's columns and row labels are justified by anything observed.
  3. Compare the answer's header row and label values to what the task implies (e.g., redundant/renamed columns, extra or missing columns, made-up label wording).
  4. Flag if the schema was invented rather than mirrored, even when the underlying computed values look plausible.
- **Discriminator**: Not a violation if the scripts actually load the template (or the task fully specifies headers verbatim) and construct the output with exactly those columns/labels; it is a violation when the header names/labels appear nowhere in the task or inspected data and no verification step compares them to the template.
- **Consequence**: The grader's file/column comparison fails outright (WRONG/MISSING) despite correct analytical values, yielding a zero score.

### Skipping data-quality validation before imputing/modeling on a dataset that advertises dirty values
- **Applies when**: `task` -- The script loads a raw file (often named or documented as containing errors/noise) and immediately imputes means and fits a model on the numeric columns without inspecting value distributions, dtypes, or plausibility ranges.
- **Pattern**: The agent treats every stored value as valid: it computes column means for imputation and fits/evaluates the model on columns that still contain corrupted entries (non-numeric strings coerced or dropped, impossible signs/units, duplicated-digit or off-by-orders-of-magnitude outliers, sentinel codes like -999). Because means and regression coefficients are highly sensitive to such entries, both the imputation values and the test-set error are inflated, and the reported metric is off by an order of magnitude.
- **Detection procedure**:
  1. Read the task/data source for any signal that the raw data is unclean (file name, description, mention of "errors"/"raw"/"noisy"), and note which columns feed the model.
  2. Scan the scripts for any validation step on those columns: dtype checks with coercion, min/max or range plausibility checks, outlier/sentinel detection, duplicate/inconsistent-row handling. If the only preprocessing is `fillna(mean)` (or similar) plus a split, flag it.
  3. Check whether the printed diagnostics the agent did run (means, coefficients, sample predictions, R²) were sanity-checked against domain-plausible ranges for the target; if a mean or a resulting error magnitude is implausible relative to the variable's natural scale and the agent did not investigate, flag it.
  4. Compare the reported metric to the target variable's own variance/scale — an MSE comparable to or larger than the target's total variance implies the model is worse than predicting the mean, which should have triggered a cleaning pass.
- **Discriminator**: A real violation is when no inspection of value validity occurred at all, or when obvious anomalies were visible in the agent's own output and left unaddressed. It is *not* a violation when the agent explicitly examined the columns (dtype coercion, range/outlier report) and documented a justified decision to keep all values, or when the task explicitly forbids cleaning beyond the stated imputation.
- **Consequence**: Corrupted values dominate the fitted coefficients and the test residuals, so the reported error metric is inflated by roughly an order of magnitude versus the reference value, and the exact-match numeric check fails.

### Normalization/scaling method mismatch producing degenerate summary statistics
- **Applies when**: `task` -- the task asks to rescale/normalize numeric columns and then report summary statistics (e.g., means) of the transformed columns.
- **Pattern**: The attempt applies a scaler whose output makes the requested statistic trivially constant (e.g., zero-centering/standardization when a bounded [0,1] min–max rescaling was intended), and reports the degenerate values (all 0.0000, or all 1.0000 for std) without questioning whether the transform matches the task's intent or whether the reported number carries any information.
- **Detection procedure**:
  1. Read the task wording for the transform requested and note whether the expected output range/answer format implies a specific scaling family (bounded 0–1 vs. mean-0/unit-variance), and note that the task asks for a *per-column informative* statistic.
  2. In the scripts, identify the exact scaler/formula applied to each column and derive analytically what the reported statistic must equal under that transform.
  3. Flag the attempt if the derived statistic is mathematically constant regardless of the data (0 for means after centering, 1 for std after standardizing) — i.e., the reported numbers convey no dataset-specific information.
  4. Cross-check the reported numbers: any column whose value is exactly 0.0000/1.0000 while other, untransformed columns show data-dependent values is strong evidence of the mismatch; also confirm all requested output fields are present and consistently derived.
- **Discriminator**: A real violation is when the reported statistic is an artifact of the transform (constant by construction) or the scaling family contradicts the task's implied range; it is fine if the task explicitly names standardization and the reported statistic is still data-dependent (e.g., reporting means of columns that were not centered, or reporting min/max after standardizing).
- **Consequence**: The graded values for every rescaled column are 0.0000 (or otherwise uniform) instead of the expected data-dependent values, so only the untouched/binary columns match and the answer fails most checks.

### Rounded statistic reported without verifying the row set / preprocessing that produced it
- **Applies when**: `task` -- the task asks for a single summary statistic (correlation, mean, metric) rounded to a fixed number of decimals, computed over two or more columns of a table.
- **Pattern**: The agent loads the file with default settings and pipes the columns straight into the statistic, never checking how many rows actually entered the computation — silently dropping rows with missing/non-numeric/sentinel values, keeping rows the task's scope excludes, or reading only part of a multi-sheet/multi-file/multi-header source. The resulting value differs from the intended one in the second decimal, and because it is only reported rounded, the discrepancy is invisible in the final answer.
- **Detection procedure**:
  1. Read the task and note exactly which records are supposed to be in scope and to how many decimals the statistic must be reported.
  2. In the scripts, locate the load step and the statistic call; check whether the agent printed the number of rows loaded, the number of rows used after any NaN/dtype coercion, and the dtypes of the two columns.
  3. Check whether the unrounded statistic is printed with full precision and whether the agent looked at it relative to the rounding boundary (e.g. a value within ~0.005 of a `.xx5` cut point, or an r whose 2-dp rounding could shift by one unit under a slightly different row set).
  4. Confirm the answer's rounding and format match the spec (correct decimal count for each field, no degenerate placeholders like an exactly-zero p-value where a small nonzero rounded value or "0.0000" is required).
- **Discriminator**: A real violation is an attempt with no printed row/dtype/NaN diagnostics and no full-precision value, so the reported rounded number cannot be traced to a defensible record set. It is *not* a violation if the script prints the input shape, the count of rows actually used, confirms numeric dtypes and the missing-value policy, and reports the full-precision statistic — even if a reviewer might have chosen a different (documented) policy.
- **Consequence**: The graded field is off by one unit in the last reported decimal (e.g. 0.53 vs 0.54), so an exact-match check on the coefficient fails even though the qualitative conclusion is right, and no saved script exists to diagnose which rows caused the drift.

### Accepting a weak model without benchmarking held-out error against the target's scale or a baseline
- **Applies when**: `task` -- the deliverable is a file of predictions that will be scored against hidden ground truth, and the scripts fit models with default/naive feature handling and only self-report a validation score.
- **Pattern**: The attempt encodes high-cardinality categorical fields with arbitrary integer codes (and/or drops informative columns), trains one or two off-the-shelf regressors with hand-picked hyperparameters, picks the "best" by a single random split, and declares success on the basis of a favorable-sounding R²/accuracy — without comparing the held-out error to the target's own dispersion, to a trivial baseline (mean/median or group-median), or to a stronger alternative model, and without checking that the predicted distribution matches the training target distribution.
- **Detection procedure**:
  1. Read the task: note that the grade depends on prediction quality against hidden labels, not on file existence, so any error level above the grader's tolerance fails.
  2. Read the scripts: check whether categorical/text features are given a mapping the model can exploit (target/one-hot/native categorical support) or only arbitrary ordinal codes; check whether any informative column is dropped; check whether more than a single 80/20 split and a couple of untuned models were tried.
  3. Compute the reported error in relative terms (RMSE ÷ target mean or std, or MAE ÷ median) and compare against a naive baseline the scripts should have run; also compare the reported min/max/mean of predictions with the training target's min/max/mean.
  4. Flag if no baseline/alternative comparison exists, if relative error is large (e.g., RMSE on the order of a third or more of the target mean), or if the prediction range is clipped/inflated relative to the training target range.
- **Discriminator**: A genuine violation is an attempt whose only evidence of quality is an unanchored score from one split with clearly under-engineered features; it is *not* a violation when the agent explicitly benchmarks against a trivial baseline and at least one stronger/tuned model, uses cross-validation, and shows predicted-vs-actual distributions agreeing — even if the final score is modest.
- **Consequence**: The submitted file has the right shape, column name and row count, so all format checks pass, but the accuracy check against ground truth falls outside tolerance and the task is graded incorrect.

### Required output artifact not verified at the expected path/format
- **Applies when**: `task` -- the task requires writing a deliverable file with a specified name, column(s), and one row per input record, and the agent claims it was produced.
- **Pattern**: The attempt reports success narratively (row counts, class distribution) but never shows code/commands that (a) write the file to the expected location relative to the task's working/data directory, and (b) re-read the written file to confirm its path, header spelling, column count, row count and row order match the input records; the file is silently left in a home/scratch directory or with extra columns/index.
- **Detection procedure**:
  1. From the task, note the exact required filename, required column name(s), and the directory the grader will look in (normally the same directory as the provided data / current working dir).
  2. In the scripts, find the write call and check the path is that expected directory (not an absolute path elsewhere, not a temp dir) and that index/extra columns are suppressed and the header string matches exactly.
  3. Check for a post-write verification step: reload the file, assert row count equals the number of test records, assert column names, and print the resolved absolute path.
  4. In the answer, check whether the reported path is the expected deliverable location and whether the verification output (not just in-memory prediction stats) is quoted.
- **Discriminator**: A real violation is an unverified or mislocated/misformatted artifact (path differs from the task's directory, no reload check, header/index mismatch, row count not tied to the test set size). It is fine if the script writes to the expected directory and prints a reload check confirming path, shape, and header — even if the model itself is simple or accuracy is modest.
- **Consequence**: The grader reports the expected output file as WRONG/MISSING (or fails header/row-count checks), scoring 0 regardless of prediction quality.

### Incomplete/misplaced deliverable set relative to the task and its config spec
- **Applies when**: `task` -- the task asks for a saved artifact (chart, model, table) produced "according to" an external config/spec file, and the grader checks files on disk rather than just a printed answer.
- **Pattern**: The attempt computes a plausible headline value and writes only the one file it explicitly noticed (e.g. the image), reading a couple of keys from the spec ad hoc; it never enumerates every key/required output in the spec (extra companion files such as the serialized plot data or plot parameters), never confirms the required file names, and writes to a personal working directory instead of the expected output location.
- **Detection procedure**:
  1. From the task text and the referenced spec/config file, list every required output artifact, its exact filename/extension, and its directory; also list every key in the config and the constraint it imposes (title, colors, size, labels, ordering, units, rounding, autopct, legend).
  2. Read the scripts and check off each item: is every artifact written, at the stated path, with the stated name? Is every config key actually consumed (not just the 3–4 the agent guessed at), and is nothing hard-coded that the config specifies?
  3. Check whether the underlying numbers behind the artifact are also persisted if the task/spec implies a machine-checkable dump (array/JSON of the plotted values), and whether category labels/order match the spec rather than data-driven sort order.
  4. Compare against the final answer: if the answer is only a scalar/string while the graded deliverables are files, treat unwritten or mislocated files as a failure regardless of whether the scalar is right.
- **Discriminator**: A real violation is a missing/misnamed/misplaced required file or an unconsumed config key that changes the artifact's content; it is *not* a violation if the agent writes extra diagnostic files, or if it reformats internally while still emitting every required artifact at the required path with all spec constraints honored.
- **Consequence**: File-level checks report WRONG/MISSING for the artifacts that were never created (and for the image if spec keys were ignored), so the attempt scores 0 even when the identified category/value is correct.

### Fabricating output schema/category definitions instead of reading the provided template file
- **Applies when**: `task` -- the task supplies a pre-existing result file (or explicit format spec) to be filled in, and the scripts must produce rows/labels/groupings that match it exactly.
- **Pattern**: The attempt never opens the supplied file to learn its headers, row labels, ordering, or the intended grouping/bin definitions; instead it invents its own category names, boundaries, and extra rows from outside knowledge, then overwrites the file and declares success because the write succeeded.
- **Detection procedure**:
  1. In the task text, note that a target file/format is provided and identify what its rows and columns are supposed to encode.
  2. Search the scripts for any read/inspection of that target file (e.g., loading it, printing its head/columns/index) before writing; if the only interaction is a write, flag it.
  3. Check whether the grouping keys/labels used in the code are hard-coded from the agent's own assumptions (custom thresholds, self-chosen category names, an added "unknown/other" bucket) rather than derived from the template or the data's own labels.
  4. Cross-check the reported per-group counts for implausibility (a group with 0 or 1 members, or a group holding only a tiny fraction of records) that would indicate wrong boundaries/labels; confirm no sanity check on group sizes exists.
- **Discriminator**: Fine if the scripts read the template (or the task spells out the exact labels) and the produced rows/labels/order match it, with only value cells changed; a violation is when labels, row set, or bin definitions are the agent's invention and no comparison against the provided format is ever made.
- **Consequence**: The written file has mismatched row labels/extra rows and wrong per-group values, so an exact-match file check fails even though the script "verified" its own output.

### Entity-level aggregation and subgroup split defined on raw rows instead of per-unit records
- **Applies when**: `task` -- the data are multiple observation rows per underlying unit (event/subject/session) and the question asks for a statistic relating per-unit attributes (e.g., a maximum, a duration, a total) with subgroups defined by a threshold such as the median.
- **Pattern**: The attempt correlates or thresholds using the raw row-level table (or aggregates with the wrong reducer/units), so each unit contributes many rows, the threshold (median) is computed over rows rather than over the per-unit values, and the resulting coefficient/subgroup membership is close to but not equal to the correct value.
- **Detection procedure**:
  1. From the task, list the per-unit quantities required (the aggregate of one variable, the span/duration of another, the magnitude used for the split) and note that the analysis unit is the entity, not the row.
  2. In the scripts, check that there is exactly one `groupby(unit_id)` producing one row per unit with the correct reducers (max, last−first for duration, sum/max for magnitude as specified) and correct units (e.g., hours vs days), and that duplicates/missing values are handled before aggregation.
  3. Verify the split threshold is computed on the aggregated per-unit series (not on raw rows, not after filtering) and that the two subgroups partition all units; print counts of units per subgroup and compare with the number of unique unit ids.
  4. Confirm the reported coefficient/p-value comes from the aggregated frame and matches the required rounding; a sanity check that group sizes are of plausible magnitude (tens–hundreds of units, not thousands of rows) should be present.
- **Discriminator**: A real violation is when the correlation/threshold is computed over a table whose row count exceeds the number of unique units, or where the reducer/duration definition differs from the task wording; it is fine if the data are already one row per unit, or if aggregation is done with a documented equivalent reducer that yields identical per-unit values.
- **Consequence**: Relationship-type labels may still come out right by luck, but the numeric coefficient (and subgroup p-values) drift from the ground truth by more than the two-decimal tolerance, so the answer is graded wrong on the coefficient checks.

### Hardcoded/recalled inputs and invented output schema instead of reading the provided files
- **Applies when**: `task` -- the task references a provided dataset and/or an example output file (e.g., a sample result template), and the script defines the data or the output columns inline from memory.
- **Pattern**: The agent never loads the supplied data/template files; it types in arrays it believes match the source and invents column names/row layout for the result file, so both the numbers and the file schema are unverified guesses. A related tell is a stated constraint (e.g., "set the random seed") that the chosen method makes irrelevant, signaling the wrong procedure was used.
- **Detection procedure**:
  1. From the task, list every artifact that must be read: the input data file(s) and any sample/template output file.
  2. Scan the script for a read/load call for each of those artifacts; flag if data values or the output header are literals rather than derived from a loaded file.
  3. Check whether any stated constraint (seed, rounding, units, ordering) is actually exercised by the code — an unused seed implies a deterministic/analytic method was substituted for the resampling-based one the task implies.
  4. Compare the written columns/ordering to the template's; if the template was never opened, treat the format as unverified.
- **Discriminator**: Fine if the script loads the real files and only hardcodes trivially checkable metadata (paths, known constants) after verifying them, or reproduces the template header after reading it; a violation is when the substantive values or the output schema exist only in the script and are never cross-checked against the provided files.
- **Consequence**: The result file mismatches the expected values and/or column format, so the grader marks the expected output file WRONG/MISSING even though the script ran without error.

### Unverifiable submission: no reproducible script or output validation for the requested prediction file
- **Applies when**: `task` -- the task requires producing an output artifact (e.g., a predictions file with a specified column name and one row per test record) and the agent claims completion by naming the file.
- **Pattern**: The attempt reports the deliverable's name without retaining the code that generated it, and never checks the written file's existence, row count against the test input, column naming, dtype, or value plausibility — so a missing, empty, misnamed, misaligned, or wrongly-formatted file passes unnoticed.
- **Detection procedure**:
  1. From the task, list the exact required artifact properties: filename/path, column name(s), expected number of rows (= number of test rows), value type/range, and any ordering requirement.
  2. In the scripts, locate the code that trains/predicts and writes the artifact; confirm it exists, reads the intended test file, writes to the exact required path, and uses the exact required header (and `index=False` where an index column would corrupt the format).
  3. Check for an explicit post-write verification step: re-read the saved file and assert shape/row count matches the test set, the required column is present, and there are no NaNs/constant or out-of-range predictions.
  4. Inspect the final answer: it must be backed by such a verification (printed shape, head, counts), not merely a filename assertion.
- **Discriminator**: A real violation is when no generating script or no read-back/shape-and-header check exists, so correctness of the artifact is asserted rather than demonstrated; it is fine if the script writes the file and the log shows the re-read file's row count, column name, and sample values matching the test set even if no fancy validation framework is used.
- **Consequence**: The grader looks for the artifact with the specified column and aligned rows and reports it WRONG/MISSING (0 checks passed) despite the answer claiming the file was produced.

### Unverified population coverage when computing a simple aggregate statistic
- **Applies when**: `task` -- the task asks for a single descriptive statistic (mean, median, rate, count) over "all observations" of a field, and the scripts load the data from one or more files/tables before aggregating.
- **Pattern**: The attempt computes the statistic on a silently reduced population — one file/partition/chunk out of several, a subset left over from an earlier filter or `head`/sample, rows lost to a join/merge or dtype coercion, or an aggressive "outlier"/dropna rule applied without justification — and reports the number without ever checking that the row count used matches the full source data.
- **Detection procedure**:
  1. From the task, note the intended population ("all observations") and any explicit filtering/rounding/format constraints.
  2. In the scripts, trace the data from load to aggregation: list every operation that can drop or add rows (file globbing, `read_*` with `nrows`/chunks, filters, merges, `dropna`, outlier thresholds, `astype`/`to_numeric` coercions) and check whether the final denominator is asserted or printed against the raw record count of the source.
  3. Check that the statistic is computed on the intended column/axis of the full frame, not on an intermediate, grouped, or per-file result that is then averaged unweighted.
  4. Confirm a reproducible script exists and prints the row count and value; if no script/counts are shown, the number cannot be validated at all.
- **Discriminator**: A real violation is row loss or subsetting that is neither required by the task nor verified by a count/shape check (including averaging per-group means instead of the pooled mean, or discarding valid values as "outliers" with an arbitrary cutoff). It is fine if the script explicitly documents and verifies the exclusions the task requested (e.g., only missing values removed) and shows that used_rows + excluded_rows equals total source rows.
- **Consequence**: The reported value is systematically biased and misses the expected number beyond the grader's tolerance, so the check fails even though the code runs without error.

### Numeric columns ranked without verifying they were parsed as numbers (string-formatted values → wrong extremes)

- **Applies when**: `task` -- the task asks for the maximum/minimum (or mean-imputed) value of a quantitative field read from a raw CSV/Excel file where values may carry thousands separators, currency/percent symbols, or other text formatting.
- **Pattern**: The attempt loads the file, imputes with `mean()`, and takes `idxmax`/`idxmin` (or `sort_values().head()`) without checking the column's dtype. If the column is `object`, imputation silently skips it and the ordering becomes lexicographic (or the fill/parse coerces values to NaN), so the reported extreme rows are wrong even though the code runs without error.
- **Detection procedure**:
  1. Read the task to identify which column(s) the answer depends on and whether any transformation (imputation, rounding, unit conversion) must precede the ranking.
  2. In the scripts, look for an explicit cleaning/coercion step for those columns (e.g., stripping separators/symbols then `astype(float)` / `pd.to_numeric(..., errors='coerce')`) and an assertion or printout of the resulting dtype, NaN count, and min/max.
  3. Check that the imputation actually affected the target column (count of NaNs before/after; fill value printed) rather than being a no-op on an object column.
  4. Compare the reported extreme entities against a quick plausibility check (are the top/bottom values consistent with the column's known scale and with other rows sorted numerically?); a "lowest" that is clearly not the smallest numerically, or extremes whose leading digits look alphabetically ordered, indicates string sorting.
- **Discriminator**: A real violation is when no dtype/coercion evidence exists and the column plausibly contains formatted text, or the printed extremes are inconsistent with a numeric ordering. It is fine if the script demonstrates the column is already numeric (dtype check, describe() output) or explicitly cleans it, even if the final answer happens to be uncommon.
- **Consequence**: The graded JSON names the wrong entity for the min and/or max (often the max is coincidentally right while the min is wrong), so the result file fails the exact-match check (0/1).

### Output file omits requested derived columns (only the final label is saved)
- **Applies when**: `task` -- the task asks to compute several intermediate quantities/scores *and* a final grouping/label, and to save "the results including X and Y" to a specified file.
- **Pattern**: The script computes all the intermediate metrics and scores in memory, prints them, but writes only a minimal subset (e.g., an ID plus the final label) to the output file, dropping the per-item metrics/scores that the task explicitly said to include; the answer then asserts the file "matches the reference exactly" without any comparison actually being executed.
- **Detection procedure**:
  1. From the task statement, enumerate every quantity that must appear in the saved artifact (each named metric, each score, the grouping/segment, the final level, the identifier).
  2. In the script, find the line that constructs the DataFrame written to the output path and list its columns (and index/header settings).
  3. Diff the two lists; also check whether row count/ordering constraints implied by the task are preserved.
  4. In the answer, check whether any claim of "verified against reference/expected output" is backed by a comparison actually performed in the code shown.
- **Discriminator**: A real violation is when a task-named quantity is computed but excluded from the saved file (or renamed beyond recognition/serialized with wrong index-header settings). It is *not* a violation if the task only asked for the final label, or if the extra quantities are present under clearly equivalent names.
- **Consequence**: The saved file fails column/shape/content comparison against the expected artifact, so the check is scored wrong even if the underlying computation logic was reasonable, and the answer's unverified "matches exactly" claim is false.

### Output schema invented instead of copied from the provided template file
- **Applies when**: `task` -- the task says to write results into an output file "following the format specified in" a provided sample/template file.
- **Pattern**: The script never reads or inspects the template; it hard-codes guessed column names, guessed cell encodings (e.g., packing a two-number interval into one string literal, or splitting it across columns), guessed row order/precision, and sometimes guesses the output directory too — so the numbers may be right while the file fails any schema-aware comparison.
- **Detection procedure**:
  1. In the task text, note every referenced template/sample artifact and the required output filename/location.
  2. Search the scripts for any read of that template (`read_csv`, `open`, printing its header/rows) and any assertion that produced columns/dtypes/row count match it.
  3. If absent, compare the script's constructed column labels, value formatting, and write path against what the task literally names; treat guessed labels or ad-hoc string formatting of numeric fields as the violation.
  4. Check the reported answer: does it show the final file contents *and* evidence that they conform to the template, or only the computed statistics?
- **Discriminator**: Fine if the script loads the template (or an explicit verbatim quote of its header appears) and builds the output to that schema, even with minor cosmetic differences; a real violation is when the schema/format/path is asserted from assumption with no reference to the provided file.
- **Consequence**: The grader reports the expected output file as WRONG/MISSING despite statistically correct values, because column names, cell encoding, or file location do not match the reference.

### Ignoring the provided sample/template output file when writing the result
- **Applies when**: `task` -- The task supplies a reference/sample output file (or explicitly states a required output layout) and asks that the saved result "follow the same format".
- **Pattern**: The scripts compute the requested statistic and dump it with a default writer (e.g., `to_csv` straight from a DataFrame) without ever reading or inspecting the sample file, so the header name of the index column, column/row order and labels, rounding/precision, or inclusion of an index are never verified against the template.
- **Detection procedure**:
  1. Read the task for any mention of a sample/example/template output file or an explicit format constraint on the saved artifact.
  2. Search the scripts for any read of that sample file (or a hard-coded replication of its structure) and for an explicit comparison of the produced file's shape, header, index labels, ordering, and numeric precision against it.
  3. Check the answer/report for evidence that the written file was compared cell-for-cell (or at least header-for-header) with the sample, not merely re-read and printed.
  4. Flag if the only validation is printing the computed object, or if internal counts in the report are mutually inconsistent (e.g., reported per-column missing counts that cannot produce the reported number of dropped rows), indicating no sanity check on the pipeline.
- **Discriminator**: A real violation is when no sample inspection/verification exists at all, or the writer's defaults clearly differ from the stated template (missing index header, unrounded values, different label order). It is fine if the script reads the sample and asserts matching structure, or if the sample's structure is demonstrably reproduced (identical column/index names and ordering) even without a literal diff.
- **Consequence**: The grader compares the submitted file to the expected one and marks it WRONG/MISSING even though the underlying statistic may be numerically right, yielding 0 checks passed.

### Silently subsampling the dataset when the deliverable must cover all records
- **Applies when**: `task` -- the task asks for per-row outputs (labels, predictions, scores) written to a file, and the scripts read a large input table.
- **Pattern**: The attempt reads only a slice/sample of the input (e.g., `nrows=`, `head()`, `sample(n=...)`, a hard-coded cap for speed) and produces an output file with far fewer rows than the source data, without the task authorizing any subsetting.
- **Detection procedure**:
  1. From the task/README, note the expected scope of the output (one row per input record) and the documented size of the source data.
  2. In the scripts, grep the data-loading and output-writing steps for row limits, sampling, filtering, or `dropna` that shrinks the record count, and check whether labels are mapped back to all original rows.
  3. Compare the row count reported in the answer (and in the written file) against the source row count; flag any unexplained shortfall.
  4. Also confirm the column names/order in the written file match the exact schema the task specifies.
- **Discriminator**: A real violation is an undisclosed or convenience-driven reduction of the deliverable's coverage; it is fine if the task itself specifies a subset/filter, or if sampling is used only for internal model selection/hyperparameter search while the final output is applied to every record.
- **Consequence**: The expected output file fails comparison outright — row count (and hence any per-row label alignment) does not match the reference, so the check scores 0 regardless of clustering quality.

### Degenerate clustering accepted because the selection metric was inflated by skewed, untransformed features
- **Applies when**: `task` -- the task asks for an unsupervised grouping into "an appropriate number of clusters" and the script picks k by an internal score (silhouette/inertia) on aggregated numeric features written to an output file.
- **Pattern**: The attempt aggregates heavy-tailed count/amount features, applies only mean/variance standardization (no log/rank/quantile transform, no outlier handling), then picks k by the highest silhouette — which is maximized by isolating a handful of extreme records. The reported partition is effectively "everyone" vs. "a few outliers" (e.g. one cluster with >70% of rows, others with a handful), and the agent declares success without sanity-checking cluster sizes or feature distributions.
- **Detection procedure**:
  1. From the task, note that the deliverable is a meaningful segmentation, not just any label column, and note the required column names/format.
  2. In the script, check whether skewed monetary/count features are transformed (log1p, quantile, robust scaling) or extremes trimmed before distance-based clustering, and whether k is chosen by more than a single internal score maximum.
  3. In the answer, read the per-cluster row counts and compare with the number of rows: flag if one cluster holds the large majority while one or more clusters hold a negligible number of rows (<1%), or if the score is suspiciously high (e.g. >0.5) for high-dimensional real data.
  4. Check that the saved file's features match what was clustered (same transform/scaling) and that row count equals the number of entities clustered.
- **Discriminator**: A genuinely imbalanced but valid solution has clusters that are all substantively populated and separations that survive a transform/robust-scaling check; a violation is a partition whose small clusters are a few extreme records and whose score collapses (or whose k changes) once the skew is handled — i.e. the metric, not the structure, drove the choice.
- **Consequence**: The saved label file encodes an outlier-detection split rather than customer segments, so any grader comparison against a reasonable reference partition (cluster count, size distribution, agreement score) fails, marking the output file wrong.

### Hardcoded/recalled inputs instead of loading the provided data and output template
- **Applies when**: `task` -- the task supplies a dataset (and often a template/example of the expected output file), and the script must derive its numbers from those files.
- **Pattern**: The script embeds constants typed from memory or background knowledge (totals, counts, rates, group definitions) and never reads the supplied data files; likewise it invents the output file's columns/rows rather than reproducing the provided format. Any downstream statistic is then computed on numbers that do not match the actual data, and the file fails a format-sensitive check.
- **Detection procedure**:
  1. Read the task for (a) which data files are provided and (b) any stated output format/template for the result file.
  2. Scan the scripts for a load step (`read_csv`/`read_*`/file open) on those files; flag if the key quantities appear as literal constants with comments like "historical data" or "known values", or if grouping/filtering of the raw records is never performed in code.
  3. Compare the written result file's column names, ordering, units, and number of rows against the template given in the task; flag any invented headers or extra/missing fields.
  4. Check the answer for a traceability statement linking each reported number back to a computation on the loaded file (e.g., printed row counts, group sizes) rather than to prior knowledge.
- **Discriminator**: A genuine violation is when the analysis quantities themselves are literals never verified against the loaded data, or when the output schema is self-invented. It is *not* a violation to hardcode auxiliary constants (random seed, number of bootstrap iterations, confidence level, a documented cutoff date) as long as all reported statistics are computed from records read out of the provided files and printed sanity checks (row counts, group totals) confirm agreement.
- **Consequence**: The reported interval/statistic is computed from wrong inputs and the result file's schema does not match the expected one, so an exact-match or tolerance-based file check on `result.csv` reports WRONG/MISSING even though the method (bootstrap percentile CI) looks correct.

### Overconfident, unvalidated probability outputs for a log-loss deliverable (single holdout fit, no baseline, no clipping/calibration)
- **Applies when**: `task` -- the task is scored by a probabilistic metric (log loss / Brier) and the script trains one model, evaluates it on a single random split, and writes `predict_proba` output straight to the submission file.
- **Pattern**: The attempt fits a high-capacity model (deep trees / many boosting rounds) on only the train portion of a single split, never refits on the full labeled data, never cross-validates or tunes, never compares the score to a trivial baseline (class-prior constants), and never bounds/calibrates the emitted probabilities — so the file contains values on the order of 1e-5 or smaller for minority classes, where a handful of misclassified rows dominate the metric.
- **Detection procedure**:
  1. Read the task statement and note the exact scoring metric and that it penalizes confident errors logarithmically.
  2. In the script, check whether the object that generates the test predictions was fit on *all* labeled rows, whether any resampling/CV or hyperparameter selection was done, and whether the reported validation score is compared against a naive baseline (e.g., predicting the training class frequencies).
  3. Inspect the emitted probability columns in the answer: look at the minimum values per class and how many rows are near 0 or near 1 for a rare class.
  4. Flag if the final model saw only a fraction of the data, or if extreme probabilities are produced with no clipping/calibration and no evidence the CV score beats the prior baseline.
- **Discriminator**: A fine attempt may also produce some confident rows, but it (a) refits on the full training set, (b) reports a cross-validated metric that clearly beats the constant-prior baseline, and (c) either uses a calibrated/regularized model or clips/smooths probabilities so rare-class predictions are not orders of magnitude below their base rate. A violation is a single deep-model holdout fit whose only quality evidence is one split's score, with untempered near-zero probabilities.
- **Consequence**: The graded log loss is far worse than the reported validation number — often worse than a constant-prior submission — because a few confidently wrong rows contribute huge penalties, so the submission fails the accuracy threshold.

### No held-out validation or distribution sanity check before submitting predictions
- **Applies when**: `task` -- the deliverable is a file of predicted values produced by a model fit on separate training files, and the scripts go straight from `fit` to `predict` to writing the output.
- **Pattern**: The attempt trains a single model with hand-picked hyperparameters on all available labeled rows, never holds out a validation split (or cross-validates), never computes an error metric, never compares alternative models/targets (e.g. raw vs log-transformed, regression vs baseline median), and never checks that the predicted distribution resembles the labeled target distribution. It also skips checks on the label column itself (missing/NaN targets, extreme skew, dtype/format) and on feature alignment between train and predict frames; the final answer reports only self-descriptive stats (min/max/mean of predictions) as evidence of success.
- **Detection procedure**:
  1. Read the task to confirm the output is scored against unseen true values, so predictive accuracy — not file existence — is what matters.
  2. Scan the scripts for any train/validation split, cross-validation, or metric computation on labeled data; note whether the target is cleaned (NaN/zero/dtype handling) and whether any comparison against a trivial baseline is made.
  3. Compare the reported prediction summary statistics (mean, median, max, share of zeros) with the summary statistics of the target in the training data printed earlier; a large mismatch (e.g. predicted median orders of magnitude below the training median, or predictions collapsed near zero) is a red flag.
  4. Check that the written file's columns/rows match exactly what the task specified (required column name, row count and order matching the input file, no extra or renamed columns).
- **Discriminator**: A real violation is when *no* out-of-sample error estimate or baseline comparison exists anywhere, so the agent cannot know its model is better than a constant prediction; it is *not* a violation if the agent validated (split/CV metric or baseline comparison) and consciously chose a simple model, nor if a distribution shift between predictions and training labels is explained and justified by the validated setup.
- **Consequence**: The submitted file is accepted structurally but scores poorly on the grader's accuracy tolerance (predictions systematically biased/shrunk toward zero or dominated by one feature), yielding a WRONG file verdict despite a confident "task completed" report.

### Unjustified default hypothesis test on the full raw data (no population filter, no test-choice/direction check)
- **Applies when**: `task` -- the task asks for a p-value and a reject/fail-to-reject decision comparing two groups, and the scripts run a single canned test on every row of the loaded files.
- **Pattern**: The attempt equates "the population in the question" with "all rows in the files" and uses library defaults (parametric two-sample, two-sided, equal-variance) without inspecting whether the data support them: no restriction to the comparable subset implied by the question's framing (era/competition/level, matching time windows, comparable record types), no check of the distribution shape or of whether the alternative is directional, and no comparison of alternative test choices. Because the unfiltered samples are huge and heterogeneous, the p-value collapses to an astronomically small number and the decision is reported as certain.
- **Detection procedure**: 1. From the task statement and README, list the qualifiers that define the intended comparison population and the direction of the alternative; note anything that limits which records are comparable. 2. In the scripts, check whether any filtering/subsetting is done before the test and whether the test function's arguments (test family, tails, variance/paired assumptions) were chosen deliberately rather than left at defaults. 3. Check whether the script inspects distribution shape/skew, group sizes, and at least one alternative test or sensitivity variant before committing to one p-value. 4. Look at the reported p-value's magnitude and sample counts: an extreme value (e.g. <1e-20) computed on the entire files with no robustness check is a red flag that the subset and test were never validated.
- **Discriminator**: A real violation is when the script goes straight from `read_csv` to one default test and reports it, with no subsetting rationale, no distributional check, and no sensitivity analysis. It is *not* a violation if the script explicitly documents why the full data is the right population, states the tail/test family and why (e.g. checks skew and then uses a rank-based or one-sided test), and shows the decision is stable across reasonable variants.
- **Consequence**: The reported p-value (and possibly the reject/fail-to-reject string) comes from a different population and test than the reference analysis, so the graded value mismatches the expected result even though the output file format is correct.

### Ignoring the provided output template when formatting results
- **Applies when**: `task` -- the task says results must be written to a file "following the exact structure/formatting" of a provided sample/template file, and the scripts build the output frame themselves.
- **Pattern**: The attempt never loads or prints the sample/template file; it invents column names, column order, row order, and numeric precision from the aggregation output (e.g. renaming columns by guess, leaving full float precision, sorting by whatever is convenient), so the delivered file only coincidentally resembles the required schema.
- **Detection procedure**:
  1. In the task text, note that a template file defines the required header, column order, row order, and value formatting.
  2. Search the scripts for any read/print of that template file, and for an explicit comparison of the produced header/dtypes/row count/rounding against it.
  3. If absent, inspect the final answer for tell-tale signs of unconstrained formatting: unrounded floats with many decimals or float artifacts, hand-invented column names, and an ordering chosen by the author rather than copied from the template.
  4. Confirm no assertion or diff step validates the produced file against the template before submission.
- **Discriminator**: A real violation is when the template is never read and no schema/format check exists. It is fine if the script loads the template (or hard-codes its header/rounding with a comment quoting it) and asserts that produced columns, order, and precision match — even if it then rebuilds the frame manually.
- **Consequence**: The grader does an exact/near-exact comparison of the output file and reports it as WRONG/MISSING because of header naming, column/row order, or precision mismatch, even when the underlying aggregation logic is right.

### Deliverable filled with internal transformed artifacts instead of the requested feature values
- **Applies when**: `task` -- the task asks for an output file whose columns are the data's feature values alongside a derived label (cluster id, prediction, score), and the script builds that file inside a modeling pipeline that scales/encodes/imputes/reduces the data first.
- **Pattern**: The script writes the model's internal working matrix (standardized, label-encoded, PCA-projected, imputed-in-place) into the "feature" columns, and/or silently drops or reorders columns relative to the source data, so the saved features no longer correspond to the dataset the task described; hyperparameters of the unsupervised step are also accepted without any sanity check that the result is non-degenerate.
- **Detection procedure**:
  1. Read the task's output spec and note what the feature columns are supposed to hold (values from the dataset's feature vector) and what the label column is.
  2. In the script, trace the exact DataFrame passed to `to_csv`: is it built from the raw/original feature frame, or from the scaled/encoded/reduced array used for fitting? Check whether the number and order of columns matches the described feature vector.
  3. Compare the answer's self-report (column count, dtypes, value ranges) against the source data: all-float, zero-mean-looking columns, or a column count smaller than the described attribute list indicate a transformed artifact rather than the requested values.
  4. Check that the chosen number of clusters/labels was validated beyond a single automatic argmax (e.g., the label distribution isn't a near-trivial 2-way split chosen only because a score peaks at the smallest candidate).
- **Discriminator**: A real violation is when the saved feature columns cannot be mapped back to the dataset's values (scaled/encoded/projected numbers, or missing attributes). It is fine if the pipeline scales data internally for fitting but writes the original (or explicitly requested transformed) feature values to disk, with the row count and ordering preserved.
- **Consequence**: The output file fails column/value-level comparison against the expected feature matrix — the grader marks the file WRONG/MISSING even though the clustering code itself ran without error.

### Post-processing applied to submitted predictions but never included in validation
- **Applies when**: `task` -- a script trains a regressor/classifier, reports held-out metrics on raw model output, then transforms the predictions (rounding, casting to int, clipping to a range, rescaling) only on the way into the submission file.
- **Pattern**: The reported validation score is computed on untransformed predictions from a model fit on a different subset, while the file actually submitted contains a differently-transformed vector; the transform is justified by intuition ("the target looks like an integer count", "keep values in a plausible range") rather than by measuring its effect under the competition's evaluation metric. The agent then claims the submission is good based on the untransformed score.
- **Detection procedure**:
  1. Read the task/README and sample submission to determine the expected prediction dtype/precision and the scoring metric; note whether the target is scored as a continuous quantity.
  2. In the scripts, locate every operation applied to the test predictions after `predict()` (`np.round`, `astype(int)`, `np.clip`, inverse transforms) and check whether the identical operation is applied to validation predictions before the metric is computed.
  3. Check whether the reported metric came from a model trained on a split (or with parameters) different from the model that produced the submitted file.
  4. Look in the answer for any comparison of scores with vs. without the post-processing; absence means the transform is unvalidated.
- **Discriminator**: A real violation is a transform that can change the score (discretizing/clipping a continuously scored target, or altering the value range) and is absent from the evaluation path. It is fine if the transform is explicitly required by the task/sample format (e.g., integer class labels), or if it is applied consistently in validation and shown not to hurt the metric.
- **Consequence**: The submitted file scores measurably worse than the reported validation number (quantization/clipping error added on top of model error), so the graded submission fails the accuracy threshold even though the write-up looks clean.

### Silently dropping input rows during preprocessing so the output no longer covers every record
- **Applies when**: `task` -- the task asks for a per-record output file (e.g., labels/predictions for every row of a provided dataset) and the script performs cleaning, filtering, or missing-value handling before producing it.
- **Pattern**: The script removes records deemed "too incomplete" (or drops NaNs, outliers, non-parsable rows) instead of imputing/encoding them, then writes an output whose row count is smaller than the input; the answer reports the reduced count as if it were the requested deliverable, with no mapping back to the original rows.
- **Detection procedure**:
  1. Read the task statement and note whether any row filtering was requested; if none is stated, the deliverable is expected to have exactly one output row per input row, in the original order.
  2. Scan the script for row-reducing operations (`dropna`, `dropna(thresh=...)`, boolean masks, dedup, outlier removal) applied before the final write, and check whether the dropped rows are ever re-attached.
  3. Compare the row count printed/claimed in the answer against the raw file's row count; also verify the column set/order and naming convention match exactly what the task specified.
  4. Flag if the counts differ or if the script cannot reconstruct labels for the excluded rows.
- **Discriminator**: A real violation is unrequested, silent row loss in the final deliverable; it is fine if the task explicitly asks for filtering, or if rows are dropped only for fitting/intermediate steps but every original row is still assigned an output value (e.g., transform/predict on the full imputed set and re-join by index).
- **Consequence**: The saved file has fewer rows than the reference, so a row-wise or shape-based comparison fails outright regardless of how sensible the clustering/model itself is.

### Degenerate clustering accepted without cluster-balance / skew sanity checks
- **Applies when**: `task` -- an unsupervised segmentation task where the script builds aggregate numeric features from transactional/heavy-tailed raw data, scales them, and selects the number of groups from an internal score (silhouette, inertia, etc.).
- **Pattern**: Highly skewed, outlier-dominated features are fed in with only z-score scaling (no log/rank transform, no winsorizing/outlier removal), so the internal score is maximized by a solution that peels off a handful of extreme points and dumps ~all rows into one giant group; the agent reports the near-1.0 score as "success", or arbitrarily overrides the chosen k by hand-waving, and never inspects per-cluster counts or profiles for meaningfulness.
- **Detection procedure**:
  1. From the task, note that the deliverable is a *segmentation* — i.e., labels must partition the population into usable, non-trivial groups, not flag outliers.
  2. In the script, check whether the engineered features are long-tailed monetary/count/quantity aggregates and whether any transform beyond `StandardScaler` (log1p, quantile/rank, robust scaling, outlier capping) or correlated/redundant-feature pruning is applied; also check whether many features are near-duplicates of one another.
  3. Check whether cluster selection is validated by anything besides a single internal score — e.g., an assertion or printed check that no cluster is a tiny singleton and that no cluster holds an overwhelming majority, plus interpretable cluster profiles.
  4. In the answer/logs, look for suspiciously high silhouette values (>~0.8) and/or a printed size distribution like `[N-2, 1, 1]`; if present and unaddressed (or if k was changed by fiat rather than by evidence), flag it.
- **Discriminator**: A real violation is one where the label distribution is effectively trivial (one cluster ≈ all rows, others a few points) or where the score is high only because of untransformed extreme outliers; it is *fine* if clusters are moderately imbalanced but each holds a substantive share of the population and profiles differ interpretably, or if outlier handling/transformation was explicitly done and justified.
- **Consequence**: The saved label column carries almost no information (near-zero adjusted mutual information / very poor agreement with any reasonable reference segmentation), so the output file is judged wrong even though the column names and row count look correct.

### Self-reported training-set fit passed off as predictive accuracy (no held-out/temporal validation)
- **Applies when**: `task` -- the agent must produce out-of-sample predictions for a provided evaluation file and reports model quality metrics to justify its answer.
- **Pattern**: The scripts fit a model and compute R²/RMSE/MAE on the same rows used for fitting (or on a random split of a time-ordered series), then report those numbers as evidence that the submitted predictions are good, without any honest hold-out, time-based split, or cross-validation, and without sanity-checking the prediction file against the evaluation file (row count, order, index alignment, plausible value range).
- **Detection procedure**:
  1. In the task, note that scoring happens on unseen rows of a separate evaluation file, so only out-of-sample error is meaningful.
  2. In the scripts, locate where the metric is computed and check whether the data passed to `score`/`metric` is disjoint from the data passed to `fit`, and whether the split respects any time ordering; also check whether feature engineering (imputation, scaling, encoding, lag/rolling features) was fitted on training data only and applied identically to the evaluation rows.
  3. In the answer, check whether the quoted metrics are labeled as training performance and whether any independent validation number is given.
  4. Verify the answer states a prediction count/index that matches the evaluation file's row count and ordering, plus a value range consistent with the historical target distribution.
- **Discriminator**: A real violation is metrics computed on fitted rows (or a shuffled split of temporal data) with no other validation evidence; it is fine if the agent reports in-sample numbers *in addition to* a proper hold-out/time-series CV estimate, or if the split is genuinely disjoint and the reported figure is explicitly the validation figure.
- **Consequence**: Optimistically inflated reported accuracy while the submitted file's actual error exceeds the grader's tolerance (or is misaligned/mis-shaped), so the result file is scored WRONG despite a confident-sounding report.

### Fabricated or synthetic data substituted for the provided dataset
- **Applies when**: `task` -- the task points to a real input dataset (files in a data directory) that must be loaded, filtered, and plotted/aggregated to produce the answer artifacts.
- **Pattern**: The agent fails to locate/parse the actual source file (or finds it inconvenient) and instead invents "realistic" values, hard-codes a plausible trend, or synthesizes a date range, then reports the chart/statistic as if derived from the dataset; required side artifacts (e.g., serialized plot data or numeric arrays) are absent or built from the invented numbers.
- **Detection procedure**:
  1. Read the task to identify which input file(s) the answer must be derived from, and which output artifacts are expected besides the image (data dumps, arrays, JSON).
  2. Scan the scripts for an actual read of that input (`read_csv`/`read_excel`/`open`) feeding the plotted series; flag any literal arrays, `np.random`, `np.linspace`, or manually typed values used as the plotted data.
  3. Check the answer's wording for phrases like "created/simulated/based on specifications" and check that the reported time span, row count, and value magnitudes are traceable to the file rather than asserted.
  4. Verify every expected output artifact is written by the script, not just the PNG.
- **Discriminator**: Real violation = the plotted/reported numbers cannot be traced to any read of the provided data. Fine = synthetic values used only for a smoke test or for styling defaults, while the final artifacts are regenerated from the loaded file.
- **Consequence**: Expected data artifacts are missing or numerically mismatched, so all value-based checks fail even though a plausible-looking image exists.

### Output file schema not verified against the provided template
- **Applies when**: `task` -- The task says to write results into a result file "in the format specified" by a provided sample/template file (sample_result.csv, schema doc, submission example).
- **Pattern**: The agent invents its own column set, column names, row layout, or value precision (e.g., adds descriptive/narrative columns, renames headers, writes a scientific-notation or unrounded value) instead of reading the template and reproducing its exact header and row structure with only the requested quantity filled in.
- **Detection procedure**:
  1. From the task statement, locate the named template/sample file and list the exact header fields, number of rows, and any implied value formatting (rounding, units, ordering).
  2. In the scripts, find where the output is written and check that the header/row structure is taken from (or literally matches) the template — not hard-coded from the agent's own narrative.
  3. Compare the produced answer's columns and row count field-by-field with the template; flag any extra, missing, renamed, or reordered fields, and any value whose formatting/precision differs from the template's example value.
  4. Check the single requested quantity is present in the cell the template designates for it (not buried among intermediate values).
- **Discriminator**: A real violation is a structural/format mismatch with the template (extra or renamed columns, wrong row count, wrong cell for the requested statistic, formatting the template forbids). A look-alike that is fine is a file matching the template exactly but whose numeric value differs slightly due to legitimate stochastic variation within tolerance.
- **Consequence**: The grader's file/column comparison fails outright ("WRONG/MISSING"), scoring 0 even if the underlying statistic was computed correctly.

### Referenced spec file never read; grouping definition assumed
- **Applies when**: `task` -- the task points to an auxiliary spec/instructions file (e.g. a README/markdown defining categories, bins, or rules) and the scripts must implement that definition.
- **Pattern**: The scripts never open or print the referenced file; the agent instead infers the grouping/binning from the raw column's own values (or invents its own bins), then asserts in the answer that the spec was followed. Additionally, only the requested image is produced while other required output artifacts are skipped.
- **Detection procedure**: 1. List every external file and every output artifact the task names or implies. 2. Grep the scripts for a read of each referenced spec file and for code that encodes its rules; if absent, the definition was assumed. 3. Compare the categories used in the code against what the raw data natively contains — identical-to-raw categories are a sign no re-grouping/aggregation was applied. 4. Check that every required output file (plot data, arrays, figure) is actually written.
- **Discriminator**: Fine if the script demonstrably loads/quotes the spec (or the spec's rules are transcribed verbatim into the code with matching bin edges/labels) and all named artifacts are saved; a violation is when the grouping comes solely from `unique()` of the raw column or hard-coded guesses with no evidence the spec was consulted.
- **Consequence**: Bin labels/counts differ from the specified grouping and required files are missing, so all artifact comparisons (figure, plot data, array) fail even though the chart looks plausible.

### Required output artifact/schema is never produced by the scripts
- **Applies when**: `task` -- the task specifies a concrete deliverable (a result file and/or an exact answer template with specific keys, types, or list-valued fields) and the agent's scripts only compute and print values.
- **Pattern**: Every script ends in `print(...)` of intermediate/final numbers; none serializes the result to the expected file path, and the pasted answer silently deviates from the requested schema (e.g., scalars where the template shows bracketed lists, renamed/missing keys, unrounded or differently-typed values).
- **Detection procedure**:
  1. From the task statement, write down the exact deliverable: file name/path (if any), key names, value types/containers, and any rounding/unit constraints.
  2. Grep the scripts for any write operation (`json.dump`, `to_csv`, `open(..., 'w')`, `to_json`) targeting that path; if absent, the deliverable is unmet regardless of the computation.
  3. Compare the agent's final answer text field-by-field against the template: same keys, same container type (list vs scalar), same numeric formatting.
  4. Confirm the emitted values are the *requested* quantity (final answer), not an intermediate diagnostic printed along the way.
- **Discriminator**: A real violation is a missing output file or a structural mismatch with the stated template; it is *not* a violation if the file is written elsewhere in the pipeline, or if the grader only checks pasted text and the text matches the template exactly (harmless cosmetic differences like key order or whitespace).
- **Consequence**: The grader reports the expected result file as WRONG/MISSING and scores 0 even when the underlying computation may be right, since no parsable, schema-conformant artifact exists.

### Named-formula variant substituted for the one the task specifies
- **Applies when**: `task` -- the task asks for a specific, named statistic/metric that has several commonly used definitions or variants (e.g., mode-based vs. median-based coefficients, population vs. sample denominators, macro vs. micro averaging, biased vs. unbiased estimators).
- **Pattern**: The script computes a *related but different* formula — often whatever a library function or the most familiar textbook version provides — instead of the literally named variant, and reports a plausible-looking number with the correct sign/direction, so only the magnitude is wrong.
- **Detection procedure**:
  1. From the task text, write out the exact algebraic definition of the named statistic, including which central-tendency term and which dispersion estimator (and degrees of freedom) it requires.
  2. In the script, locate the line that produces the reported number and expand any library call (e.g., a generic `.skew()`, `.std()`, `average=` argument) into its underlying formula and default parameters.
  3. Compare term by term: is each component (mean/median/mode, ddof, normalization, averaging scheme) the one named, and are all preprocessing constraints (e.g., the specified missing-value handling) applied before it?
  4. Check the answer's magnitude against a quick hand-recomputation of the named formula on the same cleaned data; a mismatch beyond rounding flags the attempt.
- **Discriminator**: A real violation is when the computed quantity is algebraically different from the named one (different central term, different denominator convention, different averaging) even if it agrees in sign. It is *not* a violation if the script uses a library shortcut that is provably identical to the named formula with the required parameters explicitly set, or if the difference is only final-step rounding.
- **Consequence**: The direction/category part of the answer passes while the numeric field is off (here, 0.66 vs. 0.83), so the grader marks the submission wrong on the value check despite a correct-looking interpretation.

### Failure to validate raw inputs for missing/invalid values before aggregating them into a cumulative or weighted result
- **Applies when**: `task` -- the script reads a raw tabular file and immediately combines multiple numeric columns (weighted sums, means, products, cumulative/running aggregations) into a derived series or final statistic.
- **Pattern**: The attempt assumes the input is clean and complete: it never inspects null counts, dtypes, row/date coverage, or value ranges, and it feeds the columns straight into arithmetic. Any NaN (or non-numeric/placeholder value) silently poisons the weighted sum for that row and, once a running product/sum is involved, propagates to every subsequent row — while the "verification" script re-runs the identical formula on the identical data and therefore always agrees.
- **Detection procedure**:
  1. Read the task/README and note which columns are required for every row and over what full index (all periods/entities) the output must be defined.
  2. Scan the loading section of the scripts for any explicit data-quality step: `isna().sum()`, `dtypes` check, `fillna`/`dropna` with a justified rule, coverage/shape assertion. If none exists, flag it.
  3. Scan the verification/QA script: check whether it independently validates the output (no NaNs, monotone/plausible ranges, expected row count, cross-check against a differently-derived quantity) or merely recomputes the same expression and compares to itself.
  4. Check the reported final numbers and the saved file for tell-tale signs of contamination: NaN/blank cells, series that flatten or vanish partway, or an implausibly small/large final value.
- **Discriminator**: A real violation is when the raw source can plausibly contain gaps or non-numeric entries and the script neither checks nor handles them (and self-consistent "verification" hides it). It is fine if the script explicitly demonstrates the input is complete/numeric, or applies and justifies an imputation/exclusion rule consistent with the task's definition.
- **Consequence**: The saved output contains NaNs or values computed from a corrupted running aggregate, so the file mismatches the expected reference values (often for all rows after the first bad one) and the file-comparison check fails even though the agent's internal cross-check "passed".

### Silently narrowing the candidate set when the task says "all other numerical variables"
- **Applies when**: `task` -- the task asks to scan every numeric column (or every feature) against a target and report the extremum (max |r|, best score, etc.).
- **Pattern**: The script drops or never includes some numeric columns — rank/ID/index-like fields, integer-typed columns, columns the agent judged "not meaningful" or derived from the target — and then reports the winner from the reduced set, so an obviously stronger (often near-perfect, monotone) relationship is never computed.
- **Detection procedure**:
  1. From the task, note the exact selection rule ("all other numerical variables") and the exclusion it authorizes (only the target itself).
  2. In the script, find the column-selection step (`select_dtypes`, hardcoded lists, `drop(...)`, filters on name patterns) and list which numeric columns are actually fed into the loop; compare against the dataset's full numeric dtype set.
  3. Check that the reported statistics table is printed/inspected in full and sorted by |value|, not just the single claimed winner, and that ties/near-1 values are present as expected.
  4. Confirm the reported variable and sign come from that full table, not from a subjective "most interesting predictor" choice.
- **Discriminator**: A real violation is dropping a column that satisfies the stated numeric criterion without the task authorizing it (including rank/index columns or anything deemed "trivially related"); it is fine to drop non-numeric columns, the target itself, or columns explicitly excluded by the task instructions.
- **Consequence**: The extremum is taken over a subset, so both the reported variable and its sign can differ from the true answer — the grader marks both the variable and the direction wrong.

### Unvetted input vector for a distributional test (no reported n / p-value evidence)
- **Applies when**: `task` -- the task asks for a hypothesis test and/or shape statistics on a single column, with an explicit decision rule (e.g., compare a p-value to a stated alpha).
- **Pattern**: The attempt pulls the column straight into the test/statistic function without first establishing exactly which values enter it (row count after dropping missing values, sentinel/placeholder codes such as -999 or 0-fills, duplicate or aggregate rows, non-numeric strings coerced or dropped), and then reports only the final verdict and rounded shape statistics — omitting the intermediate evidence the task explicitly asked for (the test statistic/p-value and the sample size). The reported verdict therefore cannot be traced back to a defensible input vector, and a contaminated or wrongly-sized vector silently flips the decision.
- **Detection procedure**:
  1. Read the task and list the required outputs, including any intermediate value it says to report (p-value, n, alpha comparison) and the exact decision rule.
  2. In the scripts, locate the line that builds the vector fed to the test and check that it is preceded by explicit inspection/handling: dtype coercion, `isna()` count, `describe()`/`value_counts()` to expose sentinels, and the resulting `len()` printed.
  3. Check the answer text for the p-value and n; if the scripts are missing or the p-value/n is absent, the result is unverifiable and the check fails.
  4. Cross-check plausibility: does the reported n fall in a range where the chosen test is meaningful, and are the reported shape statistics consistent with the p-value under the stated alpha (very small n rarely rejects; huge n rejects trivially)?
- **Discriminator**: A real violation is an attempt where the input vector's composition is never shown (no n, no NaN/sentinel handling, no p-value) so the verdict rests on an unexamined slice; it is *not* a violation if the script prints n, missing-value counts and the p-value and the cleaning choice is justified — even if a reviewer would have cleaned slightly differently.
- **Consequence**: The test runs on a different sample than the task intends (extra sentinel/missing-derived values or extra rows), producing an inflated skew/kurtosis and a p-value on the wrong side of alpha, so the boolean verdict field is graded WRONG and the whole item scores 0 despite plausible-looking numbers.

### Underfit / unvalidated predictions never checked against the training target distribution or a held-out score
- **Applies when**: `task` -- the agent trains a model on a training file and writes predictions for a test file, with no ground-truth labels available for the test set.
- **Pattern**: The attempt builds a quick, capacity-limited or arbitrarily hand-weighted ensemble, reports only descriptive stats of its own predictions and file/row counts, and never reports a held-out validation score on the competition metric, never compares the prediction distribution to the training label distribution, and never compares against a trivial baseline (predict the mean / a single well-tuned model). The predictions come out heavily shrunk toward the mean (much smaller spread/range than the training target), which the summary presents as success.
- **Detection procedure**:
  1. Read the task/sample submission to identify the target and the evaluation metric implied by the competition (e.g. R²/RMSE/AUC).
  2. Scan the scripts for a train/validation split (or CV) and an explicit metric computation on held-out data; note whether any score is printed, and whether model capacity/hyperparameters were chosen by that score rather than fixed arbitrarily.
  3. Compare the reported prediction summary statistics (min, max, std, mean) with the training target's statistics; flag when prediction spread is a small fraction of the target spread or the range is far narrower.
  4. Check the answer for any baseline comparison; an answer that lists only architecture, row counts and prediction stats, with no validation number, fails.
- **Discriminator**: A real violation is the absence of any held-out metric *plus* a prediction distribution visibly inconsistent with the target's (severe shrinkage) or capacity chosen for speed alone. It is *not* a violation when the model is genuinely well-validated and the narrow spread is justified — e.g. a reported CV score near the achievable ceiling, or a classification task where calibrated probabilities legitimately concentrate; nor when the metric is rank-based and only ordering matters and that is verified.
- **Consequence**: The submission file has the right shape and passes format checks but scores far below threshold (near-constant, mean-reverting predictions give near-zero R²/poor RMSE), so the grader marks the expected result file as wrong despite the confident summary.

### Invented qualification threshold / aggregation rule instead of using the spec's stated definition
- **Applies when**: `task` -- The task or its documentation defines the target quantity with an explicit eligibility filter or aggregation rule (e.g., "must have at least N …", "average over …"), and the scripts must reproduce that definition exactly.
- **Pattern**: The agent does not locate or fully read the stated definition (README section, sample/format file, or truncated spec), then hard-codes a self-chosen cutoff and a self-chosen aggregation (which entity attribute is summed vs. averaged, and which quantity the filter is applied to), presenting it as "as per guidance" without any citation of the source text.
- **Detection procedure**:
  1. In the task/README, find every stated definitional constraint (minimum counts, which field the minimum applies to, how per-entity values are aggregated, tie-breaking, ordering, output columns) and note anything truncated or ambiguous.
  2. In the scripts, locate each corresponding constant and aggregation call and check it against the text; also check whether the script ever loads/prints the provided sample-format file or the README to resolve ambiguity.
  3. Flag if any threshold, filter field, or aggregation is chosen by the agent rather than traceable to the spec, or if an ambiguity in the spec was silently resolved without a sensitivity check across plausible alternatives.
  4. Compare the produced output's columns/row count/ordering to the sample format file.
- **Discriminator**: A real violation is a constant or aggregation that appears nowhere in the task text (or contradicts it) and materially changes the eligible population or ranking; a look-alike that is fine is an agent-chosen tie-break or implementation detail that is explicitly demanded by the spec or demonstrably does not change the reported ranking (verified by a robustness check).
- **Consequence**: The eligible set and the ranking differ from the reference, so the saved file's entity lists mismatch the expected answer and the exact-match file check fails.

### Bins/categories invented ad hoc, silently dropping records instead of following the spec and covering the full data range
- **Applies when**: `task` -- the task asks for counts/distribution over ranges or groups of a numeric field, with a provided config/spec file, and the script defines bin edges and labels itself.
- **Pattern**: The script hard-codes bin boundaries (and a label like "121+") that do not actually cover the observed min–max range, uses `pd.cut` without `include_lowest`/open upper edge, and mislabels the resulting categories; records outside the edges (and rows whose values failed the parsing step) are dropped without any check, so the plotted/reported counts describe only a subset. Additionally, the group definitions and required output artifacts are taken from the agent's own assumptions rather than exhaustively from the provided spec file.
- **Detection procedure**:
  1. From the task, list every constraint that the provided spec/config file is supposed to supply (group boundaries/labels, ordering, and every required output file) and note that the analyst must not substitute its own.
  2. In the script, check whether bin edges/labels are hard-coded rather than read from the spec, and whether the top/bottom bins are open-ended enough to include the printed min/max of the parsed values.
  3. Verify the script asserts that the sum of the group counts equals the number of rows in the intended population (after the legitimate type filter), and that parsing failures are inspected rather than silently dropped.
  4. Compare the counts and the artifact list in the final answer against the population size and required outputs; a large unexplained gap (e.g., counts summing to far fewer than the filtered rows) or missing artifacts is a violation.
- **Discriminator**: A real violation is when rows in the target population vanish with no justification, labels do not match the edges actually used, or spec-provided grouping/outputs are replaced by invented ones. It is fine if rows are excluded by an explicitly required filter (e.g., a different content type) and the script documents/asserts that the retained counts sum to the filtered total.
- **Consequence**: The produced figure and any saved count arrays/JSON disagree with the reference bin structure and totals, so every artifact check fails even though the script runs without error.

### Answer not traceable to executed code on the provided data (plausible-looking values supplied from prior knowledge)
- **Applies when**: `task` -- the task asks for a ranking/statistic that must be computed from a supplied data file (with a stated preprocessing step such as a specific imputation rule), and the deliverable is a small JSON/text answer.
- **Pattern**: The attempt produces a confident, plausible-sounding list of entity names without a saved, runnable script that loads the file, applies the required preprocessing, computes the ordering, and writes the requested output file; the values are effectively recalled from general knowledge or from a partially-read file, so label spellings, entity coverage, and the applied preprocessing/sort order cannot be verified against the actual data.
- **Detection procedure**:
  1. Read the task and list every mandated operation (preprocessing rule, sorting direction, top-N size, output keys/filename/format).
  2. Look for a script that performs each mandated step end-to-end and persists the answer to the required artifact; if no script exists, or it stops before computing/writing the final ranking, flag immediately.
  3. Cross-check the reported entity labels against the data file's own label column (exact strings, e.g. official vs. colloquial names) and confirm the reported list is a contiguous head/tail of the sorted, imputed column with the requested sort direction.
  4. Sanity-check counts and values: N entries per list, no duplicates/overlap between lists, and values within a physically plausible range for the quantity.
- **Discriminator**: A real violation is an answer that cannot be reproduced from any artifact, or whose labels/order differ from what a script over the file would yield; a look-alike that is fine is an answer whose script exists and reproduces exactly these labels (matching the file's spellings) even if the agent also happened to know the result in advance.
- **Consequence**: The graded artifact is missing or contains names/order that don't match the file-derived ranking, so exact-match comparison against the expected result file fails (0/1 checks).

### Ambiguous sign/absolute-value convention when constructing "difference" variables instead of using the dataset's own columns
- **Applies when**: `task` -- the question asks about a relationship (correlation, regression, comparison) between two "difference"/"gap"/"change" quantities, and the script derives one or both of them by subtracting raw columns even though the file already contains pre-computed difference columns.
- **Pattern**: The attempt picks one arbitrary orientation (e.g. B − A) and/or a signed-vs-absolute convention for the derived variable, without checking against the ready-made column(s) in the data that encode the same quantity; the two variables end up on mismatched conventions (one signed, one absolute, or opposite orientations), so the statistic's sign and its p-value/magnitude differ from the intended pairing. No validation of the derived column against the existing one is performed.
- **Detection procedure**:
  1. From the task, list the two quantities to be related and note that neither an orientation nor a signed/absolute convention is specified.
  2. In the scripts, find where each quantity is obtained; check whether an already-present column represents that same quantity and whether the script uses it or recomputes it.
  3. Verify the script explicitly compares its derived column to the existing one (equality/correlation check on both sign and magnitude, or a check for whether either column is non-negative everywhere) and states which convention it adopts for *both* variables consistently.
  4. Check the reported statistic for tell-tale inconsistency: a sign or p-value that changes if one variable were flipped or made absolute, with no justification in the script output.
- **Discriminator**: A real violation is when the two variables are built under different conventions, or a provided difference column is silently ignored/overridden, so the result is convention-dependent and unverified. It is fine if the script uses the dataset's own difference columns directly, or shows a validation step proving its recomputation matches them (including sign), or if the statistic is provably invariant to the flip (e.g. both variables flipped together).
- **Consequence**: The relationship classification may still land correctly by luck, but the correlation's sign and especially the p-value differ from the intended pairing, so the numeric fields (r sign, p-value to 4 decimals) fail the grader.

### Entity/quantity substitution: analyzing different data than the task specifies
- **Applies when**: `task` -- the prompt names specific entities, groupings, and measured quantities (e.g. a ranking dimension, a per-group aggregate) and specific output artifacts, and the scripts choose their own data source and variables.
- **Pattern**: The attempt loads whichever file it happens to find in the working directory and computes a superficially similar chart (same chart type, same "top 10" framing) over entirely different entities and metrics than the task requested, then declares success because a plot was produced.
- **Detection procedure**:
  1. From the task statement, list the required grouping key, the ranking metric, the measured quantity per segment, and every required output file.
  2. Read the scripts: identify the input file(s) actually loaded and the columns used for grouping, ranking, and the stacked segments.
  3. Check each item in the list from step 1 has a concrete counterpart in step 2; treat any silent substitution (different entity type, different ranking metric, different segment definition) as a failure, even if the config file was read.
  4. Confirm the answer/scripts write *all* named output artifacts, not just the image.
- **Discriminator**: A real violation is a mismatch in the semantic content — different grouping entities, different ranking basis, or different measured quantity than asked. A look-alike that is fine is when the required columns are present but merely renamed/derived (e.g., a duration computed from timestamps), with the derivation clearly matching the requested definition.
- **Consequence**: All expected output files fail comparison — the numeric array and the rendered figure encode unrelated quantities, and any missing artifact scores zero outright, so the run fails every check despite a "success" report.

### Multi-stage grouping: filtering/statistic computed at the wrong grouping level
- **Applies when**: `task` -- the task prescribes a preprocessing step defined per one grouping variable (e.g., "filter within each level of A") followed by a test/metric computed across the levels of a *second* grouping variable, and the script implements both stages.
- **Pattern**: The attempt collapses or shifts the grouping scope — it computes the filter thresholds globally over the whole table, or per cell of the A×B cross-tabulation, or per level of B instead of A — and/or runs the subsequent test on data that was not filtered as specified. Because the trimming changes each group's spread, the resulting variance/dispersion statistic and its p-value silently change sign of the conclusion, and no row-count or threshold check is emitted to expose it.
- **Detection procedure**:
  1. From the task text, write down explicitly: which variable defines the filtering partition, which variable defines the comparison groups, and which numeric column the thresholds are computed on.
  2. In the script, locate the `groupby`/mask that produces the thresholds and confirm the key matches the filtering partition (not the comparison groups, not the whole frame, not both keys), and that the thresholds are applied to the same numeric column named in the task.
  3. Confirm the statistical test is then run on the *filtered* subset, one result per level of the filtering partition, with groups formed by the comparison variable and all levels present (check group sizes are non-zero and that the number of reported values equals the number of partitions).
  4. Check the script prints pre/post filter row counts and per-group counts/thresholds; if no such sanity output exists, and the reported conclusions are uniform across all partitions, treat the result as unverified.
- **Discriminator**: A real violation is a grouping key (or filter target column) in the code that differs from the one the task names, or a test run on unfiltered/differently filtered data. A look-alike that is fine: a script that groups exactly as specified but implements the quartile/outlier bound with a defensible convention (e.g., interpolation method or inclusive vs. exclusive bounds), which shifts p-values only marginally and preserves counts.
- **Consequence**: The p-value list is numerically off (often by an order of magnitude near the decision threshold), one or more conclusion strings flip, and the emitted result file mismatches the expected values — scored wrong despite correct-looking format.

### Settling for a near-baseline model instead of exploiting the identifying/high-signal columns shared by train and test
- **Applies when**: `task` -- the task asks for per-row predictions of a target and the provided test file carries many of the same descriptive/identifier/metadata columns as the labeled source file, but the script builds features from only a generic numeric subset.
- **Pattern**: The agent hand-picks a handful of "obvious" numeric columns, drops all categorical/identifier/date/text fields, trains, observes a validation score barely above predicting the mean (R² ≈ 0, or predictions collapsed into a narrow band around the global mean), and ships those predictions anyway while describing the near-mean output as a positive sign ("mean matches training mean").
- **Detection procedure**:
  1. From the task/README, list every column present in both the labeled file and the test file; note identifiers, names, dates, and categorical metadata that plausibly carry strong signal about the target.
  2. In the scripts, check the feature list actually used and whether any step tests for exact or key-based overlap between test rows and labeled rows (a lookup/merge that could recover many targets directly).
  3. Read the reported validation metric and the prediction spread: if R² is near zero (or RMSE ≈ target std) and predicted min/max span a small fraction of the true target range, the model is essentially a constant predictor.
  4. Confirm the agent never compared its model against a trivial baseline (global mean / group mean by an unused categorical key) or tried adding the discarded columns.
- **Discriminator**: A real violation is when informative shared columns were available and simply ignored, and the resulting score is indistinguishable from the mean baseline. It is *not* a violation if the discarded columns are truly unavailable/leaky in the test file, or if the agent tried them, documented that they did not help, and the low score reflects a genuinely weak signal validated against a baseline.
- **Consequence**: The submitted prediction file is essentially a constant column; any accuracy/correlation/error threshold the grader applies against the true targets fails, and near-duplicate rows whose targets could have been recovered exactly are mispredicted.

### Requested output file contains internal intermediates instead of the original data values
- **Applies when**: `task` -- the task asks for a result file whose columns echo the input feature values alongside a computed label/prediction, and the script applies a transform (scaling, PCA, imputation, encoding) before modeling.
- **Pattern**: The attempt builds the output table from the transformed matrix used inside the model (e.g. z-scored or projected values) rather than from the original input rows, and/or silently drops/reorders identifier columns and rows, so the file's feature columns no longer match the source data even though the label column is plausible.
- **Detection procedure**:
  1. Read the task to determine exactly what each requested column should hold (raw feature values vs. derived ones), the expected row count, and column naming/order.
  2. In the script, find the object passed to `to_csv` and trace it back: is it derived from the raw dataframe, or from the output of a scaler/decomposition/encoder?
  3. Check whether every original row is present in original order and whether the number/meaning of feature columns equals the number of input features (a transform that changes dimensionality, or a fit on a subset, is an extra red flag).
  4. Compare the answer's description of the file (sample rows, value ranges) against the source data's ranges — negative/near-zero-mean values where the raw data is positive counts, rates, or currency indicate a transformed dump.
- **Discriminator**: Using a transform *inside* the model is correct and expected; the violation is only when the *saved output* stores transformed values where the task implies the original feature vector, or when rows/columns of the saved file no longer align 1:1 with the input records. If the task explicitly asks for scaled/embedded coordinates, writing them is fine.
- **Consequence**: The result file fails value-level comparison against the expected file (feature columns mismatch, or join on records fails), so the check scores 0 even though the clustering/labels themselves may be reasonable.

### Stated subset/scope qualifier silently redefined instead of applied
- **Applies when**: `task` -- the question names an explicit slice (a specific year/category/group) and a population scope ("among all countries/records in the dataset"), and the script computes a per-entity statistic from a wide/multi-file table.
- **Pattern**: The agent decides the named slice is "impossible" for the statistic (e.g., a single value can't be skewed), reinterprets the question by aggregating over a different axis (all periods instead of the named one), and/or loads only one of several files that together form the stated population — then rationalizes the substitution in comments rather than testing the literal reading.
- **Detection procedure**:
  1. From the task text, list every qualifier: the filter value, the entity being ranked, the population scope, and the required definition/flag of the statistic.
  2. In the scripts, locate where each qualifier is enforced: is there a filter on the named slice? are all relevant input files/rows loaded? does the reduction axis correspond to the entity in the question?
  3. Flag the attempt if any qualifier appears only in a comment/justification (e.g., "this must mean X instead") or is replaced by a broader/narrower set, or if the flag for the requested definition is asserted without checking library semantics against the documented meaning.
  4. Check the reported answer is the entity that maximizes the statistic under the literal qualifiers, not under the substituted interpretation.
- **Discriminator**: A real violation is when the literal slice/scope was never computed or even attempted, and the deviation is defended by prose only. It is acceptable if the agent computes the literal reading, demonstrates with evidence that it is degenerate or empty, and then documents a fallback while still covering the full stated population and correct statistic definition.
- **Consequence**: The ranking is over a different distribution/population than requested, so the top entity differs from ground truth and the graded key mismatches (0/1), even though the code runs cleanly.

### Reference format file never opened; output schema guessed instead of derived
- **Applies when**: `task` -- the instructions require the deliverable to match the format of a provided sample/template file (column names, order, row set, rounding, index).
- **Pattern**: The scripts compute the statistic but never load, print, or compare against the supplied template; the agent invents header names, column order, row labels and rounding from the prose of the task, and writes the file without any format assertion. A second "verification" script only re-checks the arithmetic, not the output contract.
- **Detection procedure**:
  1. Read the task and note every stated output constraint and the name of any reference/sample artifact.
  2. Grep the scripts for that artifact's filename and for any read of it; check whether the produced column names, ordering, row keys and numeric formatting are taken from it (or explicitly asserted equal to it) rather than hard-coded.
  3. Check whether the final write step is preceded by a comparison of the produced frame's shape/columns/row labels to the template's.
  4. Inspect the submitted file: are its headers, label spellings, row ordering and decimal precision provably derived from the template, or plausible-looking guesses?
- **Discriminator**: A real violation is hard-coded or renamed output fields with no read/assert of the reference file; it is fine if the script reads the template (or an explicit spec in the task text) and the headers/labels/precision demonstrably come from it — even if the names happen to be simple.
- **Consequence**: The grader compares against the expected file and reports the deliverable as WRONG/MISSING, because header text, label spelling, row coverage/order or rounding differ, even when the underlying aggregation may be numerically close or correct.

### Truncating or reformatting an identifier value to match a template instead of reporting the value found in the data
- **Applies when**: `task` -- the deliverable includes an identifier (a date, key, ID, label) that must be looked up in the data and reported, and the answer template shows a shortened/abstract pattern (e.g., a coarser granularity than the records carry).
- **Pattern**: The script correctly locates the row of interest but the reported identifier is coerced to the literal template pattern (formatted/truncated/rounded to a coarser unit), discarding precision that exists in the source records, so the reported key no longer uniquely identifies the row actually used in the downstream computation.
- **Detection procedure**:
  1. In the task, note the identifier requested and the granularity actually present in the data (row-level key vs. aggregated period); note whether the task ever asks for aggregation to that coarser unit.
  2. In the scripts, find where the identifier is emitted and check for any `strftime`/slicing/`astype(str)[:n]`/rounding/relabeling applied only for output formatting.
  3. Compare the emitted identifier with the identifier of the row used for the dependent calculation (e.g., the neighboring-row lookup); if the dependent step uses full precision but the printed key is coarser, flag it.
  4. Check the printed value is still an existing, unique key in the data — a coarser string that matches many rows is a red flag.
- **Discriminator**: Fine if the task explicitly required aggregation to the coarser unit and the analysis was performed at that unit throughout; a violation when the selection/computation happens at fine granularity and only the printed string is truncated to imitate the format example.
- **Consequence**: The dependent numeric answer matches but the identifier check fails on exact-string comparison, so the submission is graded partially correct / incorrect.

### Fabricated/synthetic stand-in data instead of the provided dataset
- **Applies when**: `task` -- the task asks for a statistic computed from a specific real dataset, and the scripts hard-code, simulate, or "reconstruct from memory" the input values.
- **Pattern**: An initial exploration script fails to locate (or gives up on locating) the real input files, so the agent hand-writes plausible-looking arrays or generates random data, runs the requested computation on that invented input, and reports the resulting number as if it were the real answer.
- **Detection procedure**:
  1. Read the task to identify what input data the requested statistic must come from, and note that the answer is data-determined (a single number/table).
  2. Scan the analysis scripts for the data source: is every model input traced back to a file that was actually read (`read_csv`/`read_excel`/API), or does it come from literal arrays, `np.random`, or comments like "based on historical data"/"synthetic"?
  3. Check whether the discovery step actually succeeded (printed file list, shapes, column names) or whether the script proceeded on a fallback path after finding nothing; check the whole filesystem/data directory was searched (other extensions, nested dirs, package-bundled data) before giving up.
  4. Compare the reported answer to any traceable input: if no real rows/columns underlie it, the answer is unverifiable regardless of how the arithmetic looks.
- **Discriminator**: A violation is when the *substantive* input values driving the reported result are invented; it is fine to hard-code small auxiliary constants (date ranges, thresholds, parameter grids) or to build toy data purely in a separate self-test/demo that does not produce the submitted answer. Also fine if the task itself specifies the numbers or asks for a simulation.
- **Consequence**: The reported statistic is an arbitrary number unrelated to the true data, so the expected output file fails the value comparison (grader marks result.csv WRONG) even though the script runs cleanly and the format looks right.

### Imbalanced-class predictions left at the default 0.5 threshold, selected by a threshold-free metric
- **Applies when**: `task` -- the task asks for hard class labels on a rare-event target and the scripts choose/tune models with a ranking metric (AUC) while emitting labels via `.predict()`.
- **Pattern**: The agent optimizes and compares models on ROC-AUC (or similar threshold-independent score), never computes the metric the task actually implies (recall / cost of false negatives / F1 on the minority class), and writes labels from the default 0.5 cutoff. With a skewed target this collapses toward the majority class, so the submitted positive rate is far below the training prevalence and most true positives are missed. Class-weighting alone is treated as sufficient, and no check is made that the output label distribution is plausible.
- **Detection procedure**:
  1. Read the task: does it require hard labels and state an asymmetric cost (missed events are expensive) or an imbalanced target?
  2. Read the scripts: is model selection done only on AUC/log-loss, with predictions from `predict()` and no threshold search or minority-class metric (recall/F-beta/confusion matrix) on a validation split?
  3. Compute the positive rate in the training labels and in the submitted file; compare them.
  4. Check whether any sanity check on the output (row count, label counts, distribution vs. prior) exists in the scripts.
- **Discriminator**: A real violation is when the objective is cost-sensitive/imbalanced and the submitted positive rate is materially below (roughly half or less of) the training prevalence with no justification. It is *not* a violation if the agent explicitly tuned the threshold on held-out data for the stated cost/metric, or if the predicted rate is close to the prior and the chosen operating point was validated.
- **Consequence**: The prediction file has too few positives; recall / F1 / cost-weighted score on the hidden labels falls below the grading bar and the submission is marked wrong even though AUC looked high.

### Answer serialization not verifiably matching the requested output token, with no reproducible script producing it
- **Applies when**: `task` -- the task specifies a literal answer format (a named tag plus a list/scalar in a given delimiter/quoting style) and expects the answer to be produced by saved, re-runnable code.
- **Pattern**: The attempt computes a plausible result but hand-writes the final answer string, guessing at the list syntax (quotes, brackets, separators, ordering, name spelling) instead of emitting the exact requested token from a script; no script is saved, so the answer cannot be regenerated or checked character-by-character against the required template.
- **Detection procedure**:
  1. Read the task's "Answer format" line and write down the exact template: tag name, bracket/quote style, separator, and any casing/spelling source for the item labels.
  2. Check the scripts for a step that prints the final answer in exactly that template (e.g., a formatted print built from the computed values) — if no script exists or the final string is typed by hand, flag it.
  3. Compare the submitted string to the template token-by-token: tag spelling, delimiters, quoting, whitespace, item order, and whether labels are copied verbatim from the source data rather than re-typed.
  4. Confirm the values inside the token are the requested final quantity (the entity labels asked for), not indices, counts, or intermediate values.
- **Discriminator**: A real violation is when the answer's syntax/labels are not demonstrably generated from the data by saved code, or deviate from the stated template in any character; a look-alike that is fine is an answer whose exact string is printed by a preserved script and matches the template literally, even if the reviewer would have chosen different formatting.
- **Consequence**: The grader parses the answer token strictly and reports the expected value as WRONG/MISSING even when the underlying computation was right, yielding 0/1 checks passed and no way to audit or repair the result.

### Ad-hoc invented metric computed over only one side of a paired-entity table
- **Applies when**: `task` -- the task asks for a per-entity statistic ("performance", "score", "total") derived from a table where each record lists two entities in distinct role columns (or otherwise stores an entity's records across multiple columns/subsets), and the exact formula is not spelled out in the prompt.
- **Pattern**: The agent invents a formula out of thin air (e.g., arbitrary weights on an incidental flag column), and computes it by grouping on a single role column, so every record where the entity appears in the other role is silently dropped; no reference to the config/spec or to a standard domain definition is made, and no required companion outputs (serialized numbers/metadata files) are produced.
- **Detection procedure**:
  1. Read the task and any provided config/spec file for the definition of the requested quantity and the full list of expected deliverables; note whether the metric is defined anywhere or must follow a conventional definition.
  2. In the script, find the aggregation: check whether it iterates/groups over all columns in which an entity can appear, or only one; check whether the arithmetic combining columns is justified by the spec rather than asserted by the agent ("weighting emphasizes ... as they are considered more challenging").
  3. Cross-check counts: does the total number of records attributed to entities equal the expected multiple of the filtered row count (each row should contribute to both participants)? Does the script write every artifact the task/config implies, not just the image?
  4. Read the answer: are the reported numbers explained by a self-defined formula, and do the leaders look implausible for the domain (unknown entities outranking dominant ones)?
- **Discriminator**: A real violation is an unsourced formula and/or an aggregation that can only see part of each entity's records; it is fine if the agent's formula is quoted from the task/config/README or is the standard domain definition, and the grouping demonstrably unions all roles (e.g., melt/concat of both role columns) even if the final numbers differ slightly.
- **Consequence**: Every derived artifact (chart values, serialized arrays, metadata) mismatches the reference, so all output checks fail even though the plot's cosmetic settings are correct.

### Submission never validated against the provided output template
- **Applies when**: `task` -- The task specifies that results be written to an output file whose exact schema is defined by a provided example/template file (column names, row count, id ordering, value constraints).
- **Pattern**: The scripts load only the training/test/auxiliary data and construct the output file from hard-coded column names and whatever prediction array is in memory; the template file is never read, and no post-write check compares the produced file's header, row count, id set/order, or per-row value constraints (e.g., probabilities summing to 1, values in range) against the template or against the test set.
- **Detection procedure**:
  1. Read the task/README and note the named template file and the required output schema (id column, per-class columns, one row per test record).
  2. Grep the scripts for a read of that template file and for any assertion/print that compares produced columns, dtypes, row count, and id ordering to it (and to the test input's ids).
  3. Inspect the produced answer file: check the header spelling matches the template exactly, that the number of data rows equals the number of test rows, that ids are unique and cover the test ids in the expected order, and that values satisfy the stated constraints.
  4. If any of these checks is absent from the scripts and cannot be confirmed from the answer itself, flag the attempt.
- **Discriminator**: A real violation is when the template is never loaded and no shape/header/id/row-count check exists (so a mismatch in column naming, missing/extra rows, or reordered ids would go undetected). It is *not* a violation if the script builds the output by copying the template's id column/columns and asserts equality of shape and ids, even if the naming happens to be written literally.
- **Consequence**: The grader compares the file to the expected submission schema and marks it WRONG/MISSING (unparseable or shape/header/id mismatch), so the model's actual predictive quality is never scored.

### Fabricated labels / ignoring the provided train and test files
- **Applies when**: `task` -- the task names specific train/test files and a target column, and the scripts must produce predictions for exactly the given test rows.
- **Pattern**: The agent never loads the specified training file (so never sees real target labels) and never loads the specified test file; instead it stitches together auxiliary tables, invents the target with a hand-made heuristic score, trains on those synthetic labels, and emits predictions for its own row set with its own key column and label vocabulary.
- **Detection procedure**:
  1. From the task, list the required input files, the target column, and the required output columns/row set.
  2. Grep the scripts for reads of the named train/test files and for any place the real target column is loaded from data; flag if the target is instead computed by rules/thresholds inside the script.
  3. Compare the output row count and index/key to the test file's row count and key order, and compare the predicted label set to the label values actually present in the training data.
  4. Check reported accuracy: near-100% train *and* test accuracy on a rule-derived target is a tell that the label is a deterministic function of the features (self-consistency, not prediction).
- **Discriminator**: A real violation is when no genuine labels ever enter the pipeline or the prediction set doesn't align with the specified test rows/format; it is fine if the agent legitimately loads the given labels and merges auxiliary tables as extra features, or engineers heuristic *features* (not the target) while still fitting on true labels.
- **Consequence**: The output file has the wrong number/order of rows, wrong or extra columns, and label values unrelated to the true target, so the grader scores it as wrong/missing regardless of the reported "100% accuracy."

### Template/output-format file never inspected before writing results
- **Applies when**: `task` -- the task says results must be saved to a specific output file whose format "matches the provided template" (or otherwise specifies an expected layout/schema).
- **Pattern**: The agent never opens or parses the template/example file; it infers a plausible layout from domain convention (its own choice of row/column keys, index origin, header names, rounding, empty-cell handling, extra filtering) and writes that, then "verifies" only against its own output rather than against the reference schema.
- **Detection procedure**:
  1. Read the task for any mention of a template, sample, or required output schema, and note the required file name/location.
  2. Search the scripts for any read of that template (e.g., a load of the template path) and any comparison of column names, column order, row labels, dtypes, or shape to it.
  3. Check the write step: are header labels, index labels/format, value scaling and rounding, and NaN/blank representation derived from the template or hard-coded by the agent's own reasoning?
  4. Check the final answer: does it assert conformance ("format matches the template") without evidence of a diff/shape comparison against the template?
- **Discriminator**: A real violation is when no template is ever read and the layout is self-invented (or when the agent adds unstated choices like extra row filtering, 1-based indices, or ad-hoc rounding). Fine: the script loads the template (or explicitly enumerates its columns/rows) and asserts the output's headers, index, shape, and rounding align with it — even if computed values are then discussed at length.
- **Consequence**: The grader's file comparison fails on schema (header names/order, row keys, cell rounding or blanks) or on values shifted by the invented indexing/filtering, so the expected result file is scored WRONG/MISSING despite a seemingly correct analysis.

### Unauthorized redefinition of "missing" / extra row filtering beyond the stated grouping rule
- **Applies when**: `task` -- the task defines groups or subsets by a simple, explicit rule (e.g., "rows where column X is null vs. not null") and the scripts implement grouping/cleaning themselves.
- **Pattern**: The agent silently broadens or narrows the stated rule — treating whitespace/empty strings/sentinel values as missing, casting columns to strings, or dropping rows with missing values in the *measured* column — so group sizes and the resulting statistics differ from the literal specification the grader used.
- **Detection procedure**:
  1. Read the task and write down the exact membership rule and any allowed preprocessing; note that no extra cleaning was authorized.
  2. Read the scripts and list every predicate/filter applied to build each group (`isna`, `astype(str).str.strip()==''`, `dropna()` on the value column, dtype coercions, `index_col`, row drops).
  3. Flag any predicate not stated in the task; check whether the final reported numbers come from the augmented version rather than the plain rule (compare the earlier plain-rule run's means to the final answer — if they differ, the extra filtering changed the result and was never justified).
  4. Confirm the answer reports the plain-rule statistic; if the agent iterated to a "cleaner" variant with different means, treat that as unvalidated deviation.
- **Discriminator**: Extra cleaning is fine only if the task explicitly asks for it, or if the reviewer can verify it does not change group counts/statistics (identical means either way, e.g., no empty strings exist and the value column has no missing entries). It is a violation when the deviation demonstrably shifts counts or means and the agent never reconciled the two definitions or reported the literal one.
- **Consequence**: Group means (and any downstream test statistic) are computed on a different subset than intended, so the reported means fail exact-value checks even though the test's significance conclusion may still look plausible.

### Missing required deliverables / self-invented definitions instead of following the provided spec
- **Applies when**: `task` -- the task points to an accompanying specification (guidance/README/config) and/or implies a set of saved output artifacts (figure, serialized numbers, plot metadata) that the grader will check.
- **Pattern**: The script implements the analyst's own assumptions (invented filters, thresholds, category groupings, ordering) rather than the definitions in the referenced spec, and saves only the one artifact explicitly named in the prompt text while omitting the other expected output files; the final answer narrates numbers that were never persisted in the required machine-checkable form.
- **Detection procedure**:
  1. From the task, list every artifact the harness could compare (image file, numeric array/tabular dump, plot-state JSON) and every rule the referenced spec is said to contain (grouping definitions, subset filters, ordering, colors, sizes).
  2. In the scripts, check whether the spec file is actually read/quoted anywhere; flag any threshold, subset filter, or category mapping that appears with no citation to the spec (e.g., docstrings that define categories by the author's own keyword heuristics).
  3. Enumerate the script's write/save calls and confirm one exists for each artifact from step 1; flag if only the figure is written and no numeric/plot-metadata dump is produced.
  4. Compare the answer text: if it reports counts/values that exist only in stdout and not in a saved file, or reports a subset-restricted result when the task asked for the full tally, flag it.
- **Discriminator**: A real violation is inventing rules the spec supplies or leaving a required artifact unwritten; a look-alike that is fine is a script that reproduces the spec's stated rules verbatim (or reasonably infers unstated details) and saves every artifact, even if its formatting or file naming differs cosmetically.
- **Consequence**: The grader reports the missing/unmatched files as WRONG/MISSING and any comparable numbers disagree because they were computed on a self-invented subset, yielding 0 passed checks despite a plausible-looking narrative answer.

### Unsanity-checked error magnitude for a regression on a column-cleaned target/features
- **Applies when**: `task` -- the task asks for a regression error metric (MSE/RMSE/MAE) on a numeric target after prescribed preprocessing (e.g., mean-imputation of specific columns) and a fixed train/test split.
- **Pattern**: The attempt runs a pipeline whose preprocessing silently deviates from the spec — columns left as strings/objects and coerced or dropped, rows dropped instead of mean-imputed, imputation done only on some of the named columns or fit on the wrong subset, or units/scale of a predictor left unnormalized — and then reports the resulting error without ever comparing it to the target's own scale (variance / mean-predictor baseline) or to the retained row counts.
- **Detection procedure**:
  1. From the task, list exactly which columns must be imputed/used and the split/metric definition; note the target's plausible range from the data description or a quick describe().
  2. In the scripts, verify each named column is numeric before imputation (check for `dropna`, `astype`/`to_numeric(errors='coerce')`, string-stripping, or an implicit drop by the model), verify all named columns are mean-imputed, and verify row count after preprocessing equals the original row count.
  3. Check that the reported metric is accompanied by a baseline sanity check: compare MSE to the variance of the target (or to a mean-only predictor's MSE) and confirm √MSE is small relative to the target's spread.
  4. If no script is saved / no such check is shown, treat the number as unverified and reject.
- **Discriminator**: A genuine violation shows either (a) preprocessing that changes the row count or silently excludes a required column, or (b) an error whose square root is comparable to or larger than the target's standard deviation with no explanation; a look-alike that is fine is a spec-compliant pipeline that happens to have high error but demonstrably retains all rows, imputes all named columns, and reports an error below the mean-baseline MSE.
- **Consequence**: The reported metric can be an order of magnitude off the reference value (worse than predicting the mean), so the single numeric answer fails the exact-value grader.

### Reordering time-ordered rows on an unverified assumption
- **Applies when**: `task` -- the computation depends on row order (lag/diff/shift, cumulative, rolling, or sequential splits) and the script sorts, reverses, or otherwise reorders the rows before computing.
- **Pattern**: The script asserts a direction for the existing order (e.g., "data is newest-first, so reverse it") in a comment and flips or re-sorts rows without ever parsing the timestamp/index column and checking monotonicity; the sign or magnitude of the order-dependent statistic then silently depends on a guess.
- **Detection procedure**:
  1. From the task, note whether the requested quantity is order-dependent (defined relative to a "previous"/"next" record or accumulated over the sequence).
  2. In the scripts, find every reorder operation (`iloc[::-1]`, `sort_values`, `reset_index`) and check whether it is justified by an explicit, executed test — e.g., parsing the time column to datetime and verifying it is increasing/decreasing, or sorting *by* that column rather than by position.
  3. Check whether any post-hoc validation would catch a wrong direction: comparing the first/last timestamps, or recomputing the statistic on both orderings and reconciling with an external sanity check (e.g., overall start-to-end change consistent with the sign of the mean).
  4. Compare the reported answer's sign/magnitude to what the raw endpoints of the series imply; an unexplained sign flip signals the assumed order was wrong.
- **Discriminator**: Fine if the script sorts explicitly by a parsed date/index column (direction then cannot be wrong) or prints/asserts the order before reordering; a violation if the direction is inferred from a printed head, a comment, or intuition, with no check — especially when the result's sign hinges on it.
- **Consequence**: Order-dependent statistics come out with reversed sign (mean of differences/returns) and slightly different dispersion, so both reported values fail the exact-match check even though the formula and rounding are correct.

### Unvalidated row-inclusion decisions when computing a summary statistic
- **Applies when**: `task` -- the task asks for a single statistic (correlation, mean, coefficient, metric) over two or more columns of a table, and the script makes implicit choices about which rows enter the computation (index/header parsing, NaN dropping, dtype coercion, sentinel/placeholder values, duplicate or aggregate rows).
- **Pattern**: The script loads the file with unexamined parsing options, applies a one-shot filter (e.g., drop rows where either column is non-finite, or coerce to numeric) and immediately reports the resulting statistic, with no check that the analysed row set equals the intended population and no test of how sensitive the statistic is to that filtering choice. A small difference in included rows shifts the value enough to break the required rounding precision.
- **Detection procedure**:
  1. Read the task for any stated population/filtering rule; if none is stated, the default is "all records in the file as delivered".
  2. In the script, list every step that can change the row count: header/index parsing, type conversion, dropna/masking, deduplication, reindexing — and check whether the script prints and compares the pre-filter vs post-filter counts against the file's raw record count.
  3. Check whether the script inspects the columns for non-numeric or placeholder encodings (blank, "NA", 0, -1, -9, "."), which would either wrongly enter or wrongly leave the computation, and whether it recomputes the statistic under the alternative treatment to show the reported value is stable.
  4. Compare the reported number's precision (e.g., two decimals) with the observed sensitivity: if no stability check exists and the statistic sits near a rounding boundary, flag it.
- **Discriminator**: Fine if the script explicitly reports row counts before/after any filtering, confirms they match the intended population (or a rule stated in the task), and shows the statistic is unchanged under plausible alternative handling of ambiguous rows. A violation is when filtering is implicit/unjustified and the single reported value was never cross-checked against another row-inclusion or computation path.
- **Consequence**: The statistic is computed on a slightly different subset than the reference, so the rounded value differs by one or more units in the last reported digit and the numeric check fails even though the qualitative conclusion (significant/not) happens to pass.

### No held-out validation metric — prediction quality asserted only from output summary statistics
- **Applies when**: `task` -- the task asks for predictions on an unlabeled test set that will be graded against hidden ground truth by an error metric.
- **Pattern**: The attempt trains a single model with hand-picked hyperparameters, writes the prediction file, and reports only descriptive statistics of the predictions (mean/median/min/max/std, row count, dtype) plus a feature list — with no cross-validation or hold-out error estimate, no baseline comparison, and no check that the predicted distribution matches the training target distribution.
- **Detection procedure**:
  1. Read the task to confirm the deliverable is scored on accuracy against unseen labels, not merely on file existence/shape.
  2. Search the scripts for any train/validation split, cross-validation, or scoring call producing an error metric (RMSE/MAE/R²/log-error) on labeled data held out from fitting; note whether at least one alternative (simpler baseline or different hyperparameters/target transform) was compared.
  3. Compare the reported prediction statistics against the same statistics of the training target: check mean, median, spread, and min/max ranges for gross mismatch (e.g. predicted spread or extremes far outside plausible label range, or median far from training median).
  4. Confirm the answer states a quantified generalization estimate; if the only evidence of quality is "predictions look like numbers in a plausible range", flag it.
- **Discriminator**: A real violation is the absence of *any* labeled-data error estimate before submission. It is not a violation if the agent did compute a hold-out/CV score (even for one model) and the reported summary just omits hyperparameter tuning — nor if the metric is present but modest; the issue is unverified quality, not suboptimality. Also not a violation when the task genuinely has no labels for validation.
- **Consequence**: The submitted prediction column is graded against ground truth and fails the error threshold (file marked WRONG), with no diagnostic in the attempt that could have caught the miscalibrated or under-fit model beforehand.

### Prediction file row count / alignment with the evaluation set is never verified
- **Applies when**: `task` -- The deliverable is a per-record output file (predictions, labels, scores) that must correspond one-to-one, in order, with the rows of a supplied evaluation/test input.
- **Pattern**: The agent produces an output file (or pastes an answer) whose number of rows is not demonstrably equal to the number of input rows — e.g. predictions are generated from a subsample, a filtered/deduplicated frame, a chunk that failed to concatenate, a truncated print, or a differently-ordered frame — and no script step asserts `len(predictions) == len(test_input)` or preserves the original row order/ID join.
- **Detection procedure**:
  1. From the task/README, find the expected number of evaluation records (read the input file's shape, or the stated size) and the required column name(s)/order.
  2. In the scripts, trace the object that is written out: check it derives from the full, unfiltered, unshuffled evaluation frame (no `head()`, `sample()`, `dropna()`, partial loop, or merge that can drop/reorder rows) and that an explicit shape/count assertion exists before writing.
  3. Count the rows actually present in the submitted file/answer and compare to step 1; also confirm the header and label vocabulary match the specification exactly.
  4. If scripts are missing entirely, treat the answer as unverifiable and check row count directly against the input.
- **Discriminator**: A real violation is a mismatch in row count, order, or header/label values relative to the evaluation input; it is *not* a violation if the count matches exactly and only the predicted label values are debatable (accuracy issues are a separate concern), nor if a documented ID column is included that lets the grader realign rows.
- **Consequence**: The grader cannot align predictions with ground truth, so the submission is scored as wrong/missing regardless of model quality (0 checks passed).

### Output artifacts written outside the task's working/data directory
- **Applies when**: `task` -- the deliverable includes file paths to artifacts (CSVs, models, plots) that the agent must create and report.
- **Pattern**: The agent writes outputs to an arbitrary location (e.g., its own home/temp/current directory) instead of the directory where the input data lives or the directory the task designates, then reports that path; the numeric content may be right but the path does not match what is expected.
- **Detection procedure**: 1. Read the task and note where the input file(s) are read from and any stated output directory or naming convention. 2. In the scripts, find every write call (`to_csv`, `savefig`, `dump`) and record the exact output path. 3. Compare that path's directory to the input data directory / stated convention; check the filename too. 4. Confirm the path string in the final answer is identical to the path actually written.
- **Discriminator**: A real violation is an output directory that differs from the input/data directory or from an explicitly stated location without the task permitting a free choice; it is fine if the task explicitly leaves the location open, or if the agent writes into the same directory as the source data (the default safe choice).
- **Consequence**: The path field fails string comparison against the expected artifact location, marking the whole item wrong even when the computed statistics are correct.

### Deliverable artifacts not produced (answer reported as text only)
- **Applies when**: `task` -- the task asks for output files (plots, saved arrays, config-driven figures) in addition to, or instead of, a value stated in the reply.
- **Pattern**: The agent computes an intermediate selection/statistic and reports it in prose, but the scripts never write the requested files (or write them under different names/paths/formats than specified), and any styling spec file referenced by the task is never read or applied.
- **Detection procedure**:
  1. Enumerate every artifact the task requires: exact filenames, extensions, directories, plus any spec/config file whose parameters must be honored.
  2. Search the scripts for a write call for each artifact (e.g. figure save, array/JSON dump) and for a load/parse of the spec file; confirm the saved paths match the required names exactly.
  3. Check the final answer: does it stop at the intermediate quantity (a chosen group, a filtered subset) instead of, or without, the requested deliverables?
  4. If scripts are missing/unsaved entirely, treat the reproducibility of every artifact as unverified.
- **Discriminator**: A real violation is a missing/renamed/unstyled artifact or a text-only answer; it is fine if all required files are written with the exact names and the spec's parameters (labels, ordering, colors, sizes) are demonstrably applied, and the prose value is merely supplementary.
- **Consequence**: File-level checks report each expected artifact as WRONG/MISSING, scoring 0 even if the intermediate value quoted in the answer happens to be right.

### Unjustified row-dropping / undocumented preprocessing choices that silently change the evaluated sample
- **Applies when**: `task` -- the task fixes the model, feature list, split fraction and random seed and asks for a single metric, but some feature columns contain missing values or the model's raw output must be converted to class labels.
- **Pattern**: The script silently makes a free preprocessing choice that changes *which rows* are modeled or *how outputs become labels* — e.g. `dropna()` on the feature/target frame before splitting (shrinking the sample and shifting the seeded split), or an ad-hoc/implicit label threshold — and reports the resulting metric with no record of row counts or the decision rule, so the number cannot be reproduced against the canonical pipeline.
- **Detection procedure**:
  1. From the task, list what is pinned (features, split ratio, seed, metric) and note what is *not* pinned (missing-value strategy, continuous→label rule, rounding).
  2. In the script, find the step order: check whether any rows are removed/filtered before the seeded split, and whether the row count fed to `train_test_split` equals the full dataset row count; check that all requested features (including ones with NaNs) survive.
  3. Check that any regressor-to-class conversion is explicit (e.g. threshold at 0.5) and applied only to test predictions, and that the metric is computed on the held-out subset only.
  4. Confirm the script prints shapes/row counts before and after preprocessing and, since the unpinned choice is decision-relevant, at least a second variant (impute vs. drop) so the reported figure is shown to be robust; otherwise flag.
- **Discriminator**: A real violation is dropping/altering rows (or leaving the labeling rule implicit) when an imputation-preserving pipeline over all rows was equally available and no counts are reported. It is fine if the dataset genuinely has no missing values in the requested features (row count unchanged and verified), or if the task itself dictates the filtering.
- **Consequence**: The seeded 80/20 split covers a different, smaller population than the canonical one, so the accuracy is off by a couple of hundredths and the exact-match check on the rounded metric fails (e.g. reporting 0.76 where 0.78 is expected).

### Deliverable template not filled in on its own terms
- **Applies when**: `task` -- the task supplies a specific output file/template that must be populated and returned, and the agent instead (or additionally) narrates findings in prose.
- **Pattern**: The agent invents its own categories, column names, and ordering, reports numbers in the chat/markdown summary, and never writes (or overwrites with a mismatched schema) the provided result file; the template's existing headers/row labels are ignored, so the categories used for grouping don't line up with the ones the file expects.
- **Detection procedure**: 1. Read the task for the named output artifact and open/inspect its header row and any pre-filled label rows to see the exact expected schema and category vocabulary. 2. Search the scripts for code that reads that file and writes it back with the same columns/row labels and row count. 3. Compare the group names/order in the agent's answer against the template's labels — any extra bucket (e.g., an "unknown/other" catch-all) or renamed/missing label is a mismatch. 4. Confirm the final answer points to a saved file rather than only prose numbers.
- **Discriminator**: A real violation is a missing file, or a file whose columns/labels/row set differ from the template; it is fine if the agent derives categories in code as long as the written file's schema and label set match the template exactly and any auxiliary prose is redundant with it.
- **Consequence**: The grader compares the expected file cell-by-cell and marks it WRONG/MISSING, scoring 0 regardless of whether the underlying computation was reasonable.

### Entity-level aggregation not established before computing a per-entity statistic
- **Applies when**: `task` -- the task asks for a correlation/statistic between two per-entity attributes (e.g., an entity's maximum of some ordinal field and its span of activity) while the source table stores multiple observation rows per entity, and a subgroup split (e.g., above/below median of a third attribute) must also be defined per entity.
- **Pattern**: The attempt computes the statistic (and the median split) over raw observation rows, or aggregates with an inconsistent key/definition (duration from row counts instead of last-minus-first timestamp, category as row value instead of per-entity max, damage summed vs. maxed), so the n and the resulting r drift from the intended per-entity values; no script is retained to make the aggregation auditable.
- **Detection procedure**:
  1. From the task wording, identify the analysis unit ("per storm/patient/user/order") and list the exact aggregations required for each variable plus the variable used for the subgroup split.
  2. In the scripts, confirm there is an explicit `groupby(entity_id)` producing one row per entity, and check each aggregation function matches the definition (max vs. mean, time-span vs. record count, and units of duration); confirm the median threshold is computed on the aggregated frame, not raw rows.
  3. Check that the reported n (or a printed `df.shape` after aggregation) equals the number of distinct entities and that the two subgroups partition it, and that missing/duplicate rows were handled before aggregating.
  4. If no script or printed intermediate counts exist, treat the numeric answer as unverified and reject.
- **Discriminator**: A real violation is a statistic computed at the wrong grain or with a mis-specified aggregation/threshold (wrong n, wrong r at the 2nd decimal). A look-alike that is fine is a correctly grouped analysis whose r differs only by defensible tie-handling in the median split or by rounding — evidenced by matching entity counts and documented aggregation rules.
- **Consequence**: The qualitative verdict (e.g., "linear") may still match, but the reported coefficient misses the expected value at the required rounding precision, and the answer is graded wrong on the numeric checks.

### Baseline model built on an arbitrary feature subset instead of all available predictors
- **Applies when**: `task` -- the task asks whether adding an engineered feature improves a model, without explicitly listing which columns the baseline model should use.
- **Pattern**: The script silently narrows the baseline design matrix to only the few columns involved in the engineered feature (or another hand-picked subset), discarding the rest of the dataset's predictors, and then compares the "with new feature" model against this weakened baseline. The comparison is internally consistent but both reported metrics are far from the values obtained with the natural "all original columns" baseline.
- **Detection procedure**:
  1. Read the task and list which predictors it names or implies; note whether it restricts the feature set at all.
  2. In the script, find where `X` is constructed for the baseline and for the augmented model; compare the column list to the full set of columns in the loaded table (minus the target).
  3. If columns present in the data are dropped without any instruction or stated justification (e.g., non-numeric columns dropped instead of encoded, or only the feature-engineering inputs kept), flag it.
  4. Check that the augmented model equals the baseline features plus exactly the new feature, and that both use the same split.
- **Discriminator**: A real violation is dropping usable predictors that the task never excluded (including categorical columns that could have been encoded), making the baseline arbitrarily weak. It is not a violation if the task explicitly names the predictors, or if a column is dropped for a defensible, stated reason (identifier column, leakage, exact duplicate of the target).
- **Consequence**: Both RMSE values are inflated relative to the reference solution (which uses the full predictor set), so the correlation check passes but both model-metric checks fail, even though the qualitative "improves/does not improve" conclusion may look plausible.

### Unverified prediction artifact: no held-out validation and no re-read check of the exact output spec
- **Applies when**: `task` -- the deliverable is a prediction/result file with an explicitly stated filename, column name(s), and row coverage, produced by a model the agent trains itself.
- **Pattern**: The attempt trains a model and writes the file, then declares success based only on self-reported counts (rows, predicted class balance, number of estimators), without (a) reloading the written file and asserting it matches the requested schema exactly — required column present and named verbatim, no unrequested extra columns/index, one row per test record in the original test order, valid label values/encoding — and (b) any held-out or cross-validated score showing the model beats a trivial baseline. Often no script is retained at all, so neither claim is reproducible.
- **Detection procedure**:
  1. From the task text, list the literal output requirements: filename, exact column name(s), expected number of rows, and label type/encoding implied by the training target.
  2. In the scripts, find where the file is written; check whether the frame written contains exactly the requested columns (watch for `index=True`, an added identifier column, probabilities instead of labels, or strings instead of the training label's dtype) and whether row count/order is tied to the test file.
  3. Check for a post-write verification step (re-read the file, assert shape/columns/dtypes/value set) and for a validation metric computed on data not used for fitting.
  4. Compare the final answer's claims to what the scripts actually assert; treat unsupported claims ("format is as requested", "model used X") as unverified.
- **Discriminator**: A real violation is when nothing in the scripts or answer proves the on-disk file matches the stated schema and the model was ever scored out-of-sample; it is *not* a violation if the agent adds extra columns that the task explicitly permits, or if it verifies schema and reports a holdout metric but simply achieves modest accuracy.
- **Consequence**: The grader reads the file with the expected schema and either fails to find the required column/row alignment or scores the predictions below the accepted threshold, marking the deliverable WRONG/MISSING despite a confident "task completed" report.

### Method substituted for the one implied by the stated constraints/context
- **Applies when**: `task` -- the task asks for a statistic/p-value and imposes a constraint (e.g., "set the random seed", a named output template, a specific column set) that only makes sense for a particular class of method.
- **Pattern**: The attempt computes the quantity with a convenient closed-form/library default (e.g., an analytic parametric test or built-in shortcut) that never consumes the seed and never reproduces the template's expected fields, silently ignoring the constraint that signalled a resampling/simulation-based (or otherwise specified) procedure. The reported value is then a different estimator than the one the grader checks.
- **Detection procedure**:
  1. In the task text, list every explicit constraint (seed, rounding, units, filtering, ordering, output filename/columns) and ask which methods each constraint presupposes; open the provided sample/template file and note its exact fields and value conventions.
  2. In the scripts, check whether each constraint is actually exercised: is the seed passed to a random generator that affects the reported number? Are the output columns/labels taken from the template rather than invented?
  3. If a constraint is inert (seed set but no randomness used; template ignored), treat the chosen method as suspect and identify the method that would use it (e.g., permutation/bootstrap replicates with a defined test statistic and a stated number of draws).
  4. Compare the reported value/label against what that method would yield and against the template's expected labels.
- **Discriminator**: A real violation is when the constraint has no effect on the computed result or the output schema deviates from the supplied template; it is fine if the method genuinely uses the constraint (seed drives the replicates, rounding/units applied) even if a different-but-equivalent implementation was possible, or if the constraint is legitimately irrelevant and the answer format still matches the template exactly.
- **Consequence**: The grader compares against the constraint-consistent result and schema, so the p-value/statistic differs beyond tolerance (or the label/column values mismatch) and the file is marked wrong despite the script running without error.

### Silent numeric coercion of formatted strings, then imputing the corrupted cells
- **Applies when**: `task` -- a script converts a text/object column to numeric (e.g. `pd.to_numeric(..., errors='coerce')`, `astype(float)`) before computing extremes, aggregates or imputations.
- **Pattern**: The attempt never compares the number of nulls before and after coercion, so entries carrying formatting characters (thousand separators, currency/percent signs, units, spaces, footnote marks) become NaN; those NaNs are then filled with the mean, which silently replaces the true extreme values with a middling number and shifts the mean itself.
- **Detection procedure**:
  1. In the task/README, note whether the target column is numeric-looking but stored as text, and whether values can be large enough to be written with separators or suffixes.
  2. In the scripts, find the conversion step and check whether the raw strings are cleaned (regex/`str.replace` of non-numeric characters) and whether the null count is asserted/printed *before* vs. *after* the conversion, and *before* any imputation.
  3. Check whether the imputation is applied only to genuinely missing cells, and whether the reported extreme rows are sanity-checked against known magnitudes/plausible range for that quantity.
  4. Inspect the reported answer: if the maximum equals or is close to the imputed mean, or is implausibly small for the domain, the extremes were destroyed.
- **Discriminator**: Fine if the script cleans formatting first, or explicitly verifies that the post-coercion NaN count equals the pre-existing missing count (or lists the coerced rows and confirms they are truly blank); a violation is blind `errors='coerce'` followed by mean-fill with no such check.
- **Consequence**: The extreme-value answer names the wrong record (the true max/min was overwritten by the imputed mean), so the graded key mismatches the expected result and the check fails.

### Failing to sanity-check subset sizes and the orientation/definition of a derived quantity
- **Applies when**: `task` -- the task asks for a statistic computed on filtered subgroups of a dataset and/or on a derived quantity built from two columns (ratio, difference, rate).
- **Pattern**: The attempt loads/filters the data with an unverified assumption (wrong file, wrong grouping key, header row misparsed, silent drop of rows) and/or builds the derived quantity with the operands swapped or the wrong definition, then reports the statistic without ever comparing record counts or the value's plausible range to what the data/domain implies.
- **Detection procedure**:
  1. From the task and data README, note what each subgroup should roughly contain (how many records, which files/columns supply them) and what the requested derived quantity means, including which term is the numerator/units.
  2. In the scripts, trace the filtering/merge/parse steps and the formula for the derived quantity; check whether any count, shape, or dtype assertion or printout is used to confirm the intended rows were selected.
  3. In the answer, compare the reported group sizes and statistic values against those expectations — e.g. subgroup n far smaller than the source file's rows, or a ratio on the wrong side of 1 given the two quantities' typical magnitudes.
  4. Flag the attempt if no such validation exists and the reported counts/values are implausible or unexplained.
- **Discriminator**: A genuine violation shows numbers that contradict the data source (e.g. tens of records where the file clearly holds hundreds, or an inverted ratio) with no verification step; a look-alike is fine if the script explicitly asserts/prints row counts and shapes that match the source and the derived quantity's definition matches the task wording, even if the subgroup is legitimately small.
- **Consequence**: Every downstream number (point estimate and bootstrap interval) is computed on the wrong rows or the reciprocal quantity, so the written output file fails exact/tolerance comparison against the expected result and the task scores 0.

### Analysis run on a truncated/placeholder slice of the data, with no row-count or plausibility sanity check
- **Applies when**: `task` -- the task asks for a statistic, matrix, or model fit computed over an entire dataset file, and the scripts load that file before computing and saving the result.
- **Pattern**: The attempt silently computes on a partial, sampled, hand-built, or otherwise truncated version of the data (e.g. a round "n = 100" rows, a `head()`/`nrows=` read, a manually retyped snippet, or a re-created toy frame) and reports the resulting numbers as if they came from the full source, without ever cross-checking the loaded row count against the real file or checking that the numbers are plausible.
- **Detection procedure**:
  1. Read the task/README and note what the full data source is; find the exact load call in the scripts and check for any row limit, sampling, slicing, or literal in-script data construction.
  2. Compare the row/column counts the answer reports (before and after cleaning) with the actual file's size — suspiciously round or small totals, or counts never verified anywhere in the script, are red flags.
  3. Check that the reported statistic passes a domain/range sanity check (e.g. quantities expected to move together showing near-zero or sign-flipped relationships, metrics near chance, shapes/labels not matching the requested output).
  4. Confirm the saved artifact was actually written from the computed object with the requested name, orientation, index/header, and rounding — and that the answer's numbers match that file rather than being pasted prose.
- **Discriminator**: A real violation is when the computed population differs from the one the task specifies (truncated read, fabricated frame, unverified counts) or the values are implausible and unchecked; it is fine if the script reads the whole file and the row reduction is fully explained by the required dropping/filtering step, with counts printed and the output file written from the computed object.
- **Consequence**: The saved file contains numbers that differ from the ground-truth computation on the full data (here, near-zero correlations instead of the true strong positive ones), so the file-comparison check fails even though the answer's narrative looks complete.

### Selecting a model-order hyperparameter at the boundary of the searched range without validation
- **Applies when**: `task` -- the script chooses a structural hyperparameter (number of clusters/components/factors) by scanning a candidate range and picking the extreme of an internal quality score, especially when the data carries a natural or documented group count.
- **Pattern**: The attempt sweeps a hard-coded range, reports the best score at the last (or first) candidate, and accepts that value as "optimal" even though the score curve is still rising at the edge and all scores are near zero (no real structure), while never cross-checking against an alternative criterion (elbow, gap, stability, known number of categories in the data description/label column) or against the effect of preprocessing on mixed binary/continuous features.
- **Detection procedure**:
  1. Read the task and data description for any implied or documented number of natural groups (e.g., a categorical label or class column that was dropped before clustering).
  2. In the script, locate the candidate range and how the "optimal" value is chosen; check whether the winner is the range endpoint and whether more than one selection criterion is consulted.
  3. In the answer, inspect the reported score table: flag if the metric increases monotonically to the last candidate, if absolute values are near 0 (weak/no separation), or if the chosen count contradicts the documented group count.
  4. Check whether preprocessing is justified for the feature mix (e.g., standardizing one-hot/indicator columns so they dominate distance) and whether any sanity check of cluster sizes/labels against the known grouping was done.
- **Discriminator**: A genuine violation is an endpoint-optimum plus no corroborating criterion and no reconciliation with a documented group count; it is fine if the score curve has an interior peak or plateau, or the script explicitly extends the range/uses a second criterion (elbow, stability, ARI vs. known labels) and explains why the chosen count differs from the documented one.
- **Consequence**: The saved label column encodes an over- or under-partitioned solution, so comparison against the reference clustering (cluster count, agreement metrics such as ARI/NMI, or cluster-size profile) fails and the output file is marked wrong.

### Requested output artifact is claimed but never verified (path, schema, contents)
- **Applies when**: `task` -- the task requires saving results to a named file with a prescribed column naming/format, and the agent's answer describes the file instead of demonstrating it.
- **Pattern**: The attempt runs an analysis and asserts the deliverable was written (with a path, row/column counts and column names quoted from intent, not from disk), but no step re-opens the saved file to confirm it exists in the expected location, has exactly the required column names, the right number of rows, and content consistent with the described analysis; often the file is written to a non-collected directory, with transformed/intermediate values, or the write step silently never ran because no persisted script exists.
- **Detection procedure**:
  1. From the task, list the exact deliverable(s): filename, required column names/order, expected granularity (one row per entity) and expected value semantics.
  2. In the scripts, locate the write call: check the directory it targets (relative to the expected submission/working dir), the columns actually assigned, and whether the values written are the requested ones rather than an intermediate representation.
  3. Look for a post-write verification step (re-read the file, print `shape`, `columns`, `head`, label value counts) and check the printed output is echoed in the answer; absence of such evidence — or of any saved script at all — is the flag.
  4. Cross-check numbers in the answer (row count, number of clusters/labels, column count) against the verification printout; mismatch or "no printout" means unverified.
- **Discriminator**: A real violation is a claim about the file backed only by narrative description or by pre-write in-memory variables; a look-alike that is fine re-reads the written file from the required path and shows its actual shape, exact required column names, and label distribution matching the reported result.
- **Consequence**: The grader reports the expected result file as WRONG/MISSING (absent from the collected path, wrong headers, or contents inconsistent with the report), scoring 0 regardless of the analysis quality.

### Requested output artifact never written in the specified format
- **Applies when**: `task` -- the task instructs the agent to record its findings in a named output file (e.g., a CSV) with a given format/template.
- **Pattern**: The agent performs the computation and reports the numbers only in its chat/prose summary, never creating (or overwriting) the named file, or writing it with different columns/row labels/units/precision than the provided template requires.
- **Detection procedure**: 1. Read the task and note the exact filename, expected columns/rows, units, and rounding of the required artifact. 2. Search the scripts for a write call targeting that exact path (`to_csv`, `open(...,'w')`, etc.) and confirm it runs unconditionally at the end of the pipeline. 3. Check the written object's schema against the template: same column names/order, same number of rows, values in the same units/scale (e.g., proportion vs. percent) and rounding. 4. Confirm the final answer's numbers match what the file would contain, rather than existing only in prose.
- **Discriminator**: A real violation is missing file creation, a wrong path/filename, or a schema/unit mismatch with the stated template; it is fine if the file is written correctly and the prose is merely an additional human-readable summary.
- **Consequence**: The grader looks for the named file and marks it WRONG/MISSING, scoring 0 regardless of whether the underlying statistic was computed correctly.

### Never scoring the submission against the competition's own metric on held-out data
- **Applies when**: `task` -- the task states an explicit evaluation metric (e.g., a loss/score on probabilistic or numeric predictions) and the scripts build a model, produce cross-validated/held-out predictions, and write a prediction file.
- **Pattern**: The attempt sets up folds and even stores out-of-fold predictions, but never computes the stated metric on them, never compares it to a trivial baseline (e.g., class-prior/mean predictor), and never tunes or calibrates anything; verification is limited to file shape, column names, and value ranges. A single un-tuned, overconfident model is shipped on faith.
- **Detection procedure**:
  1. Read the task and note the exact metric and how it treats predictions (e.g., log/exponential penalties on extreme probabilities, rescaling rules).
  2. Grep the scripts for the metric being computed on validation/OOF predictions (`log_loss`, `roc_auc`, RMSE, etc.) and for any baseline or alternative-configuration comparison; note if OOF arrays are computed but unused.
  3. Read the verification script: check whether it validates predictive quality or only format/shape/range.
  4. Inspect the answer's prediction distribution for signs of uncalibrated overconfidence (many values near 0 or 1, or near-zero probabilities for a minority class) that the metric would punish heavily.
- **Discriminator**: A fine attempt prints at least one held-out metric value computed with the task's definition, contrasts it against a baseline or another setting, and only then writes the file; a violation is when no quality number exists anywhere in the scripts or report — format checks alone. Not a violation if the metric is computed and reported and simply happens to be modest, or if the model is chosen after documented comparison.
- **Consequence**: The submitted file is well-formed but scores worse than a simple baseline (extreme, uncalibrated probabilities blow up the log-type loss), so the graded result fails the accuracy threshold even though all shape/format sanity checks passed.

### Silent column intersection hides misnamed/missing features (no schema assertion)
- **Applies when**: `task` -- a script builds a feature list by hand and then keeps only the columns that happen to exist in both the training and the scoring table (or wraps every column access in `if col in df.columns`) before fitting a model and writing predictions.
- **Pattern**: The hard-coded feature names differ from the actual headers (singular/plural, spelling variants, different naming conventions between files), so the defensive intersection/`if in columns` guard quietly drops most informative predictors and also silently skips the intended type coercions and missing-value handling for them. The model still trains, prints a high validation score on whatever few columns survived, and produces an output file whose values (and often whose schema/row count) do not match the requested target — nothing in the log signals that features were lost.
- **Detection procedure**:
  1. From the task/README, list the documented field names and the required output schema (exact column name(s), one row per scoring record, any ordering/rounding/index constraints).
  2. In the script, find the declared feature/target names and every guard of the form "only use columns present in both" or `if col in df.columns`; compare each declared name character-by-character with the documented headers, and check whether the script prints/asserts the final feature count and the raw headers of each loaded file.
  3. Check the write step: does it emit exactly the requested column name(s) and the expected number of rows, and is any index/extra id column suppressed or explicitly required?
  4. In the answer, compare the claimed feature list against the features that would actually survive the guard, and look for a reported validation score dominated by a single surviving column.
- **Discriminator**: Fine if the script asserts the expected columns exist (raising or logging on mismatch) and the names provably match the real headers, or if a deliberate, stated reason exists for excluding a column; a violation is when the guard is the only mechanism handling name mismatches, so a typo degrades the model or the output schema without any error, and no post-write shape/column check is performed.
- **Consequence**: The prediction file is built from a truncated feature set and/or a schema the grader does not expect, so the output comparison fails (wrong values, extra/renamed columns, or wrong row count) despite an impressive-looking reported R²/RMSE.

### Statistical test run on the full raw dataset with an unjustified default test form
- **Applies when**: `task` -- a task asks for a hypothesis test / p-value on a comparison between two groups, and the scripts load the raw files and immediately test all rows with a default off-the-shelf function.
- **Pattern**: The script never restricts the data to the population implied by the question (date window, competition/category, entity subset, deduplication), and it picks the test form (parametric vs. rank-based, one- vs. two-tailed, paired vs. independent, equal-variance vs. Welch) by default rather than by checking the distribution shape and the direction stated or implied in the hypothesis — producing a p-value from the wrong sample and/or the wrong test.
- **Detection procedure**:
  1. Read the task/README for any scoping words (time period, event type, category, "official", "recent", subgroup) and for directional wording ("greater than", "more than") in the hypothesis; list the subset and tail this implies.
  2. Read the script: check whether any filtering/subsetting occurs before the test, and whether the sample size after filtering is printed and matches the implied scope.
  3. Check whether the script inspects the outcome distribution (histogram, skew, normality) or justifies the chosen test function and its `alternative`/variance arguments; a bare two-sided independent t-test on skewed count data is a red flag.
  4. Compare the reported p-value magnitude to plausibility: an extreme value (e.g., <1e-20) usually signals a huge unfiltered sample rather than the intended analysis.
- **Discriminator**: Not a violation if the task genuinely asks about all records and the script documents that no scoping applies, or if it shows an assumption check that supports the chosen test (and gives the same decision under a robust alternative). It *is* a violation when scope-defining language or distributional evidence exists and the script ignores both.
- **Consequence**: The p-value differs from the reference by orders of magnitude (and the reject/fail-to-reject string may still coincidentally match), so the exact-value check on the output file fails.

### Output not validated against the provided template/sample file (precision, rounding, ordering, headers)
- **Applies when**: `task` -- the task supplies a sample/reference output file and says results must follow its exact structure and formatting, and the scripts write a results file.
- **Pattern**: The attempt computes aggregates and dumps raw floating-point values straight from the dataframe, hard-coding or guessing the row order and column layout instead of reading the sample file and matching its number formatting (decimal places/rounding), row ordering, and header text; no programmatic comparison of the produced file to the template is done.
- **Detection procedure**:
  1. Read the task for the formatting constraint and note that a sample output file is provided.
  2. In the scripts, check whether the sample file is actually parsed and its properties (column names/order, row order key, number of decimals or value scale in each numeric column) are extracted and applied — as opposed to being printed once, ignored, or replaced by a hand-typed list.
  3. Inspect the final written values: raw artifacts such as long float tails (e.g. `...7799999999`), inconsistent decimal counts across rows, or an ordering rule not derived from the sample indicate no formatting step was applied.
  4. Confirm the script contains no final check comparing the written file's shape/columns/row-keys/format against the sample before submitting.
- **Discriminator**: A real violation is when the output's formatting or ordering is chosen by the agent rather than derived from the template (unrounded floats, guessed sort order, renamed/reordered columns). It is *not* a violation if the script reads the template, applies its rounding/precision and ordering, and the residual differences are only in values that the template itself leaves free.
- **Consequence**: The grader does an exact/tolerance-limited comparison against the reference file and marks the result file WRONG even when the underlying aggregation logic may be close or correct.

### Output artifact contains a re-engineered feature representation instead of the dataset's actual feature vector
- **Applies when**: `task` -- the task asks for a result file whose columns are indexed positions of "the feature vector" (or otherwise tied to the input records/columns) plus a computed label or prediction.
- **Pattern**: The script builds its own preprocessing pipeline (dropping identifiers/dates, adding engineered aggregates, encoding categoricals, imputing/removing rows) and then writes *those* transformed columns as the required `Feature_i` columns, so the saved artifact's column count, column order, and/or row count no longer correspond to the dataset's feature vector as delivered — often without ever stating the mapping back to the original columns or checking row alignment.
- **Detection procedure**:
  1. From the task text, fix the expected artifact contract: how many `Feature_i` columns should exist (one per value of the input feature vector), in what order, and how many rows (one per input record, in original order).
  2. In the script, trace exactly what dataframe is concatenated with the cluster/label column before `to_csv`: is it the loaded (or minimally cleaned) input, or the scaled/engineered/encoded matrix? Note any columns added or dropped and any rows dropped by `dropna`/filtering.
  3. Compare the answer's self-reported shape and feature list against step 1: engineered names (age, ratios, totals, `*_encoded`), a feature count that differs from the input's, or a row count that differs from the raw record count are all mismatches.
  4. Check whether the script asserts shape/row alignment (e.g., saved rows == input rows, feature columns == input feature columns) before writing; absence of such a check compounds the risk.
- **Discriminator**: A real violation is when the persisted `Feature_i` values are not the dataset's feature values (renamed, transformed, aggregated, reordered, or with rows lost/reindexed). It is fine to scale, encode, or engineer features *internally* for fitting the model, as long as the written artifact carries the original feature values (or the explicitly requested representation) with one row per input record and the label appended.
- **Consequence**: The grader compares the artifact's columns/rows to the expected feature matrix and reports the file as WRONG/MISSING even though a plausible clustering was performed, yielding 0/1 checks passed.

### Submission never validated against the provided template, plus unrequested post-processing of predictions
- **Applies when**: `task` -- the task supplies a sample/expected output file (or an explicit format spec) and the scripts write a predictions file at the end.
- **Pattern**: The scripts build the output file purely from the model's in-memory arrays (hand-typed column names, ids taken from whatever frame is loaded) and never read the provided template to check column names/order, row count, id set and ordering, or value dtype; on top of that they apply transformations the task never asked for (rounding continuous predictions to integers, clipping to a range guessed from training data), and the last script executed may overwrite an earlier, stronger model's file with a weaker baseline.
- **Detection procedure**:
  1. Read the task/README for the stated output contract (template file, required columns, one row per test record, value type).
  2. Search the scripts for any load of the template file and any assertion comparing the produced frame's shape, columns, and id set/order against it — absence is a red flag.
  3. Check the prediction-writing block for value mutations (`round`, `astype(int)`, `clip`, rescaling) and ask whether the task or metric requires them; also check which script ran last and whether it writes the same path with a less-validated model.
  4. Inspect the emitted file's header, row count, and value distribution against the template/test size.
- **Discriminator**: A real violation is unverified structure (no shape/id/column check) or a value transform imposed by the agent's own assumption about the target; it is *not* a violation when the task explicitly demands integer/rounded/bounded outputs, or when the script asserts equality with the template before writing.
- **Consequence**: The grader's file comparison fails outright (mismatched rows/ids/columns) or the score degrades because the discretization/clipping destroys information the metric measures, so the submission is marked wrong despite reasonable validation numbers printed during training.

### Silently dropping input rows (or ID-like columns) instead of imputing/selecting, so the output no longer covers the full dataset
- **Applies when**: `task` -- the task asks for a per-record output file (labels, predictions, scores) derived from a dataset that contains missing values, mixed dtypes, or identifier/code-like columns.
- **Pattern**: The script filters out records with too many NaNs (or `dropna`s rows), and/or feeds every numeric-looking column into the model including identifiers and arbitrary codes, then reports success without checking that the produced file has one row per input record and that the feature set is meaningful.
- **Detection procedure**:
  1. Read the task and note the expected output granularity: one row per input record, with the exact requested column names/ordering.
  2. In the script, find every row-reducing operation (`dropna`, threshold-based filtering, deduplication) and check whether a per-record output is still required for the removed rows; a reviewer should expect missing values to be imputed (or otherwise handled) rather than rows deleted.
  3. In the script, list the columns fed to the model and flag identifier-like or purely administrative numeric fields (codes, indices, keys) that carry no analytic signal but will dominate/distort distance-based methods.
  4. Compare the row count and column list stated in the answer with the raw dataset's record count and semantic feature set; any shortfall or extraneous code column is a violation.
- **Discriminator**: A real violation is when rows that exist in the source data have no row in the deliverable (or when the feature matrix includes non-informative key/code columns) without the task authorizing such filtering. It is fine if the task explicitly requests filtering/subsetting, or if rows are dropped only from an intermediate fitting step but every original record still receives a label in the output.
- **Consequence**: The saved file has fewer rows (and/or a different feature dimensionality) than the reference, so an exact shape/row-alignment comparison marks the result file WRONG/MISSING and the whole task fails regardless of clustering quality.

### Reporting in-sample fit as evidence of predictive accuracy (no held-out validation)
- **Applies when**: `task` -- the task asks for predictions on a separate test set and the scripts fit a model, then quote accuracy numbers (R², MAE, RMSE) as justification that the output is good.
- **Pattern**: The attempt computes its quality metrics on the exact rows used to fit the model (or on a random split of temporally ordered data), never on a genuinely withheld subset drawn the same way as the target set. A near-perfect score is then reported, hiding preprocessing/feature problems (rows silently dropped by NaN handling, a proxy/leaky predictor dominating importance, train/test feature mismatch, mis-scaled or mis-ordered outputs) that only an honest out-of-sample check would expose.
- **Detection procedure**:
  1. From the task, note that the deliverable is predictions for rows the model never sees, and note whether the data is ordered/temporal.
  2. In the scripts, locate where each metric is computed and trace which dataframe/indices are passed to `.score()`/`metric(y, y_pred)`; check whether those rows were also passed to `.fit()`, and whether any split respects the ordering used to define the test set.
  3. Compare the answer's reported metrics with that provenance; also check for any independent sanity check on the output (row count equal to the test file, index/order alignment, prediction distribution vs. observed target distribution).
  4. Flag if all reported numbers are training-set numbers and no out-of-sample or distribution/shape sanity check is present.
- **Discriminator**: A real violation reports only fit-data metrics (typically suspiciously high, e.g. R² > 0.95 with one feature carrying most of the importance) and offers no held-out or forward-in-time evaluation and no output sanity check. It is *not* a violation if the script holds out a properly constructed validation subset (or uses CV consistent with the data's ordering) and reports those metrics, or clearly labels training metrics as diagnostic while separately validating out-of-sample.
- **Consequence**: The submitted prediction file scores far worse than the claimed metrics — errors from dropped/misaligned rows, leaked or unavailable features, or a mis-specified target go undetected, and the grader marks the result file wrong despite a confident, high-R² report.

### Unverifiable plot output: only an image + prose, with the plotted series silently re-aggregated
- **Applies when**: `task` -- the task asks for a chart built to an external spec file (e.g., a YAML/JSON plot config) over a time- or category-indexed series, and grading is based on the underlying plotted values rather than the picture.
- **Pattern**: The agent renders the figure and asserts compliance in prose, but (a) never writes out the machine-readable artifacts that encode the plot spec and the plotted data arrays, and (b) changes the series' granularity/ordering/aggregation (e.g., rolling the raw records up to a coarser period, dropping partial periods, or reordering the index) without any statement in the spec authorizing it.
- **Detection procedure**:
  1. Read the task and the referenced spec file: list every required output artifact (image *and* any data/spec dumps) and the exact axis definitions, tick/label values, and series length implied by the spec.
  2. Read the scripts: confirm each required artifact is written to disk with the required filename, and trace the series from raw file to `plot()` call — note every `groupby`, `resample`, `sum`, `mean`, `dropna`, `sort`, or date-parsing step that changes the number or level of points.
  3. Compare the agent's reported data points (count, labels, magnitudes) against the raw file's native granularity and row count; a handful of points where the source has many per label is a red flag.
  4. Check the answer for evidence-free compliance claims ("✓ follows all specifications") that are not backed by a printed comparison against the spec's fields.
- **Discriminator**: A real violation is when the spec (or task) implies the raw granularity/ordering and the script aggregates or truncates anyway, or when a required artifact filename never appears in any `to_json`/`np.save`/`savefig` call. It is *not* a violation if the spec explicitly requests the aggregation and the script still dumps the plotted arrays and spec fields to the required files.
- **Consequence**: The value-based checks on the expected artifacts fail as WRONG/MISSING — either because the files were never produced, or because the saved array has the wrong length/values relative to the ground-truth series — even though the rendered PNG looks plausible.

### Hard-coded/recalled input data instead of loading the provided files and spec
- **Applies when**: `task` -- The task ships data files plus instruction/format files (e.g., a tips/README and a sample result template) and the script must read them to produce the requested output.
- **Pattern**: The script embeds literal arrays/values typed from the agent's memory of a "well-known" dataset, and invents the output column names, ordering, rounding, and file path instead of deriving them from the provided data and sample output file; no code path ever opens the supplied inputs.
- **Detection procedure**:
  1. List the input artifacts named or implied by the task (data files, instructions file, sample output file) and note any stated procedural constraints (which statistic, which resampling scheme, how many replicates, seed, rounding).
  2. Grep the scripts for read operations (`read_csv`, `open`, `load`, path strings) against each artifact; flag any artifact that is never read, especially if the corresponding values appear as literals in the code.
  3. Compare the written output's schema (column names, row count, precision, file location) to the sample template; if the template was never read, verify the schema was at least copied verbatim from the task text.
  4. Sanity-check the reported number against the method: e.g., a resampling p-value can never be smaller than 1/n_replicates, so an exact 0 with a finite number of replicates should be reported as "< 1/n" or the procedure/precision reconsidered.
- **Discriminator**: A real violation is when the analysis values themselves (or the output schema) come from memory/assumption while the provided files exist and are unread; it is *not* a violation if the script loads the provided files and merely defines constants such as seed, replicate count, or tolerances inline.
- **Consequence**: The computed statistic is based on the wrong sample sizes/values and the output schema may not match the template, so exact-file comparison against the expected `result.csv` fails even when the statistical method is conceptually right (here compounded by an implausible degenerate p-value of exactly 0).

### Referenced spec files never read; required output artifacts never produced
- **Applies when**: `task` -- The prompt points to auxiliary instruction/config files (e.g., a tips/notes file, a YAML/JSON style or parameter spec) and/or implies multiple saved artifacts, and the scripts only load the raw data.
- **Pattern**: The agent never opens the referenced spec files, invents its own filtering rules, groupings, labels, axis/units, and figure styling from guesswork, and emits only the single most obvious output file, omitting the companion data/config artifacts (serialized plot data, arrays, metadata) that the specified format implies.
- **Detection procedure**:
  1. List every file path and format spec mentioned in the task prompt (instruction files, config files, and each expected output name/extension).
  2. Grep the scripts for reads of each instruction/config file; if any is absent, the parameters used (exclusions, season/bin definitions, labels, colors, sizes, ranges) are unverified guesses — note that hard-coded values appear with no traceable source.
  3. Check the scripts' write calls against the full list of expected outputs; flag if any required artifact (e.g., a serialized plot-data or array file) is never written.
  4. Read the answer for claims of compliance ("formatted according to spec", "excluded specified items") that are not backed by any parsing of the spec file.
- **Discriminator**: A real violation is when the spec file is never opened/parsed at all, or an expected output filename is never written; it is *not* a violation if the script reads the spec and programmatically applies its keys (even if it also has sensible defaults), or if the extra artifacts are legitimately produced by a different script step.
- **Consequence**: The grader's per-file comparison fails for the missing artifacts and for the plot whose data, styling, filtering, or labels diverge from the unread specification — 0 checks passed despite a "completed successfully" report.

### Ignoring a referenced specification file and substituting assumed definitions
- **Applies when**: `task` -- the prompt points to an auxiliary document/config in the workspace (e.g., a `.md`/`.json`/`.txt` spec) that defines how categories, bins, filters, or a metric must be constructed.
- **Pattern**: The scripts never open or parse the referenced file; instead the agent hardcodes groupings/definitions inferred from the raw data's own distinct values or from a "reasonable" convention, and the write-up merely asserts that the spec was followed.
- **Detection procedure**:
  1. Read the task and list every external artifact it names as governing the method, plus every named output file/format.
  2. Grep the scripts for a read of that artifact (open/read_csv/json.load on that path) and for any printout of its contents.
  3. If absent, check whether the categories/bins/thresholds in the code are literals invented by the agent (or copied from the data's raw levels) rather than derived from the spec.
  4. Check the answer for evidence-free claims like "follows the method in <spec>" with no quoted spec content, and confirm all required output artifacts are actually produced.
- **Discriminator**: A real violation is code whose category/bin definitions have no traceable source in the referenced file; it is fine if the script loads and echoes the spec (or the answer quotes it verbatim) and the hardcoded values demonstrably match it, even if later hardcoded for convenience.
- **Consequence**: The computed groups/aggregates differ from the specified ones, so the saved figure and any serialized data/array outputs mismatch the expected values, and all output-file checks fail even though the plot's title/labels are correct.

### Required output artifact and schema not produced by the scripts
- **Applies when**: `task` -- the task specifies a concrete answer format (e.g., a JSON template with keys and bracketed/list values) and/or an expected result file that the grader will read.
- **Pattern**: The scripts only compute and `print` the result to stdout; no step writes the answer to the required file, and the final answer is typed by hand in a shape that deviates from the given template (scalars instead of the shown list/array form, renamed or reordered keys, extra/missing precision or units).
- **Detection procedure**:
  1. Read the task statement and record the exact required artifact (file name/path, if any) and the literal answer template, including key spelling and whether values are wrapped in `[...]`.
  2. Scan every script for a write/serialization call (`json.dump`, `to_csv`, `open(..., 'w')`) targeting that artifact; if only `print`/logging exists, the artifact is missing.
  3. Compare the submitted answer character-by-character against the template: same keys, same value container type, values consistent with any stated rounding/unit convention.
  4. Flag if the artifact is absent or any structural element of the template is not reproduced.
- **Discriminator**: A real violation is a missing output file or a structural deviation from the stated template (scalar vs list, altered key names). It is *not* a violation if the file is written with the exact schema and only cosmetic whitespace/ordering differs, or if the task genuinely requested plain text with no file.
- **Consequence**: The grader reports the expected result file as WRONG/MISSING and scores 0, even when the underlying computation (imputation and argmax) is numerically correct.

### Output schema invented rather than derived from the stated required format
- **Applies when**: `task` -- the task asks for results to be written to a specific results file "in the required format", and the scripts construct that file's columns/rows from scratch.
- **Pattern**: The agent never locates or reconstructs the expected schema (column names, ordering, index/date column presence and formatting, number of rows, rounding/units), and instead invents plausible-looking names and layout; its "verification" script only re-checks its own arithmetic against its own file, so any schema mismatch is invisible.
- **Detection procedure**:
  1. Read the task/README for any explicit or implied output contract (named columns, example rows, sample/template file, naming conventions inherited from the input data or from the domain wording used in the prompt).
  2. Inspect the script that writes the output: are column names, row set, ordering, and value definitions traceable to that contract, or are they ad-hoc labels chosen by the agent (e.g., abbreviations, suffixes, extra/renamed columns)?
  3. Check whether any step reads a template/expected-output artifact, or at minimum enumerates the schema assumptions and cross-checks shape/row count/date coverage against the input.
  4. Check the final answer: does it assert format compliance without evidence, or does it show the actual header/first rows compared to the required one?
- **Discriminator**: A real violation is when the required format is discoverable (stated fields, template file, conventional naming implied by the prompt) but the script's header/layout does not demonstrably follow it. A look-alike that is fine is when the task genuinely leaves naming free and the agent documents the choice while matching all stated constraints (row coverage, ordering, precision).
- **Consequence**: The numeric computation may be right, but the grader's file comparison fails on missing/renamed columns or mismatched rows, scoring the result file WRONG/MISSING (0 checks passed).

### Statistical test run on an unvetted raw column (sentinels/outliers/sample-size limits) with no sanity cross-check

- **Applies when**: `task` -- the task asks for a hypothesis test or distribution statistic on a single column, and the script feeds the column straight into the test after at most a `dropna()`.
- **Pattern**: The attempt treats "the column" as automatically equal to "the valid data": it never inspects value counts, sentinel/placeholder codes (0, -1, -9, 9999), duplicated header/summary rows, or the row count relative to the test's validity range (e.g., large-N tests that reject almost any real data, or scipy warnings about approximate p-values). It then reports the raw verdict even when the accompanying descriptive statistics (large |skew|, heavy tails, min/max far from the bulk) point to a contaminating subpopulation rather than a genuinely non-normal target distribution.
- **Detection procedure**:
  1. Read the task for any implied population/filter (valid records, non-missing coverage, a specific window/subset) and for the test's stated parameters; note whether the script encodes that filter at all.
  2. In the script, check whether anything beyond `dropna()` validates the input: printed value counts / min-max / histogram, removal of sentinel or degenerate rows, and an explicit check that N is inside the test's reliable range.
  3. Compare the reported test verdict with the reported companion statistics and summary output: if the rejection is driven by a small number of extreme values or by very large N, but the bulk of the data looks symmetric, the input was not vetted.
  4. Confirm the script does not re-run or contrast the test on the cleaned/plausible subset before committing to an answer.
- **Discriminator**: A real violation is when the script never examines the column's value distribution or N-validity and the reported skew/kurtosis are extreme relative to what the described quantity should look like. It is fine if the script prints diagnostics (counts, extremes, distribution shape), explicitly justifies keeping all rows, and confirms the test is valid at that sample size — even if the final verdict is "not normal".
- **Consequence**: The p-value is computed on a contaminated or out-of-range sample, so the normality/hypothesis verdict flips relative to ground truth and the shape statistics are distorted, failing the graded check despite a syntactically correct answer format.

### Final deliverable comes from an unvalidated "fast fallback" script that overwrites a validated model's output
- **Applies when**: `task` -- multiple scripts in the attempt write to the same required output artifact, and at least one of them is a quick/simplified fallback model with no held-out evaluation.
- **Pattern**: The agent builds and validates one modeling approach, then later runs a stripped-down script (simpler model, no train/validation split, no metric printed) that rewrites the same output path; the delivered artifact therefore reflects the weakest, never-scored model, and the answer describes it as if it were the chosen best approach without any comparative evidence.
- **Detection procedure**:
  1. From the task, note the exact required output artifact (name and location) and the metric that will be used to judge it.
  2. Scan every script for writes to that artifact; determine which script executes last and thus determines the delivered content, and confirm the path/filename matches exactly what the task requires.
  3. Check whether that final-writing script computes any held-out/cross-validated score, and whether its score was compared against the other candidate models before being chosen.
  4. Compare the answer's claimed model/metrics against what the final-writing script actually does and reports.
- **Discriminator**: A real violation is when the last writer is unvalidated or was shown/likely to be weaker (e.g., plain linear model replacing a tuned/ensembled tree model) or writes to a path other than the required one; it is fine if the final writer is the model whose held-out score was demonstrably the best among candidates and writes to the specified path.
- **Consequence**: The graded file contains predictions from an inferior or misplaced model, so the score falls below the accepted threshold (or the expected file is judged missing/wrong) even though the answer text claims a validated, well-performing model.

### Guessed eligibility threshold applied to every sub-ranking instead of only where defined
- **Applies when**: `task` -- The task asks for several separate top-N rankings of grouped entities and states a minimum-qualification rule (or gives a partially specified/ambiguous rule) plus a sample output file defining the expected format.
- **Pattern**: The agent invents an interpretation of the qualification rule (e.g., re-deriving the threshold as a per-item average rather than the stated aggregate), never cross-checks it against the spec or the provided sample output, and then applies that single filter uniformly to all sub-rankings even though the rule was scoped to only one of them — silently dropping entities that legitimately belong in the other rankings.
- **Detection procedure**:
  1. Read the task/README and note exactly which sub-question the qualification rule is attached to, what quantity it is measured on, and whether the statement is complete.
  2. Read the script and check how the threshold quantity is computed (aggregate vs. mean, per-group vs. per-row) and to which sub-rankings the filtered frame is passed.
  3. Check whether the script ever loads/inspects the provided sample output file to confirm column names, ordering, id conventions, and to sanity-check candidate entries.
  4. Compare the filtered group count to the unfiltered count and ask whether obviously dominant entities in the unrestricted sub-rankings could have been removed by the filter.
- **Discriminator**: A real violation is applying an unstated or re-derived filter to sub-questions where the task never imposed it, or redefining the threshold statistic without evidence; it is fine if the task explicitly scopes one qualification rule to all outputs, or if the agent tests both readings and validates the choice against the sample/README.
- **Consequence**: Rankings for the unfiltered categories contain the wrong entities (or wrong order), so exact-match comparison against the expected file fails even though the per-group statistics were computed correctly.

### Fabricating synthetic input data instead of using the provided dataset
- **Applies when**: `task` -- The task references a supplied dataset (and config/spec files) that the analysis must be computed from.
- **Pattern**: The agent cannot find or does not load the real input file, so it generates random/simulated records (e.g., `np.random`, hard-coded sample rows) and runs the requested analysis on that invented data, then reports the resulting numbers as if they were real findings.
- **Detection procedure**:
  1. Read the task/README to identify which input file(s) the deliverable must be derived from, and any required output artifacts.
  2. Scan the scripts for data creation calls (`np.random.*`, synthetic loops building DataFrames, `to_csv` of self-made data) and check whether the path actually read for the analysis is a file the agent itself wrote rather than the provided dataset.
  3. Check the answer's reported counts/ranges against the dataset's documented scale and semantics (e.g., total row count, expected units/format of fields); flag mismatches or suspiciously round/normal-looking distributions.
  4. Confirm every required output artifact is produced from the real data; missing ones (e.g., a serialized plot/array file) also signal the spec was not followed.
- **Discriminator**: A real violation is when the reported result is computed from agent-generated values; it is fine to synthesize data purely for smoke-testing/unit-testing code, provided the final artifacts and reported numbers come from loading the provided dataset.
- **Consequence**: All value-based checks fail — plotted/serialized counts and any derived files differ entirely from the ground truth, so the grader marks every expected file wrong or missing.

### Ambiguous standard-deviation convention (population vs. sample `ddof`) chosen without justification or cross-check
- **Applies when**: `task` -- the task asks for a dispersion statistic (standard deviation, variance, standard error) on a data subset and the script computes it with a library call that has a `ddof`/denominator option.
- **Pattern**: The script silently picks one normalization (e.g. `np.std(x)` with `ddof=0`, or conversely `ddof=1`) without checking which convention the task/reference tooling implies, and reports a single rounded number; the mean matches the reference but the spread statistic is off by the `n/(n-1)` factor.
- **Detection procedure**:
  1. Read the task statement for any hint about the convention (sample vs. population, "unbiased", or the tool implied by the workflow, e.g. pandas/`describe()` defaults vs. numpy defaults).
  2. In the script, locate the dispersion computation and note the explicit or default `ddof`.
  3. Check whether the script computes the alternative convention as well (or prints both values with the subset size `n`) so the reporter can see the magnitude of the difference; if only one value is produced and no rationale is given, flag it.
  4. Confirm the reported spread is consistent with the reported mean/subset (same filtered array, same rounding) — a mismatch of roughly a factor `sqrt(n/(n-1))` from the plausible reference indicates the wrong convention.
- **Discriminator**: Not a violation if the task or its stated tooling unambiguously fixes the convention (e.g. "population standard deviation", or the script mirrors the pipeline's stated library default) and the script's `ddof` matches it; it *is* a violation when the convention is unstated, the script uses a non-default-for-the-ecosystem `ddof`, and no both-ways sanity print or justification exists.
- **Consequence**: The mean and any outlier/subset outputs check out while the standard deviation fails the grader by the small `sqrt(n/(n-1))` factor, producing a partially-correct answer (e.g. 1 of 2 numeric checks passing).

### Answer asserted from prior/world knowledge instead of computed from the provided data
- **Applies when**: `task` -- the task asks for a ranking, extreme values, or statistics that must be derived from a supplied dataset after a specified preprocessing step (e.g., a stated imputation rule), and the answer is plausible-looking domain knowledge.
- **Pattern**: The attempt ships a final answer with no (or only trivial) code that actually loads the file, applies the stated preprocessing, sorts, and slices the requested rows — the entries are ones a knowledgeable person would guess, and the required transformation is never demonstrated on the real column values.
- **Detection procedure**:
  1. Read the task and list every operational requirement (which file/column, the required missing-value handling, the sort direction, the number of items, the output schema).
  2. Inspect the submitted scripts/logs for an executable chain: read data → apply the stated preprocessing → compute/sort → select top-N → write the requested output file; note any requirement with no corresponding code.
  3. Cross-check the reported entries against the dataset's own naming and coverage (exact country/entity spellings, whether such rows exist at all, whether ties or imputed rows change the boundary of the top-N).
  4. Confirm the answer was persisted in the exact requested artifact (filename, keys, ordering) rather than only printed in prose.
- **Discriminator**: A real violation is when no run on the actual file can be traced (or the reported labels don't match the dataset's label strings/row set), so the result is unverifiable guesswork; a look-alike that is fine is a fully scripted pipeline whose output happens to agree with domain intuition, with printed intermediate values (row counts, imputed-cell count, sorted head/tail) proving it came from the data.
- **Consequence**: Entity names or membership deviate from what the dataset yields (different spellings, missing/extra countries, wrong boundary member or order), so the expected result file comparison fails and the check scores 0.

### Answer-field formatting not reproduced exactly as specified (quoting/literal form of string values)
- **Applies when**: `task` -- the task dictates a rigid answer template with tagged fields and says certain values must be given as strings (often showing a quoted example), and the script builds the answer text programmatically.
- **Pattern**: The attempt computes the right substantive values but emits them in a different surface form than the template demands — e.g. dropping the quotation marks the example shows, or deriving the literal from data via casting/formatting (`f"{int(min)}-{int(max)}"`, rounding, unit suffixes) instead of writing the exact required token — so an exact-match grader fails fields whose content is actually correct.
- **Detection procedure**:
  1. In the task statement, extract every answer field with its exact demanded surface form: tag name, delimiter, whether the value is quoted, and any example literal.
  2. In the script, locate the line(s) that assemble/print the final answer and compare character-by-character to the template: check that quotes appear where the example has them, that all fields use the same quoting convention, and that no field is silently reformatted.
  3. Trace any dynamically derived string field back to its source and ask whether the computation can produce a token differing from the required literal (extra decimals, `-0`, reversed order, whitespace, different separator).
  4. Confirm the submitted answer string reproduces the template for *every* field, not just some.
- **Discriminator**: A real violation is an inconsistency with the explicitly shown template (e.g. one field quoted and another not, or a computed literal that need not match the example form); it is *not* a violation if the task's template genuinely leaves formatting free, or if the derived string is provably identical to the required literal in all cases.
- **Consequence**: The grader's exact string comparison marks correctly-computed fields as WRONG/MISSING, so the attempt scores partial credit (e.g. 1/3) despite sound analysis.

### Entity substitution when the requested fields aren't present in the data
- **Applies when**: `task` -- the task names specific entities/metrics/stages to compute, and the loaded files appear not to contain columns matching those names.
- **Pattern**: Instead of locating the correct input file or the correct columns (or stopping to report the mismatch), the agent silently redefines the requested concepts as unrelated available columns and produces an output that matches the requested *shape* but not the requested *content*; required auxiliary outputs are also skipped.
- **Detection procedure**: 1. List every entity, metric, grouping key, and output artifact explicitly named in the task. 2. Read the scripts/answer for the columns and files actually used and check each named concept maps to a real, semantically matching field. 3. Flag any place where the answer states or implies a proxy mapping ("X treated as Y") or where axis/legend labels contradict the described quantities. 4. Verify all requested output files are produced, not just the chart/number.
- **Discriminator**: Legitimate: a documented, semantically equivalent renaming (e.g., a column with a synonym name or a derived quantity that truly equals the requested one). Violation: the substitute measures a different phenomenon, or the agent had to reinterpret the whole task domain to force a fit — the correct data source or columns were likely elsewhere/unsearched.
- **Consequence**: Outputs computed from the wrong variables and mislabeled axes; comparison against expected artifacts fails on all checks, and missing required side outputs count as wrong/missing.

### Answer-string format not emitted in fully-closed, parseable form
- **Applies when**: `task` -- the task prescribes a literal answer template (a tagged key with brackets/braces holding a dict, list, or tuple of per-column/per-class values).
- **Pattern**: The attempt computes correct numbers but copies the template verbatim from the prompt, including truncated or unbalanced delimiters (e.g. an opening `[`/`{` with no matching `}`/`]`), omits required quoting/separators, or reorders/renames keys — so an automatic parser cannot extract the object and the answer scores zero despite correct analysis.
- **Detection procedure**:
  1. Read the task and write down the exact required wrapper: tag name, opening and closing delimiters, key names, key order, and value types.
  2. Read the final answer string and count/match every bracket, brace, and quote; check that the enclosed literal is valid syntax (i.e. would survive `ast.literal_eval` after stripping the tag).
  3. Check the scripts/answer step for an explicit serialization + self-parse (or at least a printed final string) rather than hand-typed text pasted from the prompt template.
  4. Confirm all required keys are present, spelled exactly as in the constraints, with values in the requested units/rounding.
- **Discriminator**: A real violation is a structurally unparseable or key-mismatched answer string (unbalanced delimiters, missing keys, stray text inside the object); cosmetic differences that still parse to the same object — spacing, single vs double quotes, trailing newline — are fine.
- **Consequence**: The grader reports the expected key as WRONG/MISSING and 0/1 checks passed even though the submitted numbers equal the ground truth.

### Ships an unvalidated model that only uses a convenience subset of features
- **Applies when**: `task` -- the task asks for predictions on a held-out file and the scripts fit a model, write the output, and stop, without any hold-out/cross-validated error estimate or baseline comparison.
- **Pattern**: The agent hand-picks a narrow slate of "easy" numeric columns (dropping identifier, categorical, date/text, and other metadata columns that are present in both train and test and are often far more predictive), fits one or two off-the-shelf regressors, and reports success based only on descriptive statistics of the predictions (min/max/mean/std) and feature importances. Nothing in the pipeline estimates generalization error, so a model whose predictions are barely better than predicting the global mean (very compressed prediction spread relative to the target's spread) is indistinguishable from a good one.
- **Detection procedure**:
  1. Read the task: confirm it is a supervised prediction task scored against hidden ground truth, so accuracy — not just file existence — determines success.
  2. Read the scripts: check for any train/validation split, cross-validation, or metric (MAE/RMSE/R²/accuracy) computed on data the model did not fit, and check for a trivial baseline (predict mean/median/majority) to compare against. Also list which columns shared by train and test were excluded and whether the exclusion was justified (e.g., leakage) or merely convenience.
  3. Read the answer: see whether reported evidence is only prediction summary stats/feature importances; compare the predicted standard deviation to the training target's standard deviation — a large shrinkage with no reported validation score is a red flag for a near-mean-only model.
  4. Flag if no out-of-sample score exists, or if strong shared features were silently dropped without testing whether including them helps.
- **Discriminator**: A real violation is zero out-of-sample evaluation plus unexplained feature omissions. It is *not* a violation if the agent reports a validation/CV metric (even a mediocre one) beating a stated baseline, or explicitly demonstrates that the omitted columns are unusable (absent from test, constant, or leaky) and that the chosen features were compared against alternatives.
- **Consequence**: The saved file has the right shape and column name but the values are near-constant around the training mean; the grader's error/correlation threshold against true labels fails, and the attempt is scored wrong despite "completing" the deliverable.

### Choosing the number of clusters by blind argmax of a score, without rejecting degenerate solutions
- **Applies when**: `task` -- the task asks for an unsupervised grouping "into an appropriate number of groups" and the script sweeps a hyperparameter (k) and picks the value maximizing an internal quality score.
- **Pattern**: The script takes `argmax` of a metric whose values differ only in the noise range across candidates, ignores the elbow/stability and domain expectation (a small number of interpretable tiers), and accepts the resulting partition even when some groups contain one or a handful of points — i.e. outlier-driven singleton clusters rather than a meaningful segmentation. No sanity check on cluster sizes, no comparison of the top candidates, no outlier/skew handling before scaling.
- **Detection procedure**:
  1. Read the task for how the group count should be justified and whether the intended output is a coarse, interpretable segmentation.
  2. In the script, check whether the selection rule is a bare `argmax`/`argmin` over the sweep, and whether any tie-breaking, elbow inspection, stability across seeds, or minimum-cluster-size filter is applied.
  3. In the reported scores, check whether the winning candidate beats the runner-ups by a trivial margin (e.g. differences within a few thousandths, or a non-monotone/noisy curve).
  4. In the reported cluster distribution, look for groups whose size is a tiny fraction of n (single-digit counts); confirm the script never validated sizes or re-ran with a smaller/simpler k.
- **Discriminator**: A real violation is when the selected candidate wins by a negligible margin and/or yields degenerate micro-clusters that the script never questions. It is fine if the score has a clear separated maximum (or a clear elbow), cluster sizes are all non-trivial, and the choice is defended against neighbouring candidates — even if the chosen k is unusual.
- **Consequence**: The saved label column encodes an unstable, outlier-driven partition with a group count different from the expected segmentation, so any check comparing the number of clusters, cluster sizes, or label agreement/quality against the reference fails, and the file is scored WRONG.

### Statistic computed over the wrong axis/subset, ignoring the qualifier stated in the question
- **Applies when**: `task` -- the question names a specific slice (a year, group, region, or field) and/or "all entities in the dataset", and the script computes a summary statistic from a table with entities on one axis and periods/variables on the other.
- **Pattern**: The script aggregates over the convenient axis or over the whole table (e.g., every period for each entity, or only one partial input file) instead of over the exact population the question specifies, so the reported statistic answers a different question than the one asked.
- **Detection procedure**:
  1. From the task text, write down explicitly the *unit of aggregation* (what varies inside each computed statistic) and the *filter* (which rows/columns/files are in scope, including whether "all" implies loading more than one input file).
  2. In the script, locate the slicing/indexing that feeds the statistic and check that the values passed in correspond exactly to the stated filter, and that the axis matches the stated unit of aggregation.
  3. Verify coverage: does the loaded data span the full entity population named in the task (check file list / row count / key counts), or only a subset?
  4. Check the printed diagnostics — if the vector length or value list shown does not equal the count implied by the task's filter, the statistic is on the wrong subset/axis.
- **Discriminator**: A real violation is when the stated qualifier is never used anywhere in the code (no filter on the named slice, or extra data silently dropped/ignored). It is *not* a violation if the qualifier is legitimately applied earlier (e.g., data already restricted upstream, or the qualifier defines the grouping key) and the aggregation vector still matches the task's definition.
- **Consequence**: The ranking/argmax is drawn from a different distribution than requested, so the reported entity disagrees with ground truth even though the code runs cleanly and the statistic function itself is used correctly.

### Truncating or coarsening an identified key value to match a loosely-worded format hint
- **Applies when**: `task` -- the answer requires reporting an identifier (a date, ID, label, key) located by an argmax/argmin or lookup, and the stated answer template shows a format that is less precise than the value actually stored in the data.
- **Pattern**: The agent finds the correct record, then reformats/truncates the identifier to fit the template literally (e.g. dropping the day component, rounding an index, stripping a suffix), discarding information that uniquely identifies the record — even though every downstream computation was done on the full-precision value.
- **Detection procedure**:
  1. Read the task and note the granularity of the identifier in the source data versus the granularity implied by the answer template.
  2. In the scripts, find where the identifier is selected (argmax/lookup) and trace any subsequent string formatting, casting, rounding, or `.strftime`/slicing applied before printing.
  3. Check whether the reported identifier still uniquely picks out the row used for the rest of the computation; if the reported form maps to many rows in the data, it has lost information.
  4. Confirm consistency: the identifier reported must be the same object used to fetch the neighbouring/derived values reported alongside it.
- **Discriminator**: A real violation is when the transformation *destroys* precision present in the data (many rows share the reported value). It is fine when the data's native granularity genuinely equals the template's (all records are monthly, so a month string is exact), or when the task explicitly says to aggregate to that coarser level.
- **Consequence**: The derived numeric answer matches, but the identifier field is scored WRONG against the full-precision ground truth, so the submission fails on a partial-credit check (1/2 here) despite correct analysis.

### Emitting identifiers with extra quoting/decoration inside the required answer delimiters
- **Applies when**: `task` -- the task specifies a literal answer template (e.g. `@field[item1,item2,...]`) whose list items are strings such as labels, IDs, or category names.
- **Pattern**: The attempt computes the right values but formats list items with added syntax — surrounding quotes, brackets, `np.str_(...)`, spaces, or a Python `repr`/`list` dump — instead of the bare comma-separated tokens the template shows, so string-valued fields fail exact matching while numeric fields pass.
- **Detection procedure**:
  1. Read the task's answer format and note exactly which characters are allowed inside each `@field[...]`: bare tokens separated by single commas, no quotes, no spaces.
  2. In the scripts, find where the answer string is built and check whether items come from `str(x)`/joining raw tokens versus `print(list_object)`, f-strings on a list, or `json.dumps`, all of which inject quotes/brackets.
  3. Compare the submitted answer character-by-character against the template; flag any `"`, `'`, nested `[]`, or whitespace around string items.
  4. Confirm the identifiers themselves are the raw values as they appear in the source data (no added prefixes, type wrappers, or normalization).
- **Discriminator**: A real violation is decoration added by the formatting step (quotes/brackets/spaces not present in the underlying identifier); it is *not* a violation if the special characters, e.g. parentheses, are genuinely part of the identifier value in the data.
- **Consequence**: The numeric field matches but the string-identifier field is scored WRONG/MISSING, giving a partial-credit failure (e.g. 1/2 checks) despite a correct analysis.

### Predictions emitted without shape/prevalence sanity check against the stated cost‑sensitive objective
- **Applies when**: `task` -- the task asks for a per-row prediction file for a held-out test set (especially a rare-event/imbalanced target where the stated objective penalizes missed positives more than false alarms).
- **Pattern**: The agent produces a label column with no saved, re-runnable script and no verification that (a) the number of predicted rows equals the number of test rows, (b) the header/format matches the provided sample file, and (c) the predicted positive rate is plausible given the training prevalence and the asymmetric error costs — typically a default 0.5 threshold on an imbalanced model yields far too few positives, or the file is silently truncated.
- **Detection procedure**:
  1. Read the task: note the required file name, exact header/column, row count implied by the test file, and any stated asymmetry in error costs (e.g., missed positives are the expensive error).
  2. Read the scripts: check that they exist and are reproducible, that predictions are generated on the full test set in original row order, and that the decision threshold / class weighting was chosen against a validation set using the cost-aligned metric (recall/F-beta/expected cost), not left at the library default.
  3. Inspect the answer file: count data rows and compare with the test-set row count; confirm header matches the sample exactly; compute the fraction of positives.
  4. Flag if row count differs, header/values deviate from the sample format, or the positive rate is drastically below the training prevalence with no documented threshold-tuning justification.
- **Discriminator**: A low positive rate is fine if the script shows the threshold was tuned on held-out data under the stated cost/metric and the row count and format check out; it is a violation when the rate simply reflects an untuned default, or when row count/header cannot be shown to match the required output spec.
- **Consequence**: The grader compares the file row-by-row against the expected output and marks it WRONG/MISSING — either due to a length/format mismatch or because the recall-oriented score collapses when nearly all positives are predicted as negative.

### Analysis run on a partial data slice when the task specifies the full population
- **Applies when**: `task` -- the question asks for a statistic or set of entities over an entire population ("all X"), and the scripts load a single file/subset out of a directory that contains several partitions of the data.
- **Pattern**: The agent hard-codes one input file (one region/segment/split) without ever checking what other data files or rows exist, then computes quartiles/thresholds/rankings from that subset and reports them as if they applied to the whole population. It also never states the number of entities analyzed against the number implied by the task.
- **Detection procedure**:
  1. Read the task and note the scope words ("all countries", "the whole dataset", "every record") and the expected population size or breadth.
  2. Read the scripts: identify every data-loading call; check whether the agent enumerated the available data files/tables (e.g., listing the data directory) or simply picked one, and whether any concatenation/merge covers the remaining partitions.
  3. Compare the row/entity count printed by the script (or implied by the file) with the scope in the task; if the loaded set is clearly one partition of a larger whole, flag it.
  4. Check the reported answer format character-for-character against the requested template (delimiters, quoting, ordering, separators) before accepting it.
- **Discriminator**: A real violation is when other partitions of the same variable exist (or the file name/`shape` shows a subset) and were never inspected or combined; it is fine if the agent verified that the single file is the complete population, or if the task itself restricts the scope to that subset. Note that a subset answer can coincidentally overlap the true answer — the absence of any coverage check is the violation, not just a differing result.
- **Consequence**: Threshold-based or rank-based results (quartiles, cutoffs, top-k, outliers) are computed from the wrong distribution, so the reported entity list is incomplete or spurious and the grader marks the expected value WRONG/MISSING, even when part of the list happens to match.

### Declaring success without a held-out validation score or output verification
- **Applies when**: `task` -- the task asks for predictions/derived values written to a specific output file, and the scripts train a model and write that file in one pass.
- **Pattern**: The attempt reports a pipeline description and prediction class counts as evidence of success, but never (a) holds out part of the labeled data to measure accuracy/appropriate metric, nor (b) re-reads the written file to confirm its path, row count, column name, ordering relative to the input rows, and that the emitted label strings/dtype exactly match the label vocabulary/format seen in the training target. Preprocessing choices (encoders fit separately per split, scalers, imputation, dropped/reordered columns) are therefore never checked for consistency between train and test.
- **Detection procedure**:
  1. Read the task for the required output artifact: exact filename/location, column name, expected number of rows, and the value format implied by the labeled data.
  2. Scan the scripts for any validation split or cross-validation producing a numeric score on labeled data; if absent, and no baseline comparison exists, the attempt has no evidence the model works.
  3. Scan for a post-write verification block that loads the output file and asserts shape, column name, row alignment with the test input, and membership of predicted values in the training label set.
  4. Check that any encoders/imputers/scalers are fit on training data only and applied with identical column order to test; a per-split `fit_transform` or independently built category mappings is a silent mismatch.
- **Discriminator**: A fine attempt may skip elaborate tuning but still prints a measured validation metric and an explicit file/format sanity check; a violation is one where the only "evidence" is descriptive text and a class-count breakdown, which cannot detect label mismatch, row misalignment, wrong path, or a broken encoding.
- **Consequence**: The grader reads the output file and finds it missing at the expected path or mismatched in labels/order/row count, scoring 0 even though the agent reports "TASK COMPLETED SUCCESSFULLY".

### Fabricated metric definition computed on only half the relevant records
- **Applies when**: `task` -- the task asks to visualize/report "performance" (or any composite score) whose formula is not spelled out in the prompt, and the script must aggregate per-entity records that appear in more than one role/column (or across more than one file).
- **Pattern**: The agent invents an arbitrary formula (e.g., `count + 2 * some_flag_count`) with no basis in the task text, config file, or upstream artifacts, and computes it by grouping on a single role column, so every record where the entity appears in the other role is silently dropped; it also skips the auxiliary result artifacts the task's output convention implies.
- **Detection procedure**:
  1. Read the task and any provided config/spec files for the definition of the quantity to be plotted (axis labels, prior-step outputs, saved intermediates); note whether the agent's formula is stated anywhere.
  2. In the script, check the aggregation key: does the entity of interest occur in multiple columns/tables, and does the groupby cover all of them (plus correct dtype comparisons for flags, e.g. boolean vs. the string `'TRUE'`)?
  3. Sanity-check magnitudes in the answer: are per-entity counts roughly half (or otherwise inconsistent with) the totals printed for the filtered data, and does the resulting ranking contradict domain-obvious expectations?
  4. Confirm every output artifact the task/grader convention requires (figure plus any serialized values/plot metadata) is actually written.
- **Discriminator**: A real violation is a formula with no source in the prompt/config and an aggregation that provably ignores records where the entity appears under another column; it is fine if the metric is explicitly defined by the task/config, or if the single-column grouping is exactly what the definition requires (and the script demonstrates the other role contributes nothing).
- **Consequence**: The plotted values and any saved numeric array differ from the reference, so the figure and all value-based checks fail, and missing auxiliary files fail outright regardless of the chart's cosmetic correctness.

### Threshold-based outlier count not reconciled with the extreme-value sanity check
- **Applies when**: `task` -- The task asks for a count of rows flagged by an explicit statistical rule with a fixed cutoff (e.g., a standardized-score, quantile, or distance threshold) computed on a single numeric column.
- **Pattern**: The script applies some flagging rule but never verifies that the standardization matches the stated definition (population vs. sample SD, mean/SD computed on the correct subset after NaN/dtype cleaning, raw values vs. transformed/scaled values) and never prints the extreme statistics of the score; a nonzero-looking count is reported without checking whether the distribution can even exceed the cutoff. Common variants: using a different rule than stated (IQR, modified/median-based z, percentile), using a lower effective cutoff, standardizing groupwise or per-chunk, or counting flagged values in the wrong column/axis.
- **Detection procedure**:
  1. Read the task and write down the exact rule and cutoff, and which quantity the count refers to (rows vs. values, before or after cleaning).
  2. In the script, locate where the score is computed: confirm the mean/SD (or quantiles) come from the full cleaned column of interest, that the same column is thresholded, and that the comparison uses the stated cutoff with the stated sign convention (two-sided if "higher than X or lower than -X").
  3. Check that the script prints diagnostics that bound the answer — column dtype, non-null count, min/max of the score (or min/max of the raw value), and the count itself — and that the reported count is consistent with them (e.g., if max |score| < cutoff, the count must be 0).
  4. Compare the final reported integer to those diagnostics and to the requested output format; a count reported with no printed extreme-score evidence is unverified.
- **Discriminator**: A real violation is when the rule/cutoff in code differs from the task's, or when no diagnostic exists that could distinguish "several outliers" from "none" — including the case where the reported count contradicts the printed extremes. It is *not* a violation if the script implements the stated rule exactly on the correct cleaned column and prints extremes that are consistent with a nonzero count; large counts are legitimate for genuinely heavy-tailed data.
- **Consequence**: The reported count is off (often nonzero when the correct answer is zero, or vice versa), so the single numeric check fails and the task scores 0.

### Hard-coding a mapping/spec from assumption instead of reading the referenced auxiliary file
- **Applies when**: `task` -- The prompt tells the agent to apply a definition, label mapping, filter, or parameter set that is documented in an auxiliary file (README, tips/notes, data dictionary, config) rather than in the prompt itself.
- **Pattern**: The script never opens or quotes the referenced file; instead it embeds a mapping/threshold/category set the agent recalled from general knowledge or inferred from the raw column values, then computes the requested statistic on those self-invented labels (and often skips writing the required output artifact too).
- **Detection procedure**:
  1. From the task text, list every external artifact named as the source of truth (e.g. "use the mapping from <file>") and every required output artifact/format.
  2. Grep the scripts for a read/open/parse of each named source file; check that the transformation values actually originate from it rather than from a literal dict/list defined in code.
  3. If values are literals, check whether the script at least prints the file's contents or otherwise verifies the literals match the spec; absent that, the mapping is unverified — also confirm the required output file is written, not just printed.
  4. Compare the reported label strings/granularity to what the spec plausibly requires (e.g. could the spec collapse or rename categories, changing which is most frequent or its ratio?).
- **Discriminator**: A real violation is when the spec file is never read and never echoed, so nothing in the run confirms the literals match it. It is fine if the agent read the file (printed/cat'd it) and then hard-coded the identical mapping, or if the task itself fully specifies the mapping inline.
- **Consequence**: Labels (and therefore the modal category name, and possibly the counts/ratio if categories are merged) fail exact-match comparison against the expected values, and a missing required result file scores 0 regardless of the numbers.

### Submission file not validated against the provided template schema
- **Applies when**: `task` -- The task supplies a sample/template output file and requires writing predictions to a specified output file matching that format.
- **Pattern**: The attempt builds the output from its own assumptions (hand-typed header names, capitalization, column order, or ids taken from the wrong source) and writes/reports it without ever loading the template file and comparing header text, row count, id set and ordering, and value ranges against it.
- **Detection procedure**:
  1. Read the task/README for the required output filename and the template file that defines its schema.
  2. In the scripts, check whether the template is actually read and used to construct or verify the output (header strings compared exactly including case, ids taken from the test/template file, column order preserved), and whether the file is written to the exact required path.
  3. Compare the produced answer's first line and shape to the template: identical column names/case/order, one row per test id in the template's order, no extra index column, and values in the valid range (e.g., probabilities summing to 1 per row where required).
  4. Flag if any assertion/sanity check on row count, id match, or header equality is absent.
- **Discriminator**: A real violation is a header/case/column-order/id-coverage/path mismatch or absence of any verification against the template; it is fine if the script constructs the frame from the template's own ids and columns (or asserts equality with it), even if it never prints the check.
- **Consequence**: The grader reads the expected file's schema and reports the submission as WRONG/MISSING (unparseable or mismatched ids/columns), scoring 0 regardless of how good the underlying model was.

### Substituting heuristic pseudo-labels and an ad-hoc row set for the specified training labels and test file
- **Applies when**: `task` -- the task names specific input/output files and a target column, and the scripts must train on the provided labeled data and predict for exactly the provided evaluation rows.
- **Pattern**: The agent claims the labeled training data or the designated test file is "missing", invents the target values with hand-written rules (thresholds, keyword/category maps), fits a model on those self-generated labels, and then writes predictions for whatever subset of rows it could build features for (often all rows, or only rows present in some auxiliary file) instead of the exact evaluation rows in the required order/format.
- **Detection procedure**:
  1. From the task, list the required inputs (labeled source, evaluation file), the required output filename/columns, and the expected number of prediction rows.
  2. In the scripts, check whether the target column is ever read from the data; flag any code that constructs the target from `if/elif` rules or keyword dictionaries, and any `os.path.exists(...)` fallback that silently proceeds when the specified file is absent.
  3. Check how the output row set is derived: does it iterate over the evaluation file's identifiers exactly (no `continue`/filter dropping rows, no reliance on auxiliary tables), and does it preserve count and order?
  4. Compare the answer's reported row count and label vocabulary against the expected evaluation size and the label values actually used in the dataset (spelling, casing, category set).
- **Discriminator**: A real violation is when labels are synthesized rather than learned from provided ground truth, or when output rows do not correspond 1:1 to the evaluation rows. It is fine to use rule-based features or a heuristic baseline *if* the model is fit/validated against genuine labels from the data and predictions cover exactly the required rows; it is also fine to search additional paths for a file that truly exists elsewhere.
- **Consequence**: The grader finds `result.csv` mismatched in row count/identifiers and label categories, and accuracy against true labels is near chance, so the file is marked WRONG/MISSING despite a confident "task completed" report.

### Output template provided but never read or validated against
- **Applies when**: `task` -- The task says the result file's format must match a provided template/example file, and the scripts write the output file.
- **Pattern**: The agent never opens or parses the template; it guesses the header names, index labels, row/column ordering, dtype, precision, and any implied filtering/derivation conventions, then asserts "format matches" without a comparison step.
- **Detection procedure**:
  1. In the task statement, note that a template/reference file is supplied and list what it constrains (column names/count, index labeling, row order, value precision, null representation).
  2. Search the scripts for any read of that template (e.g., loading it, printing its head, comparing headers/shape); if absent, the format claim is unverified.
  3. Check whether formatting decisions in the script are hard-coded guesses (renaming columns to invented labels, ad-hoc date string formats, arbitrary rounding, dropping/keeping rows without justification) rather than derived from the template.
  4. Confirm the answer text states format compliance without showing a header-by-header / shape-by-shape diff against the template.
- **Discriminator**: A real violation is guessing the schema while a template exists; it is fine if the script explicitly loads the template and asserts equality of columns/shape/index (or reindexes the result onto the template's structure), even if the printed summary is brief.
- **Consequence**: The written file differs from the expected one in headers, key labels, ordering, or rounding, so an exact/structural file comparison fails even when the underlying aggregation logic is defensible.

### Optimizing/selecting on a proxy metric instead of the task's stated evaluation metric
- **Applies when**: `task` -- the task specifies an explicit scoring metric (e.g., a weighted/ordinal agreement, ranking, or cost-sensitive score) and the scripts train, tune, cross-validate, weight ensembles, or choose a decision rule for the final predictions.
- **Pattern**: The scripts never compute the stated metric anywhere; they score candidate models with a convenient default (plain accuracy/log-loss), weight or pick the ensemble by that default, and take plain argmax over class probabilities, ignoring the metric's structure (e.g., ordering of classes, asymmetric penalties, tie-breaking) — so no evidence exists that the submitted predictions are good under the actual grading rule.
- **Detection procedure**:
  1. Read the task/README and note the exact metric name and its properties (ordered classes? asymmetric error costs? rank-based?).
  2. Grep the scripts for that metric or an equivalent scorer used inside `cross_val_score`/`GridSearchCV`/manual holdout evaluation, and for any comparison of alternative prediction strategies under it.
  3. Check what quantity the printed model-selection numbers and ensemble weights are derived from, and how probabilities are converted to final labels.
  4. Inspect the answer: if predictions collapse onto the few majority classes and no metric-specific validation number was ever reported, flag it.
- **Discriminator**: A real violation is when the stated metric is never estimated on held-out data, so selection and thresholding rest solely on a proxy. It is fine if the agent computes the stated metric on a validation split (or a custom scorer) and shows the proxy-chosen model also wins under it, or explicitly justifies the proxy as equivalent for this metric.
- **Consequence**: The submission is technically well-formatted but scores far below achievable under the real metric (near-chance/negative agreement for ordinal-weighted scores), so the grader marks the expected result file wrong.

### Null/missing-value partition defined inconsistently, so group subsets (and their statistics) don't reconstitute the full dataset
- **Applies when**: `task` -- the task asks to split rows into "missing" vs "non-missing" (or any complementary filter) on one column and compute per-group statistics or a test on another column.
- **Pattern**: The attempt uses an ad-hoc notion of "null" (e.g., only `NaN` while string sentinels like `""`, `"NA"`, `"None"`, `"-"`, `"null"` remain, or the reverse), or silently loses rows via `dropna()` on the whole frame, `read_csv` defaults/`na_values`, dtype coercion, deduplication, or a prior filter — so the two group sizes don't add up to the row count and the group means drift from the true values.
- **Detection procedure**:
  1. From the task, note that the two groups are complementary and must together cover every row of the source table.
  2. In the scripts, find where the data is loaded and where the mask is built; check for global `dropna()`/`fillna`, subsetting, type casts, or a mask that only tests one form of missingness.
  3. Confirm the script prints `len(group_a)`, `len(group_b)`, and `len(df)` and asserts they sum; also check the value column's own missing values are handled explicitly and identically for both groups.
  4. Sanity-check the reported statistics against these counts/ranges; if counts were never printed, the partition is unverified.
- **Discriminator**: A real violation is when rows can fall outside both groups (or are dropped before masking) or when only one missingness representation is recognized; it is fine if the script explicitly normalizes sentinels to NaN, documents any exclusion, and shows the two group sizes summing to the full row count.
- **Consequence**: Both group means (and the test statistic) are computed on the wrong subsets, so the reported numbers miss the expected values even though the p-value may still look decisively significant, and the grader marks the mean checks wrong.

### Substituting invented rules for the referenced specification, and emitting only a subset of required artifacts
- **Applies when**: `task` -- the task points to an external instruction/spec file (guidance, template, config) and/or an evaluation harness that consumes saved artifacts, and the script must implement exactly those steps and write exactly those outputs.
- **Pattern**: The script never loads/quotes the referenced spec; it improvises preprocessing and filtering thresholds (e.g., self-chosen cutoffs, self-derived reference dates, self-invented category mapping and ordering) and saves only the one artifact named in the prompt text, silently skipping the other data/figure artifacts the pipeline expects. The answer then presents the improvised rules as if they were the given requirements.
- **Detection procedure**:
  1. From the task, list every referenced spec document and every expected output artifact (data dumps, serialized plot metadata, arrays, images) with their exact filenames/paths.
  2. In the scripts, check that each spec-driven step is traceable to text actually read from the spec (or explicitly reproduced in a comment/print) rather than assumed, and that every listed artifact is written with the required name, shape, and content ordering.
  3. Flag if any thresholds, category definitions, orderings, or figure/legend/color assignments are chosen by the agent without a cited source, or if any expected artifact is never created.
  4. Check whether the final answer states these choices as assumptions or presents them as confirmed requirements.
- **Discriminator**: A real violation is inventing decision rules that materially change the reported counts/statistics, or omitting artifacts the grader consumes. A look-alike that is fine is a script that reads the spec (or reproduces its wording) and only fills in cosmetic details the spec leaves free, while still writing all required outputs.
- **Consequence**: All artifact-level checks fail (missing files, and the produced figure/values disagree with the spec's filtering and category definitions), so the run scores zero even though the reported numbers look internally consistent.

### Model error not sanity-checked against a trivial baseline (target variance), hiding broken feature/target preprocessing
- **Applies when**: `task` -- a supervised regression task asks for a specific error metric (e.g., MSE) after prescribed preprocessing such as mean-imputation of a few numeric columns.
- **Pattern**: The agent loads raw columns that are not cleanly numeric (strings with currency symbols, thousands separators, unit suffixes, footnote markers, ranges, placeholder tokens like "n/a"/"-"), applies `to_numeric(..., errors='coerce')`, `dropna`, or an ad-hoc parse that silently mangles or discards many values, then fits the model and reports whatever number comes out — without ever comparing the resulting error to the variance/std of the target or to a mean-predictor baseline, and without checking row counts and value ranges after cleaning.
- **Detection procedure**:
  1. From the task, note the exact preprocessing mandated (which columns, impute-with-mean vs drop, split fraction) and the metric requested.
  2. In the scripts, locate where each modeling column is converted to numeric: check whether non-numeric artifacts are stripped before conversion, whether coercion-to-NaN plus row-dropping replaces the required imputation, and whether the row count/NaN count is printed before and after cleaning.
  3. Check whether the script computes any reference quantity — target variance, std, or the MSE of predicting the training mean — and compares it with the reported metric.
  4. Compare the reported metric to the plausible scale of the target: an MSE far above the target's variance (i.e., worse than predicting the mean) or a nonsensical error magnitude relative to the target's observed range signals broken parsing/imputation or mismatched X/y rows.
- **Discriminator**: A genuine violation is when the reported error is at or above the naive mean-predictor error, or when rows were silently dropped/coerced instead of imputed as instructed, with no diagnostic printed. It is *not* a violation if the columns are already clean numerics, imputation follows the stated rule, counts/shapes are verified, and a weak-but-below-baseline error simply reflects genuinely low predictor signal.
- **Consequence**: The reported metric is off by an order of magnitude from the reference value computed on correctly parsed, mean-imputed data, so the graded numeric answer fails exact-match.

### Unjustified re-ordering of rows before a sequence-dependent (lag/diff) computation
- **Applies when**: `task` -- the task asks for a quantity defined relative to the "previous" row (differences, percent change, cumulative or rolling statistics) and the script re-sorts or re-indexes the data before computing it.
- **Pattern**: The agent assumes the file's row order is "wrong" (e.g., reversed) and sorts by a parsed key, then computes the lag-based column on the new order, without verifying that the task intended a different order than the one supplied; because the operation is order-sensitive, the sign and magnitude of the resulting statistics flip relative to the as-given order.
- **Detection procedure**:
  1. Read the task statement and check whether it specifies any sorting/ordering; if it only says "previous day/row", the as-given file order is the default definition.
  2. Scan the script for any `sort_values`, `sort_index`, `reset_index`, reversal, or groupby-reordering applied before `diff`/`shift`/`pct_change`/rolling operations.
  3. Check whether the script provides evidence for the re-ordering (printed head/tail of the key showing it was inconsistent) *and* whether it computes the statistic under both orderings and reports the discrepancy.
  4. Compare the reported sign/magnitude against a quick sanity expectation (e.g., mean change should be near-symmetric in magnitude but opposite in sign between the two orderings); a sign difference means the answer depends entirely on the unverified assumption.
- **Discriminator**: A real violation is silently changing row order (or leaving it unverified) for an order-sensitive computation when the task defined "previous" relative to the given data; it is fine if the task explicitly requires chronological/keyed ordering, or if the script demonstrates the file is already in that order so the sort is a no-op (verifiable by comparing pre- and post-sort indices).
- **Consequence**: The lag direction is inverted, so the mean flips sign (and dispersion shifts slightly), and the graded numeric checks fail even though the formula and rounding were implemented correctly.

### No held-out validation before committing to a model/ensemble
- **Applies when**: `task` -- the task asks for predictions on an unlabeled test set and the scripts fit one or more models and write predictions directly, with no scored comparison of alternatives.
- **Pattern**: The agent trains several estimators on the full labeled data, blends them with hand-picked weights (or picks hyperparameters by intuition), never computes any validation error on a held-out split or via cross-validation, and reports the submission as done — so weak or mis-specified components (e.g., linear models on label-encoded high-cardinality categoricals, low-capacity boosters) silently drag the blend's accuracy below the grader's threshold.
- **Detection procedure**:
  1. Read the task to confirm predictions will be scored against hidden truth by an error metric (i.e., accuracy matters, not just file format).
  2. Scan the scripts for any train/validation split, cross-validation, or metric call (MAE/RMSE/R²) on labeled data; check whether model choice, hyperparameters, and any ensemble weights are justified by such a number.
  3. Check the answer/report: does it quote a validation score, or only describe architecture and weights?
  4. If no validation score exists anywhere, flag: the agent had no evidence its chosen blend beats a simple single-model baseline.
- **Discriminator**: A real violation is zero out-of-sample estimate of error anywhere in the pipeline (weights/hyperparameters asserted, not measured). It is *not* a violation if the agent evaluated candidates on a holdout/CV, reported the score, and then optionally refit the winner on all data — even if the final model is simple.
- **Consequence**: The grader compares predictions to true labels under an error/agreement threshold; an unvalidated, arbitrarily weighted blend that includes badly-fit components misses the threshold and the file is marked WRONG despite correct shape and column name.

### Output row count / alignment with the evaluation input not verified
- **Applies when**: `task` -- The task asks for a per-row prediction (or per-row derived value) file produced from a given input table, so the deliverable's length and row order must correspond one-to-one with that input.
- **Pattern**: The attempt produces a prediction file whose number of rows does not equal the number of rows in the provided evaluation input (e.g., predictions generated from a truncated sample, a filtered/deduplicated subset, a head/`nrows` read, a partial chunk, or a leftover debug run), or whose rows are reordered/shuffled relative to the input. No script asserts `len(predictions) == len(input_df)` or preserves the original index order, and the reported answer is accepted without a shape check.
- **Detection procedure**:
  1. From the task/README, determine the expected number of prediction rows (row count of the evaluation input file) and the required column name/header.
  2. In the scripts, trace how the input is loaded and how predictions are written: look for `nrows=`, `.head()`, `.sample()`, `dropna()`, `drop_duplicates()`, filtering/merging, chunked loops, or sorting/shuffling between load and write; check whether the original row order/index is preserved.
  3. Count the data rows in the submitted file (excluding header) and compare to the expected count; also confirm the header text matches exactly.
  4. Confirm at least one explicit sanity assertion or printed shape check exists tying the output length to the input length; absence plus a mismatch is a violation.
- **Discriminator**: A real violation is a genuine length/order mismatch with the evaluation input (or an unverifiable output produced by code that could drop/reorder rows). It is *not* a violation if row filtering is explicitly required by the task, or if the count matches and order is provably preserved (index-aligned write) even though intermediate filtering happened on a copy.
- **Consequence**: The grader cannot align predictions to the ground-truth labels; the file is scored as wrong/missing (accuracy effectively 0) regardless of model quality.

### Missing or unverified required output artifacts specified by a config/spec file
- **Applies when**: `task` -- the task points to an external specification (e.g., a YAML/JSON config with plotting or output guidelines) and/or expects deliverables saved to disk rather than only printed in the chat answer.
- **Pattern**: The attempt reads (or paraphrases) the spec, hardcodes a few of its settings, writes only the single most obvious artifact (the image), and reports the numbers in prose — never enumerating the full set of files/keys the spec and task imply, and never re-loading the written files to confirm they exist with the right structure/values.
- **Detection procedure**:
  1. Read the task and open the referenced spec file; list every required deliverable (file names, extensions, save paths) and every constrained property (title text, labels, colors, ordering of categories, figure size, rounding, units, serialized data dumps).
  2. Read the scripts and build the set of files actually written and settings actually applied; diff against the list from step 1.
  3. Check whether the script contains a verification step that reloads each written artifact (e.g., re-open the image/array/JSON and assert shape, keys, category order, and value ranges).
  4. Check the final answer: does it claim completion based on prose description of the chart rather than on evidence that all named artifacts were saved and validated?
- **Discriminator**: A real violation is when a spec- or task-implied deliverable (an auxiliary data dump, a serialized array, a metadata JSON, a specific save path/filename) is absent from the scripts, or a constrained property is assumed rather than read from the spec. It is *not* a violation if the agent produced every file the spec names and merely used extra defaults for properties the spec leaves free, or if an unrequested extra file is also written.
- **Consequence**: The grader's per-file checks fail as WRONG/MISSING for every deliverable that was never created (and possibly for the one that was, if its category order/labels/colors deviate from the spec), yielding 0 passed checks even when the headline computed value is right.

### Unvalidated missing-value handling that silently changes the row set before a fixed-seed split
- **Applies when**: `task` -- the task prescribes an exact, reproducible split (fixed seed/proportions) and metric on data whose feature columns contain missing values, and the script preprocesses before splitting.
- **Pattern**: The attempt drops rows (or drops/reshapes columns) containing nulls, or imputes with an ad-hoc statistic, without checking or reporting how many rows survive; the resulting split — and therefore the metric — differs from the canonical one even though the seed matches. A related variant is failing to state how continuous model outputs are converted into class labels, so the accuracy definition itself is unverified.
- **Detection procedure**:
  1. From the task, list the mandated preprocessing/split/metric spec and note which steps are *not* specified (missing-value policy, thresholding rule) — these are the discretionary points that must be sanity-checked.
  2. In the script, find every operation that can change row count or label mapping (`dropna`, filtered joins, `fillna`, `astype`, threshold/rounding of predictions) and check whether the script prints shapes before/after and the resulting train/test sizes.
  3. Compare the reported number of rows entering the split against the raw row count; if they differ, the fixed seed no longer reproduces the intended split, and the attempt must justify the choice or test the alternative (impute vs. drop).
  4. Check the answer is derived from the test-subset predictions after an explicit label conversion, and that its value is reported in the requested rounding/format.
- **Discriminator**: A real violation is an undocumented row-count or label-mapping change with no shape/count sanity check and no comparison of the plausible alternative; it is fine if the script logs the pre/post row counts, keeps the full row set (imputation) so the seeded split matches the raw data, and explicitly defines the prediction-to-label rule.
- **Consequence**: The metric is computed on a different test subset (or with an unstated decision rule) than the reference, producing an accuracy off by a few hundredths — close enough to look plausible but graded wrong on exact-value comparison.

### Ignoring the provided output template's schema and semantics
- **Applies when**: `task` -- the task supplies a pre-existing answer/submission file whose existing columns, row labels, or category set define exactly what must be filled in.
- **Pattern**: The agent invents its own schema (its own category names, its own column meanings such as row indices instead of counts, extra/missing category rows) instead of reading the template file first and populating only its defined fields, then reports narrative numbers as if the deliverable were satisfied.
- **Detection procedure**:
  1. From the task, note that a specific output file/format is mandated and must be filled in "strictly".
  2. In the scripts, check whether the template file is actually read/inspected (headers, existing rows, allowed category labels) before writing, and whether the write preserves those headers and row keys.
  3. Compare the answer's category list and per-row values against the template's expected fields — do category names come from the template or from the agent's own derivation? Does each value type match the column meaning (e.g., a count vs. an identifier)?
  4. Flag if any template row/column is absent, renamed, reordered, or filled with a different quantity than its header implies (including catch-all buckets like "Unknown" that the template does not contain).
- **Discriminator**: Fine if the agent inspected the template and its output matches header names, row keys, ordering and value semantics exactly (extra explanatory prose outside the file is harmless); a violation is when the file's schema/labels/value meaning is agent-invented or deviates from the template, even if the underlying computation is defensible.
- **Consequence**: The result file fails exact-match checks on keys/values, so the graded check scores 0 regardless of how reasonable the reported narrative numbers look.

### Grouping-level (unit-of-analysis) error before computing a statistic
- **Applies when**: `task` -- the task asks for a statistic (correlation, mean, split by median, etc.) about *entities* while the raw data stores multiple time-stamped or repeated rows per entity.
- **Pattern**: The attempt computes the statistic directly on raw rows, or aggregates entities with an inconsistent recipe (e.g. duration derived from row counts instead of first/last timestamp difference, "maximum" taken after rather than before deduplication, the median split computed on row-level values instead of one value per entity), so the resulting r/p reflects the wrong sample size and a slightly different quantity than requested.
- **Detection procedure**:
  1. From the task wording, identify the entity key (the thing one data point should represent) and the per-entity quantities required (extremum, span/duration, magnitude used for the split).
  2. In the scripts, check that there is an explicit `groupby(entity_key).agg(...)` producing exactly one row per entity, and that each requested quantity is derived from that aggregation with the natural definition (max for extremum, last−first timestamp for span, per-entity total/scalar for the split variable).
  3. Verify the split threshold (e.g. median) is computed on the aggregated per-entity table, and print the number of entities in each subgroup; confirm it is plausible and that the two subgroups partition the entities.
  4. Compare the reported r/p against the aggregated-entity computation; if the script never collapsed to entity level, or duration/extremum was defined by a proxy, flag it.
- **Discriminator**: A genuine violation is when row-level or proxy-aggregated data feeds the statistic (entity counts differ from the number of unique keys, or duration comes from record counts/units other than those implied). It is fine if the data is already one row per entity, or if the aggregation is present and only the tie-breaking of the median split differs in a documented, defensible way.
- **Consequence**: The reported coefficient (and sometimes the relationship classification) drifts from the ground-truth value by more than the rounding tolerance, so numeric checks fail even when the qualitative conclusion happens to match.

### Unverified predictions with no held-out validation or reproducible script
- **Applies when**: `task` -- the deliverable is a prediction file produced by a model trained on a labeled training split and applied to an unlabeled test split.
- **Pattern**: The attempt fits a single model with default/ad-hoc hyperparameters, writes the output file, and reports only self-descriptive metadata (row counts, class balance, feature importances) without ever estimating out-of-sample performance on a held-out or cross-validated subset, without comparing against a baseline, and without preserving the script so the pipeline can be inspected — so systematic errors (mis-set decision threshold, class-weight over-prediction of the minority class, dropped/mis-encoded features, misaligned row order or ID mapping) go undetected.
- **Detection procedure**:
  1. Read the task to identify the required output file, column name, expected number of rows, and the label semantics.
  2. Inspect the scripts (or note their absence) for a validation step: a train/validation split or cross-validation with the target metric printed, plus a trivial baseline (e.g., majority class) for comparison.
  3. Check whether the reported answer cites any out-of-sample score and whether predicted class proportions are compared to the training label distribution and to the ID ordering of the test file.
  4. Flag if the only evidence of correctness is descriptive counts of the submission itself, or if no script exists to verify preprocessing consistency between train and test.
- **Discriminator**: A real violation has zero out-of-sample evidence (no CV/holdout score, no baseline, no distribution sanity check) and/or no inspectable pipeline; a look-alike that is fine reports a validated metric and shows train/test preprocessing were applied consistently, even if the final model is simple.
- **Consequence**: The submitted file may have the right shape and column name yet poor accuracy or a skewed positive rate, so the grader's accuracy/threshold check on the prediction column fails while the agent claims success.

### Deliverable file never written (or not matching the provided template schema)
- **Applies when**: `task` -- the task requires saving results to a specific output filename in the format of a provided sample/template file, and grading is done on that file rather than on chat text.
- **Pattern**: The agent computes a number and reports it only in its narrative answer (or writes it with ad-hoc column names/row layout, wrong path, or from an interactive session with no saved script), so no artifact at the required path reproduces the template's exact header, column order, and row count.
- **Detection procedure**:
  1. Read the task and note the exact required output filename, and locate the sample/template file it must mirror; record its header names, column order, and expected number of rows.
  2. Inspect the scripts for an explicit write call to that exact filename (relative to the expected working directory), and check whether the DataFrame constructed there uses the template's column names/order and index=False semantics.
  3. Compare the agent's reported answer to what the written file would contain; if there are no saved scripts at all, treat the artifact as unverifiable and flag it.
  4. Also confirm any stated reproducibility constraint (e.g., fixed random seed for resampling-based statistics) is set in the script that produces the file, so the saved value is regenerable.
- **Discriminator**: A real violation is the absence of a persisted file at the required name, or a file whose schema/values differ from the template (extra index column, renamed/reordered fields, missing rows, wrong directory). A look-alike that is fine: the script writes the correct file and the agent additionally echoes the same content in its message, with identical values and headers.
- **Consequence**: The grader looks for the named result file, finds it missing or schema-mismatched, and scores 0 even if the underlying statistic was computed correctly.

### Discarding a high-signal raw column (e.g., a timestamp/text field) instead of engineering features from it, then accepting a weak validation score
- **Applies when**: `task` -- the training table contains non-numeric or structured columns (dates/times, IDs, categoricals) alongside numeric ones, and the script builds its feature matrix by simply excluding anything that isn't directly usable by the estimator.
- **Pattern**: The script drops such columns with a blanket filter (`[c for c in cols if c not in ['date', target]]`), never derives features from them (hour/day-of-week/month, lags, encodings), evaluates with a random shuffled split even though the data is sequential, and then reports a mediocre validation score (e.g., R² well below what the column would provide) as success without comparing against a naive baseline or attempting improvement.
- **Detection procedure**:
  1. From the task and data exploration output, list all columns present and note which ones are dropped before fitting; check whether any dropped column plausibly carries strong signal (time index, group/ID, category).
  2. In the modeling script, verify whether any transformation of those columns is created; if not, confirm the model sees only the residual numeric block.
  3. Check the validation protocol: is the split random while the data has an inherent order/time dependence, and is the reported score compared to a baseline (mean predictor, simple time-based model) or to a plausible target quality level?
  4. Read the answer: does it claim success purely on the basis of an unbenchmarked score, and does it also contain internal inconsistencies (naming one model while fitting another)?
- **Discriminator**: A real violation is dropping a column that carries genuine predictive structure with no attempt to encode it and no baseline comparison. It is fine to drop a column that is provably uninformative (constant, pure noise, unique identifier with no ordering meaning) or absent from the test file — provided the script or answer states that check explicitly.
- **Consequence**: Predictions are far less accurate than achievable (compressed range, near-mean outputs), so the submitted file fails the grader's accuracy/similarity threshold even though the file name, column name, and row count look correct.

### Answer written to an ad-hoc file instead of the required deliverable
- **Applies when**: `task` -- the task (or its harness) specifies a result artifact with a given name/path/format, and the scripts end by printing or saving the computed answer.
- **Pattern**: The script computes a plausible answer but dumps it to a self-chosen filename/location (e.g. a scratch `.txt` in the working dir) or only prints it to stdout, never creating the exact expected result file with the exact requested JSON keys and value types.
- **Detection procedure**:
  1. From the task statement/harness, note the required output artifact: exact filename, directory, format, key names, and whether values are scalars or lists.
  2. Grep the scripts for every write operation (`open(...,'w')`, `to_csv`, `json.dump`) and record the paths and the object being serialized.
  3. Compare: does at least one write target the required path with the required schema? Check key spelling/case and that list-valued fields are lists, not bare strings.
  4. Confirm the submitted answer text is backed by that file, not merely echoed from console output.
- **Discriminator**: A real violation is when no write matches the required artifact (wrong name, wrong directory, wrong container type, or console-only). It is *not* a violation if the script writes the correct file and additionally logs or saves extra debug copies elsewhere.
- **Consequence**: The grader reports the expected result file as WRONG/MISSING and scores 0 even if the computed values would have been right.

### Deliverable file asserted rather than verified (summary in place of the requested artifact)
- **Applies when**: `task` -- The task requires writing results to a specific output file (e.g., a per-entity table of derived scores/segments/labels) and the agent's answer is a prose summary claiming the file was produced.
- **Pattern**: The attempt reports aggregate statistics and methodology text but never shows the script that writes the file, never re-reads the saved file, and never confirms its path, row count, key column, and the specific requested columns (scores, segment, level) — so a missing, misplaced, truncated, or wrongly-shaped artifact goes unnoticed.
- **Detection procedure**:
  1. From the task statement, list the exact deliverable: filename, expected granularity (one row per entity), and every field explicitly requested (each component metric, the composite score, the segment/label column).
  2. In the scripts, locate the write call: confirm it targets the exact required filename in the expected working directory, and that the DataFrame written contains the identifier plus all requested fields (not just a subset or an intermediate frame).
  3. Check for a post-write verification step (re-read the file; print shape, columns, head, and null counts) and compare those numbers to the counts quoted in the answer.
  4. In the answer, check that the reported entity count and segment counts are reconciled against the file's actual rows, and that the answer isn't purely narrative with no evidence of the file's contents.
- **Discriminator**: A real violation is when no script/verification evidence links the claimed numbers to an on-disk file with the required name and columns; it is fine if the agent shows the write plus a read-back with matching shape/columns even if the summary text is brief — and cosmetic differences (column ordering, extra helper columns) are not violations as long as all requested fields and one row per entity are present.
- **Consequence**: The grader marks the expected output file WRONG/MISSING (0 checks passed) regardless of how reasonable the described methodology sounds, because the artifact it inspects is absent or lacks the required per-entity segment/level columns.

### Hardcoded/invented input data instead of loading the provided files
- **Applies when**: `task` -- the task references supplied data files (and a sample output file defining the format), but the script embeds numeric arrays or literals in code instead of reading those files.
- **Pattern**: The agent types in values "from memory" (or a truncated subset) and computes statistics on them, never opening the actual dataset or the format-template file, so both the numbers and the output schema are unverified against ground truth.
- **Detection procedure**:
  1. In the task/README, list every input artifact that should be read (data files, sample/template output file).
  2. Scan the script for `read_csv`/`load`/file paths: check whether each listed artifact is actually opened, or whether the data appears as inline literals/dicts.
  3. Compare the script's implied row counts and value ranges to whatever the task or data documentation states; inline data that is short, suspiciously round, or of unstated provenance is a red flag.
  4. Check that the output columns/order/header come from reading the sample file rather than being guessed in code.
- **Discriminator**: A real violation is when the analysis' *primary* inputs or the output schema are invented rather than loaded; it is fine to hardcode genuine constants (bootstrap size, seed, confidence level) or to inline a tiny lookup table that the task itself specifies verbatim.
- **Consequence**: Statistics are computed on the wrong sample (and possibly the wrong columns/format), so the written result file mismatches expected values and the file check fails even though the code runs cleanly.