Notebooks lie, dashboards lie: runners, logs, exit codes

Runner lab: four runs over L2.1's same 50 cases, one dataset and one seed, print 44.0% / 32.0% / 50.0% / 40.7% with no trace row changed. The naive `rate >= 0.40` rule passes three of them; the coverage contract passes one and withholds the two that graded 20 and 27 of their 50 promised cases. Comparing run B with run C reads as +18 points and 6 regressions to a case-id join, and as 30 unmeasured cases and 0 real flips to a differ that checks coverage first. Of the 34 failing rows, 15 miss exactly one check — 9 reply-only, 4 outcome-only, 2 redaction-only — so the failure count is a property of how many checks ran.

This Evals Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.