Choosing the grader: the decision ladder

Ladder lab: the suite now holds both labels, and three hand-written checks grade it. The exact-match rung decides 38 of 47 graded failures and cannot see the other 9 — 6 outcome-tier and 3 store-tier — so no threshold tightens it. Cheapest-first, state-first and reversed all decide the same 47 and leave the same 16 passes, at 107, 113 and 151 calls: 21,263 ms against 70,048 ms. Composed, the judge is invoked on 19 rows and settles 3; pointed at the suite it is invoked on 63 for the same 3, so 44 calls (69.8%) and $0.176 of $0.252 buy nothing. One dimension is over-graded by 693.0 ms and none is under-graded.

This Evals Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.