The eval that lies

Trace-detective lab (Module 8, L8.1): an eval measures what its construction allows, not what its author intended. Two scripted agents on 12 templated tasks: on verbatim tasks the solver and the memorizer both score 12/12 -- THE LIE; with fresh inputs per run the solver stays 12/12 and the memorizer drops to 0/12. Three lying evals graded behaviourally -- training-data overlap (canary grep 1 hit), prose graded by prose (judge 9.2 while executed artifacts pass 6/12), and test-set access (3 reads of tests_hidden/). Student names each channel and its fix, plus the claim. In-memory only; scripted agents, no model or provider call.

This Harness Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.