The bias lab: position, verbosity and self-preference

Bias lab: L5.3 calibrated a pointwise judge and cleared both floors, so this lesson changes the instrument to the pairwise one and finds the thing that breaks is not accuracy. Ten probe pairs over the same five payloads: the inherited judge names the payload printed first on 14 of 20 decided probes (70.0%) and 4 of 10 verdicts flip when the positions move — yet the graded score reads 5 of 6 either way, because b06 is lost and b07 gained. Only 4 of 6 graded probes survive reciprocal permutation, all 4 right. A neutral tail of characters and no digits on the second side takes the score from 5 of 6 to 4 of 6 and flips 6 verdicts. An own-family label moves two graded verdicts in opposite directions, so the accuracy is identical and the error moves from b07 to b01. Evidence first answers 6 of 6, abstains on all 4 pairs with no winner, and flips 0 of 10.

This Evals Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.