Online monitoring, honestly scoped

Monitoring lab: a generated three-day stream of 2400 events and 106 failures, sampled at 120 of 800 a day. Error-biased tail sampling keeps 106 of 106 failures at 29.4% precision; uniform head sampling keeps 13 (12.3%) and "the first N of the day" keeps 8 (7.5%). At 40 a day the same tail policy overruns by 34 rather than dropping an incident. 20 of the 106 failures fire no signal at all, all from the two mechanisms the ledger check cannot see, so no rate on the page includes them. One alert rule detects 4 of 4 post-deploy blocks under tail sampling, 0 under head sampling at any threshold above two, and fires once on background noise under first-N — four blocks before the deploy.

This Evals Engineering lesson is part of a paid plan. Sign in with a plan that includes this course to read it.