Why evals decide whether AI products live

Curriculum Module 1 / Lesson 1.1. Lab shape: trace-detective. Runtime: in-browser Python (Pyodide — a Python runtime that runs in the browser) and local starter workspace (courses/evals-engineering/).

Watch it fail

A billing team launches an AI support assistant that answers refund requests. The policy has two clauses:

Standard plans qualify for a full refund only within 30 calendar days of purchase.
Enterprise plans also qualify after 30 days when there is a verified service-level agreement (SLA) breach (a documented outage violating contracted uptime).

Before launch, an engineer pastes five favourite test prompts into a chat window. Four of the five look great, one is borderline, and the team records a 95% pass rate. Look at what happens over the next six weeks as routine improvements ship:

Week 0 — Launch. On real customer traffic, the assistant follows policy on 94% of tickets. The five-prompt eyeball check reads 95%. [ILLUSTRATION — simplified]
Week 1 — Tone tweak. A product manager asks the team to make the assistant sound warmer. An engineer adds "Always prioritize customer delight and resolve frustration generously" to the system prompt, re-runs the five favourite prompts (95%), and merges. On live tickets, frustrated customers at day 45 start receiving unauthorized $240 refunds. True policy compliance drops from 94% to 82% — a 13-percentage-point blindness gap. [ILLUSTRATION — simplified]
Week 2 — Retrieval chunk resize. To cut latency, another engineer halves document chunk size from 800 tokens to 400 tokens. The Enterprise SLA exception clause (Section 4.2) splits into a second chunk that falls out of top-1 retrieval. Enterprise customers at day 38 with confirmed outages are now falsely denied. True compliance drops to 71%; the five-prompt check reads 94%. [ILLUSTRATION — simplified]
Week 4 — Minor model version swap. A routine model checkpoint update slightly shifts how date arguments are formatted in tool calls. True compliance drops to 64%; the five-prompt check still reads 94%. [ILLUSTRATION — simplified]
Week 6 — Finance audit. Accounting closes the quarter and finds that 36% of audited refund replies violated policy (64% true compliance). Because nobody measured which change broke which slice, leadership freezes all prompt and retrieval deploys. [ILLUSTRATION — simplified]

Before you scroll: why did the five-prompt check stay at 94–95% while one in three real tickets failed? Write one sentence.

(This six-week trajectory is a synthetic teaching fixture drawn from recurring production failure patterns. Every number is locked in the lesson fixture and computed in the practice cell.)

The mental model

An eval is not a public leaderboard score or a five-prompt vibe check. It is a measurement system with an owner, a budget, and a concrete decision it feeds.

Compare the three ways teams judge an AI system:

Approach	What it actually tests	Decision it can honestly support
Vibes (eyeball testing)	3–5 prompts the author remembers to paste into chat	None — creates prompt astrology (debating adjectives in pull requests without data)
Public benchmark	Someone else's static exam on general capability	Narrowing 20 candidate base models down to 2–3 before you test on your own data
Product evaluation system	Your task specification, your failure slices, your binary pass/fail graders	Block or merge a pull request; page on-call or stay quiet; promote or roll back a release

Interview sentence: Brakes do not exist to make a car go slow — they exist so you can drive fast without going off a cliff.

The mechanism: the three loops

No single test suite catches every failure. A working evaluation system closes three distinct loops, each catching what the others miss:

Loop	When it runs	What it inspects	What it catches	Decision it gates
1. Offline loop (`offline`)	Pre-merge (every pull request or prompt/model edit)	Curated golden dataset + past production regressions	Regressions on known edge cases before a single user sees the change	Merge or block the pull request
2. Online loop (`online`)	In production (live traffic)	Structured traces, synchronous guardrails, sampled async graders	Unknown live drift: new customer phrasing, broken upstream APIs, distribution shift	Block an unsafe reply in flight; page on-call or roll back a canary
3. Human loop (`human`)	Weekly calibration & trace review	30–50 raw production traces read by a human with binary pass/fail labels	Grader blindness: when your automated grader says 97% pass while users are churning	Recalibrate the automated grader and promote new failures into the offline golden set

Notice two design rules that govern all three loops across this entire course:

Binary verdicts (PASS / FAIL), never 1–5 Likert scales. Asking a reviewer or an automated judge to rate a response "3 out of 5 vs 4 out of 5 on helpfulness" produces noisy averages that hide policy violations. Instead, decompose quality into crisp binary checks: Did the assistant issue a refund past 30 days on a Standard plan? (True / False).
A metric without a decision rule is dashboard theater. If a score can drop 15 percentage points and the pull request still merges automatically, you do not have an evaluation gate — you have a chart.

Inspecting raw traces & the starter workspace

Open the interactive Local Offline Trace Explorer (no API key, zero network calls). Three multi-span traces from the Billing Support Assistant (trace_w0_001, trace_w1_002, trace_w2_003) are pre-loaded:

Click trace_w1_002 (Week 1 warm-tone regression) and expand its five spans: root → retrieval → tool_call → completion → parse. Notice that retrieval fetched the exact 30-day rule and tool_call returned days_since_purchase: 45, yet completion overrode the boundary and parse executed a $240 refund.
Click trace_w2_003 (Week 2 chunker regression). Here the model followed the retrieved text faithfully — the bug sits inside the retrieval span, where the 400-token chunk split dropped Section 4.2.

In your local starter workspace (courses/evals-engineering/), run the offline smoke suite against the recorded YAML cassette ($0, outbound sockets blocked by default):

cd courses/evals-engineering
pytest modules/m01_eval_mindset/l01_1_why_evals/ -k "not student" -v

The course map: what you will assemble

Across 12 modules, you will build one complete evaluation harness layer by layer:

Modules 1–3 (Foundations & Data): the three-loop mindset, open coding 30–50 raw traces into a grounded failure taxonomy, and constructing a coverage-matrix golden dataset (datasets/v1/).
Modules 4–5 (Graders & Calibrated Judges): fast deterministic code/schema graders first, followed by bias-tested binary automated judges calibrated against human labels with TPR/TNR correction.
Modules 6–9 (Architecture & Stress): component-level diagnostics for retrieval (2x2 context vs faithfulness), tool use and multi-step agent trajectories, multi-turn memory probes, and adversarial safety/robustness suites.
Modules 10–12 (Decisions, CI/CD & Capstone): bootstrap confidence intervals, McNemar paired tests, two-tier CI gates, live production guardrails, and your end-to-end Capstone scorecard and ship/no-ship release memo.

The failure mode

Treating one loop as the whole system:

Offline-only ("Pre-Merge Bubble"): 100% green CI while an upstream schema change breaks live production.
Online-only ("Live Firehose"): shipping unvetted prompt edits straight to customers and discovering regressions days later from angry tickets.
Uncalibrated automation ("Autopilot Lie"): trusting an automated grader for months without checking whether its binary verdicts still agree with a human reading raw traces.

The lab

Open the practice cell below (or courses/evals-engineering/modules/m01_eval_mindset/l01_1_why_evals/exercise.py locally). Five fictional team setups (A–E) and the six-week billing regression timeline are loaded. Every diagnosis slot starts blank (None):

Replace each None in diagnoses ("A" through "E") with the matching tag from DIAGNOSIS_TAGS:
- "missing_offline_loop"
- "missing_online_loop"
- "missing_human_loop"
- "no_decision_rule"
- "complete_three_loop_system"
Replace first_broken_week = None with the integer week where the blindness gap (perceived_pass_pct - true_pass_pct) first exceeds 10.0 percentage points.

Run the cell (or pytest modules/m01_eval_mindset/l01_1_why_evals/). Every wrong pick prints a specific diagnostic hint until all checks pass. Practice seed is locked at 0. No API key.

One-line close

If a metric cannot block a bad change or trigger a rollback, it is decoration — not an evaluation.

What breaks next: once a team agrees it needs all three loops, they almost always build them in the wrong order — writing 200 synthetic automated tests before looking at a single real production failure (Lesson 1.2: The Analyze–Measure–Improve lifecycle).

Home numbers / honesty table

Item	Value	Label
Six-week perceived pass rates (Weeks 0, 1, 2, 4, 6)	`95%, 95%, 94%, 94%, 94%`	ILLUSTRATION fixture
Six-week true pass rates (Weeks 0, 1, 2, 4, 6)	`94%, 82%, 71%, 64%, 64%`	ILLUSTRATION fixture
Blindness gap (`perceived - true`)	`1%, 13%, 23%, 30%, 30%`	REAL: computed in cell & `prove-home-numbers.py`
First week with gap `> 10%`	`Week 1` (`13%` gap)	REAL: computed in cell & `test_exercise.py`
Fixture trace count (`sample_support_trace.jsonl`)	`3` traces, `5` spans each	ILLUSTRATION fixture
Cassette replay cost	`$0.00` (sockets blocked in `conftest.py`)	REAL: verified by `test_offline_socket_guard_blocks_live_network`
Lesson video	not shipped	Manim script in `docs/evals-engineering/curriculum/modules/m01/media/l1-1-three-loops.py`

Video

No rendered lesson video is shipped in this build. A Manim script for this lesson exists in the curriculum docs under the repo (docs/evals-engineering/curriculum/modules/m01/media/l1-1-three-loops.py); it is a plan for a future render, not a playable file linked from this page.