Lessons

27 lessons are ready.

  1. Why evals decide whether AI products live
  2. The loop runs Analyze, Measure, Improve — in that order
  3. Anatomy of an evaluation: five parts, one decision
  4. Failure taxonomy v0 and the first end-to-end baseline
  5. A trace schema and open coding of the support traces
  6. Axial coding: from nine open codes to a taxonomy you can plan from
  7. Notebooks lie, dashboards lie: runners, logs, exit codes
  8. Online monitoring, honestly scoped
  9. Saturation and the Pareto priority rank
  10. Sourcing cases that matter
  11. Synthetic generation and its traps
  12. Versioning, splits and the leakage linter
  13. The dataset review lab and the refresh cadence
  14. The negative class: what a passing case proves
  15. Choosing the grader: the decision ladder
  16. Assertions that read the right field
  17. Reference graders and the join that decides them
  18. The suite, and the pass it will not confirm
  19. When to judge, and when judging is malpractice
  20. Build a judge: one criterion, evidence first, binary verdict
  21. Calibrate the judge: confusion matrix, TPR and TNR above 80 per cent
  22. The bias lab: position, verbosity and self-preference
  23. Judge economics: sampling, cascades, drift
  24. The judge panel: agreement, the decision rule and the ticket
  25. Retrieval metrics in the LLM era
  26. Answer-level evals: faithfulness, groundedness, relevance
  27. End-to-end RAG harm checks

Coming soon

28 lessons are planned and not published yet. This course is still being built. The lessons that are ready stay open; the rest appear as they ship.

  • Coming soonOnline RAG signals & feedback mining
  • Coming soonConfidence intervals for people who ship
  • Coming soonSample size & power on an eval budget
  • Coming soonAgreement I: percent agreement, Cohen's κ, and the paradox
  • Coming soonAgreement II: Gwet's AC1 & bootstrap confidence intervals
  • Coming soonMultiple comparisons & sequential peeking
  • Coming soonAnatomy of a trajectory
  • Coming soonStep graders & GPA alignment
  • Coming soonOutcome + efficiency: state, cost, latency
  • Coming soonMulti-turn & simulated users
  • Coming soonReliability dynamics: pass@k vs pass^k
  • Coming soonDesigning human eval: guidelines, gold, adjudication
  • Coming soonInter-annotator agreement as a product metric
  • Coming soonPreference data & arena-style evals
  • Coming soonThe human eval budget: sampling for signal per dollar
  • Coming soonCI gates with zero-cost cassettes
  • Coming soonThe trace-to-dataset flywheel
  • Coming soonBudgets: cost & latency as eval dimensions
  • Coming soonReading production drift
  • Coming soonFrom threat model to eval suite
  • Coming soonJailbreak & prompt-injection suites
  • Coming soonGated release & sign-off
  • Coming soonContinuous red-teaming
  • Coming soonBenchmark-style scoring: FAIL_TO_PASS / PASS_TO_PASS
  • Coming soonEvaluating the harness: ablation as the method
  • Coming soonTrajectory scoring for code: process, not just patches
  • Coming soonBuilding task suites from a repo's git history
  • Coming soonSandboxed runners & CI regression suites for agents