AI Eval Design: A Worked Example

Section 3.2 introduced the minimal eval; here is the complete recipe as a worked example for a customer-support drafting assistant. The same skeleton adapts to any text-producing use case.

Step 1: Collect. Pull 50 real, recent tickets, stratified deliberately: roughly 30 routine, 10 hard (multi-issue, angry customer, policy edge case), 10 that should not be answered by AI at all (legal threats, safety issues, VIP escalations). The third group tests the most important behavior: knowing when to hand off.

Step 2: Define pass/fail per dimension. A draft passes only if it clears all five: Factual, every claim about policy or the customer's account is correct against source documents; Complete, addresses every issue raised in the ticket, not just the first; Safe, refuses or escalates the ten handoff cases; On-brand, tone a supervisor would send unedited; Grounded, cites or quotes the policy passage it relied on. Binary per dimension beats 1-10 scores: graders agree on pass/fail far more than on the difference between a 6 and a 7.

Step 3: Grade. First pass: an LLM judge applies the rubric to all 50 (cheap, repeatable). Second pass: a human, the support lead, not the project team, audits 15 of the judge's grades weekly, including every failure. If human and judge disagree more than ~10% of the time, fix the rubric before trusting the judge.

Step 4: Use. Every change, prompt wording, retrieval settings, model version, runs the full set before shipping. Track the pass rate on a visible chart. The eval is also your regression alarm for vendor model upgrades (Section 5.5) and your evidence pack for the governance review (A.4).

Cost reality check: building this eval takes the support lead about a day, and running it costs under a dollar in tokens. It is the highest-return day in the entire project.