Agent Evals — Prove It Got Better
Operationalizes the measurement of an agent: whether a change made it better, and whether anyone would find out if it made it worse. The governing idea: do not accept an agent improvement backed by three manual examples — demand a representative suite and a comparison against a stated baseline. This is the discipline traditional testing already applies to deterministic code, applied to non-deterministic, judgment-laden output. Structurally almost everything transfers — fixed inputs, a runner, CI, red-before-green, coverage as a question worth asking — and exactly one thing does not: the assertion. A deterministic system lets you compare exact values; a judgment-laden one needs a grader that tolerates variation and still says something falsifiable. Everything hard about evals lives in that substitution, and everything else is a solved problem being re-solved badly.
Principios
- · Aplica los principios documentados en el pack agent-evals.
Patrones
- · Start from failures, not features. The first twenty cases should be things you have actually seen go wrong, not a tour of the happy path. A suite built from the spec measures the spec.
- · Twenty imperfect cases beat two perfect ones, and beat zero indefinitely. The suite that exists and grows is worth more than the representative corpus that is still being designed.
- · Write the case before the fix. A production failure becomes a red case first, then gets fixed — the same discipline as a regression test, for the same reason.
- · If a case took an hour to write, it is probably an end-to-end case, and that's fine. Expect the altitudes to cost different amounts and budget accordingly.
- · Fixed, representative, versioned — say which one you're compromising. Most real suites compromise representativeness. Saying so out loud is what keeps the number honest.
Riesgos
- · Ver anti-patrones específicos del dominio en el pack.
Checklist
- [ ] ★ History check: no baseline was updated in the same commit or the same command as a run that failed against it. (AEE54)
- [ ] ★ The harness offers no flag, script, or target that re-records a baseline as a side effect of running the suite. (AEE55)
- [ ] ★ Every case has been verified by deliberately breaking the behavior it tests and observing it go red; any case never seen red is marked unverified. (AEE70)
- [ ] ★ No grader is the same model instance, in the same context, that produced the output under test. (AEE24)
- [ ] ★ Every LLM-as-judge grader that gates anything has been calibrated against human-labeled cases, with the agreement rate recorded, at its current model version. (AEE22, AEE23)
- [ ] ★ Eval cases run against a clean context — no carried session state, no memory files, no leakage from an earlier case in the same run. (AEE64)
Fuentes
En la bibliografía AKOS
Este pack cita las siguientes fuentes registradas en el índice bibliográfico:
Explorar todas las fuentes →