1 min read
Sample post — placeholder content
Notes on evaluating LLM outputs
LLM-as-judge is cheap and useful — as long as you keep checking the judge. A few practices that kept my evals honest.
#evals#llm#testing
You can't improve what you don't measure, and with LLMs you can't even *keep* what you don't measure — a prompt tweak that fixes one case quietly breaks three others. So every project now starts with an eval suite, not a prompt.
Calibrate the judge
LLM-as-judge scales, but the judge drifts like any model. The practice that kept it honest: hand-label a small golden set, and score the judge against it whenever the judge's prompt or model changes.
textsnippet
suite: support-bot v14
faithfulness 0.94 (+0.03)
tone 0.88 (−0.01)
judge-vs-human agreement: 0.91 ✓The diff view matters more than the score: seeing exactly which cases regressed turns a vague 'the model got worse' into a fixable bug report.