experiment · growing
Evals before prompts
Twenty cases and a diff beat any amount of prompt intuition.
Prompt tuning without evals is just vibes with extra steps. The eval does not need to be sophisticated; it needs to exist before the prompt changes.
20 casesreal inputs
Runfixed seed
Graderubric or assert
Diffvs last run
Starting set
| Kind | Count | What it catches |
|---|---|---|
| Golden | 10 | regressions on the happy path |
| Adversarial | 5 | prompt injection, refusal bait |
| Boring | 5 | empty input, huge input, wrong language |
Grading with a model
Use one when the answer is genuinely open-ended, and then:
- Grade one dimension at a time. A single "is this good" score is noise.
- Give the judge the rubric and a reference answer.
- Track judge/human agreement on a sample. A judge you have never audited is a random number generator with good manners.