Define a result you can check
Specify what should be present, what must survive the change and what evidence will verify it. “Looks good” is difficult to compare.
| Task | Acceptance criterion | Check |
|---|---|---|
| Find a record | Return the known record and report a miss accurately | One known hit and one known miss |
| Extract fields | Match the source and identify missing values | A document with one empty and one ambiguous field |
| Fix search | Reproduce the fix without changing the sort order | The failing query and a sort regression check |
Write the task card
Record the goal, required constraints, allowed inputs, expected output, ordinary and edge cases, missing-data cases and what is outside the test.
Compare fairly
Use the same card for both versions. Check whether the result is usable before comparing time and cost. Save failure reasons. If another model judges the result, give it the rubric and have a human check disputed decisions.
Add newly discovered failures to the suite and keep the old cases. When the business requirement changes, version the test before comparing again.
Sources and origin
- OpenAI · Evaluation best practices
Primary guidance on task-specific evaluation and coverage. The examples here are synthetic.
Source scope and limitations →