SkufosaurRU
Menu

Recipe / Evaluation

Decide what counts as a pass before optimising

Acceptance checks for search, extraction and code changes.

90%

Acceptance checks for search, extraction and code changes.

Sections and scope

A comparison of agent workflows with a clearly bounded task.

These are editorial examples. Adapt the criteria to your domain and risk.

Define a result you can check

Specify what should be present, what must survive the change and what evidence will verify it. “Looks good” is difficult to compare.

Task Acceptance criterion Check
Find a record Return the known record and report a miss accurately One known hit and one known miss
Extract fields Match the source and identify missing values A document with one empty and one ambiguous field
Fix search Reproduce the fix without changing the sort order The failing query and a sort regression check

Write the task card

Record the goal, required constraints, allowed inputs, expected output, ordinary and edge cases, missing-data cases and what is outside the test.

Compare fairly

Use the same card for both versions. Check whether the result is usable before comparing time and cost. Save failure reasons. If another model judges the result, give it the rubric and have a human check disputed decisions.

Add newly discovered failures to the suite and keep the old cases. When the business requirement changes, version the test before comparing again.

Sources and origin