Set up the comparison
Choose one idea. Name the baseline and the candidate, then define what counts as a usable result before looking at numbers. Record the tasks, environment and known model versions, tool access, limits and quality criteria. Use synthetic edge cases and exclude private prompts, credentials and account data.
Run the pair
- Run the baseline and save the output and available measurements.
- Change the one approach you want to test.
- Run the candidate under comparable conditions.
- Apply the same acceptance criterion to both.
- Repeat the pair and report variation.
- Include failed attempts and supporting work.
Keep the report auditable
Record the goal, versions, limits, result and reason for acceptance or rejection. Show time to a usable result, usage with source and coverage, and any cost estimate as a separate number. List all runs, including failures, and explain what the test does not cover.
Sources and origin
- Skufosaur Master Codex v5.3
Internal editorial method. It describes how to compare workflows and reports no measured result.