SkufosaurRU
Menu

Recipe / Evaluation

Compare two approaches on the same tasks

A repeatable baseline-versus-candidate protocol for quality, time, usage and cost.

90%

A repeatable baseline-versus-candidate protocol for quality, time, usage and cost.

Sections and scope

Local experiments using synthetic or authorised anonymised tasks.

This is a recipe. Sample size and acceptance criteria depend on the task.

Set up the comparison

Choose one idea. Name the baseline and the candidate, then define what counts as a usable result before looking at numbers. Record the tasks, environment and known model versions, tool access, limits and quality criteria. Use synthetic edge cases and exclude private prompts, credentials and account data.

Run the pair

  1. Run the baseline and save the output and available measurements.
  2. Change the one approach you want to test.
  3. Run the candidate under comparable conditions.
  4. Apply the same acceptance criterion to both.
  5. Repeat the pair and report variation.
  6. Include failed attempts and supporting work.

Keep the report auditable

Record the goal, versions, limits, result and reason for acceptance or rejection. Show time to a usable result, usage with source and coverage, and any cost estimate as a separate number. List all runs, including failures, and explain what the test does not cover.

Sources and origin

  • Skufosaur Master Codex v5.3

    Internal editorial method. It describes how to compare workflows and reports no measured result.