What to do
- Record the dataset, harness, agent and model versions.
- Run shortlisted agents against the same tasks and budgets.
- Compare results alongside cost and failed tasks.
Why the version matters
The May 6 release repaired 28 of 89 tasks. In the comparison published by the benchmark authors, Claude Code with Opus 4.6 gained 12.1 percentage points. Several other pairs gained less or declined slightly.
A leaderboard without a version is a stopwatch without a start button.
Primary source and results by configuration