Tested against
the agent it replaces.

Success: an attempt succeeds only when every required Outcome, Grounding, Rules, and mechanical check passes on a valid scored attempt. Pass³ is the share of cases that succeed on all three attempts, counted directly rather than cubed from the attempt rate. An inconclusive attempt, where the simulated user was invalid, counts as not passed.

Design: old and new runs alternated on one day, five of each with the old agent first in each pair, under the same approved evaluator calibration. Each holdout run covers all 25 cases three times. One training run per agent covers the 50 training cases three times.

Uncertainty: the page reports every run and its range. Only aggregate scores leave the holdout store, so this entry publishes no case-level bootstrap interval.

The production agent: v1.1.2’s prompt and tools ran on the harness commit that introduced 2.5.19. Its prompt, agent and tool code match v1.1.2 byte for byte, except a version label that reads 2.3.16. Earlier holdout runs of the same agent, on October 10 under harness 2.5.18, scored 76%, 60% and 48%.

Provenance: the candidate was measured at commit 532f9fed. Between the measured commit and v1.2.0, only the RDS trust-bundle pin and this scorecard changed; no prompt, tool or agent code. The measurement cost $4.77 across all model roles.

Success: all required Outcome, Grounding, Rules, and mechanical checks must pass on a valid scored attempt. Headline success uses all 300 planned slots. The baseline’s one inconclusive receives no success credit; its scored-only rate is 257/299 (86.0%), distinct from the headline 257/300 (85.7%). Nothing was dropped or replaced.

Uncertainty: descriptive 95% percentile intervals use 10,000 bootstrap draws, seed 360, resampling the 99 scenario groups in sorted order. Linked cases and all three attempts stay together. The change interval resamples the same groups across baseline and final, compares their success rates using each draw’s slot count, and does not treat attempt numbers as paired random seeds. The baseline interval is supplemental: its canonical runner interval is null because of the inconclusive. The final success interval matches the recorded runner result.

Consistency and dispersion: cases passing all three attempts are counted directly. Their intervals preserve the same scenario groups and use each draw’s case count. SD is the sample standard deviation of the three repetition success rates, in percentage points. These repetitions share cases and unseeded model sampling; a narrow observed spread is not a guarantee.

What changed: an explicit cancellation instruction stops clarification and pricing after cancellation. Directed entry-to-exit relationships in the location catalog, with selection guidance, help choose direction variants together. The combined candidate’s aggregate gain does not prove which edit caused which change.

Limits: these are exposed development cases under frozen tool replay, after candidate search on the same set. They do not estimate unseen-production accuracy, execute live pricing, or establish production qualification. Recorded grades are preserved, including known grading limitations. The selected search run is subject to selection bias and is not pooled with final verification.

Provenance: final verification evaluated commit 46476e87. It recorded prompt metadata 2.3.14 with the candidate SOP; publication advanced only that version literal to 2.3.15. All 100 rendered prompt hashes, tool schemas, model settings, and measurement inputs matched. The normal development workflow still defaults to one repetition. No holdout was evaluated for this entry.