Evaluation explorer · Loading real scheduled results…

Evaluation explorer

Good answers.
Visible evidence.

Scheduled conversations test TollChat against real road conditions. Inspect the answers, pricing evidence, and checks behind every result.

The latest picture.

Six selected scenarios, scheduled for the road conditions they exercise.

Live scheduled evaluations
No snapshot loaded
Last seven days · oldest first
PassedFailedExecution error

What we put to the test

Open a result to see the conversation, evidence, and verdicts.

6 scenarios
28 scheduled runs / week

Recent runs

A run passes only when all three checks pass.

Rolling seven-day history · All times Eastern

Behind the results

What these checks
can tell you

Evidence, not a general accuracy claim

These six scenarios cover selected current-toll conversations. They do not measure every route, annual commute estimates, or every possible user request. An honest “current pricing is unavailable” can be a correct answer.

Model judgment + an exact rule

AI judges assess completeness and factual correctness against the tool evidence. Code counts the pricing lookups. Judges can make mistakes, so individual examples remain inspectable.

How scheduling and grading work

One scenario runs per scheduled window, with 28 runs each week. Weekday and Saturday northbound windows use different scenarios. Their latest results will naturally have different timestamps; a scenario awaiting its first scheduled run has no score.

Strands ActorSimulator supplies follow-up messages, with at most three assistant turns. Recorded evidence identifies the models, settings, and grading policy used for each run. Completeness uses Strands GoalSuccessRateEvaluator; correctness uses CorrectnessEvaluator with the conversation and tool evidence. The current policy fails only on material errors or an unmet user goal. A deterministic check requires pricing evidence when needed; a simple confirmation does not need another lookup.

Current-policy scores count runs without critical failures divided by completed, graded runs under that policy. Historical grades are preserved; compare results within the same policy. Execution errors are shown separately and are not counted as passes or graded failures. Missing or unscored results never become a passing score. This page shows the environment named at the top. Results refresh every minute while the page is visible.