Evaluation explorer
Good answers.
Visible evidence.
Scheduled conversations test TollChat against real road conditions. Inspect the answers, pricing evidence, and checks behind every result.
The latest picture.
Six selected scenarios, scheduled for the road conditions they exercise.
No snapshot loaded
What we put to the test
Open a result to see the conversation, evidence, and verdicts.
28 scheduled runs / week
Recent runs
A run passes only when all three checks pass.
Behind the results
What these checks
can tell you
Evidence, not a general accuracy claim
These six scenarios cover selected current-toll conversations. They do not measure every route, annual commute estimates, or every possible user request. An honest “current pricing is unavailable” can be a correct answer.
Model judgment + an exact rule
AI judges assess completeness and factual correctness against the tool evidence. Code counts the pricing lookups. Judges can make mistakes, so individual examples remain inspectable.
How scheduling and grading work
One scenario runs per scheduled window, with 28 runs each week. Weekday and Saturday northbound windows use different scenarios. Their latest results will naturally have different timestamps; a scenario awaiting its first scheduled run has no score.
Strands ActorSimulator supplies follow-up messages, with at most three assistant turns. Recorded evidence identifies the models, settings, and grading policy used for each run. Completeness uses Strands GoalSuccessRateEvaluator; correctness uses CorrectnessEvaluator with the conversation and tool evidence. The current policy fails only on material errors or an unmet user goal. A deterministic check requires pricing evidence when needed; a simple confirmation does not need another lookup.
Current-policy scores count runs without critical failures divided by completed, graded runs under that policy. Historical grades are preserved; compare results within the same policy. Execution errors are shown separately and are not counted as passes or graded failures. Missing or unscored results never become a passing score. This page shows the environment named at the top. Results refresh every minute while the page is visible.