Test Runs
A test run is created when you execute a test set against an endpoint. It captures all individual test results with detailed metrics, conversation history, execution metadata, and performance data for analysis.
Test Runs List
Use the list to compare status, pass rate, executor, and annotation activity across runs. The Annotations column shows how many tests carry a human verdict. Hover the value to see how many of those were corrected, meaning the human verdict changed the automated result.
Open the filter drawer and set Annotations to With or Without to include only runs that have annotations or only runs that do not.
By clicking on a test run, you’ll see the overview of how the test run went. From here you can:
- Annotate: Record a human verdict over the automated test evaluation
- Compare: Compare against a baseline test run
- Re-run: Execute the test set again with the same configuration
- Download: Export test run data as CSV
Test Run Detail View
The test run detail page includes an expanded Overview and Conversation workflow for debugging failures faster:
- Go to Test: Open the original test definition directly from a result detail
- Context panel: Expand/collapse captured context items when available
- Metadata panel: Expand/collapse structured metadata for the result
- Files / Output Files panels: Inspect and download both input attachments and generated output files
- JSON-aware rendering: Response content that is valid JSON is pretty-printed in the overview tab
For multi-turn tests, the Conversation tab supports trace-driven debugging:
- Click assistant responses to open the trace drawer
- Navigate turn-by-turn while keeping test result context visible
Parameters Used
When a test run targets a parameter experiment, it snapshots the experiment’s resolved values at queue time. This guarantees that test runs remain reproducible even if the experiment or environment changes later.
The Test Run detail page displays an Experiment card showing the experiment name, version (e.g., v3), the source environment (if any), and an expandable Resolved Values panel. (Values for secret_ref slots are masked so credentials never leak into the test run record.)
Because Test Results inherit this snapshot from their parent test run, there is no need to track configurations on a per-result basis. The test-runs list page also includes an Experiment filter to help you slice historical results by configuration.
Version Information
A test run also records the version of the application it tested, taken from the endpoint’s Version Information or reported by the endpoint in its own response.
The Configuration tab shows the recorded JSON and notes which of the two it came from. The test runs list carries a Version Information column for comparing runs at a glance. Like the experiment snapshot above, the version is captured when the run is created, so editing the endpoint afterwards does not change what earlier runs report.
Multi-turn Result Details
For multi-turn tests, the run detail includes conversation-specific execution data:
- Turns used: How many turns were consumed during execution
- Configured max turns: The max turn budget from test configuration
- Goal achieved flag: Whether the multi-turn goal was met
- Conversation-linked trace access: Open the trace drawer to inspect tree, sequence, graph, and conversation views for the related trace
Cancel In-Progress Test Runs
You can cancel runs that are still waiting or running:
- Open the Test Runs list view
- Select one or more runs with status Queued or Progress
- Click Cancel Test Run or Cancel Test Runs
- Confirm the action
The API route used by the UI is POST /test_runs/{id}/cancel.
| Current status | Cancel allowed | Result |
|---|---|---|
Queued | Yes | Run is marked Cancelled immediately |
Progress | Yes | Run is marked Cancelled; in-flight work stops cooperatively |
Completed / Failed | No | API returns 409 |
| Missing run ID | No | API returns 404 |
Cancellation sets the run status to Cancelled immediately so dashboards and
run lists update without waiting for all worker tasks to finish.
Annotate results
An annotation records a human verdict over the automated evaluation. Open a result, use its Annotations tab, and see Annotations for the targets available and what an override does.
Two things are specific to a run:
Open the filter drawer on a run and use Annotation Status to show Annotated, Not Annotated, or Conflicting results. Conflicting are the ones where a human verdict disagrees with the automated one.
In a comment, typing @ mentions a metric or a turn, and the first mention sets what the annotation targets:
Compare Test Runs
Compare test runs to identify regressions and improvements between executions.
How to Compare:
- Click the Compare button (top right, next to Download)
- Select a baseline test run to compare against
- View the test-by-test comparison of the test set
Comparison Filters:
Use filters to focus on specific changes:
- All Tests: Show all tests from both runs
- Improved: Tests that now pass but failed in baseline
- Regressed: Tests that now fail but passed in baseline
- Unchanged: Tests with the same pass/fail status
Test Run Metrics
Each test result records the metrics it was evaluated against. Which metrics run is resolved from a priority hierarchy (execution-time → test set → requirement) configured when you start the run. See Test Execution for how to set metrics at each level.
The Metrics tab of a result lists every metric with its verdict and the reason the judge gave. Filter to Passed or Failed to narrow a long list.
Next steps
- Review failed tests to understand issues
- Compare against baseline runs to detect regressions
- Annotate results where the automated verdict is wrong
- Export results for reporting or analysis