Test Runs
A test run is created when you execute a test set against an endpoint. It captures all individual test results with detailed metrics, conversation history, execution metadata, and performance data for analysis.
Test Runs List
Use the list to compare status, pass rate, executor, and review activity across runs. The Reviews column shows how many tests received human reviews. Hover the value to see how many reviewed tests were corrected because the human verdict changed the automated result.
Open the filter drawer and set Reviews to With or Without to include only runs that have reviews or runs that do not.
By clicking on a test run, you’ll see the overview of how the test run went. From here you can:
- Review: Manually revise the automated test evaluation
- Compare: Compare against a baseline test run
- Re-run: Execute the test set again with the same configuration
- Download: Export test run data as CSV
Test Run Detail View
The test run detail page includes an expanded Overview and Conversation workflow for debugging failures faster:
- Go to Test: Open the original test definition directly from a result detail
- Context panel: Expand/collapse captured context items when available
- Metadata panel: Expand/collapse structured metadata for the result
- Files / Output Files panels: Inspect and download both input attachments and generated output files
- JSON-aware rendering: Response content that is valid JSON is pretty-printed in the overview tab
For multi-turn tests, the Conversation tab supports trace-driven debugging:
- Click assistant responses to open the trace drawer
- Navigate turn-by-turn while keeping test result context visible
Parameters Used
When a test run targets a parameter experiment, it snapshots the experiment’s resolved values at queue time. This guarantees that test runs remain reproducible even if the experiment or environment changes later.
The Test Run detail page displays an Experiment card showing the experiment name, version (e.g., v3), the source environment (if any), and an expandable Resolved Values panel. (Values for secret_ref slots are masked so credentials never leak into the test run record.)
Because Test Results inherit this snapshot from their parent test run, there is no need to track configurations on a per-result basis. The test-runs list page also includes an Experiment filter to help you slice historical results by configuration.
Multi-turn Result Details
For multi-turn tests, the run detail includes conversation-specific execution data:
- Turns used: How many turns were consumed during execution
- Configured max turns: The max turn budget from test configuration
- Goal achieved flag: Whether the multi-turn goal was met
- Conversation-linked trace access: Open the trace drawer to inspect tree, sequence, graph, and conversation views for the related trace
Cancel In-Progress Test Runs
You can cancel runs that are still waiting or running:
- Open the Test Runs list view
- Select one or more runs with status Queued or Progress
- Click Cancel Test Run or Cancel Test Runs
- Confirm the action
The API route used by the UI is POST /test_runs/{id}/cancel.
| Current status | Cancel allowed | Result |
|---|---|---|
Queued | Yes | Run is marked Cancelled immediately |
Progress | Yes | Run is marked Cancelled; in-flight work stops cooperatively |
Completed / Failed | No | API returns 409 |
| Missing run ID | No | API returns 404 |
Cancellation sets the run status to Cancelled immediately so dashboards and
run lists update without waiting for all worker tasks to finish.
Review Test Runs
Reviews allow human evaluators to validate or override automated test evaluations.
- Select a Test Run from the Test Runs overview page
- Click on the Reviews tab, then Add Your Review
- Select a Review Status:
- Pass: The test passes the metric
- Fail: The test fails the metric
- Add a comment explaining your review decision
Review targets and override behavior
Reviews can now target different parts of a test result, not only the overall result.
| Target type | What it reviews | Reference format |
|---|---|---|
test_result | Overall pass/fail decision for the test result | null |
metric | A specific metric result in test_metrics | Metric name |
turn | A specific turn in multi-turn conversation output | Turn label such as Turn 3 |
When a review targets a metric or turn, Rhesis stores an override marker on the affected object and recomputes effective status in the test run views.
In review comments, type @ to mention metrics or turns. The first metric or turn mention determines the
review target automatically.
On a test run detail page, open the filter drawer and use Review Status to show reviewed, unreviewed, or conflicting results. Conflicting shows results where a human review disagrees with the automated verdict.
Compare Test Runs
Compare test runs to identify regressions and improvements between executions.
How to Compare:
- Click the Compare button (top right, next to Download)
- Select a baseline test run to compare against
- View the test-by-test comparison of the test set
Comparison Filters:
Use filters to focus on specific changes:
- All Tests: Show all tests from both runs
- Improved: Tests that now pass but failed in baseline
- Regressed: Tests that now fail but passed in baseline
- Unchanged: Tests with the same pass/fail status
Test Run Metrics
Each test result records the metrics it was evaluated against. Which metrics run is resolved from a priority hierarchy (execution-time → test set → behavior) configured when you start the run. See Test Execution for how to set metrics at each level.
Next steps
- Review failed tests to understand issues
- Compare against baseline runs to detect regressions
- Add human reviews to test results
- Export results for reporting or analysis