Test Runs & Results

Test runs track the execution of tests against endpoints. Each run produces test results containing the endpoint’s response, evaluation metrics, and review status.

These entities use the shared entity interface for fetching, filtering, updating, and deleting.

TestRun

A TestRun represents a batch execution of tests. When you execute a test set against an endpoint, a test run is created to track the execution.

Properties

Property	Type	Description
`id`	`str`	Unique identifier
`name`	`str`	Display name
`test_configuration_id`	`str`	Associated test configuration
`status`	`RunStatus`	Execution status: `Progress`, `Completed`, `Partial`, or `Failed`
`user_id`	`str`	User who initiated the run
`organization_id`	`str`	Organization ID
`owner_id`	`str`	Owner of the test run
`assignee_id`	`str`	Assigned reviewer
`attributes`	`dict`	Custom attributes

Fetching Test Runs

fetch_test_runs.py
from rhesis.sdk.entities import TestRuns

# List all test runs
for run in TestRuns.all():
    print(f"{run.name}: {run.status}")

# Get by ID
run = TestRuns.pull(id="run-123")

# Filter by status
completed_runs = TestRuns.all(filter="status/name eq 'Completed'")

Getting Test Results

Retrieve all results for a test run:

get_results.py
from rhesis.sdk.entities import TestRuns

run = TestRuns.pull(id="run-123")
results = run.get_test_results()

for result in results:
    print(f"Test: {result['test_id']}")
    print(f"Output: {result['test_output']}")
    print(f"Metrics: {result['test_metrics']}")

TestResult

A TestResult contains the output and evaluation for a single test execution.

Properties

Property	Type	Description
`id`	`str`	Unique identifier
`test_run_id`	`str`	Parent test run
`test_id`	`str`	Executed test
`prompt_id`	`str`	Test prompt
`status_id`	`str`	Result status
`status`	`Status`	Status object with name/description
`test_output`	`dict`	Endpoint response data
`test_metrics`	`dict`	Evaluation metric scores
`test_reviews`	`dict`	Human review data
`test_configuration_id`	`str`	Test configuration used

Fetching Test Results

fetch_results.py
from rhesis.sdk.entities import TestResults

# Get all results
all_results = TestResults.all()

# Get by ID
result = TestResults.pull(id="result-123")

# Filter by test run
run_results = TestResults.all(filter="test_run_id eq 'run-123'")

# Filter by status
failed_results = TestResults.all(filter="status_id eq 'failed'")

Working with Results

analyze_results.py
from rhesis.sdk.entities import TestResults

result = TestResults.pull(id="result-123")

# Access output
if result.test_output:
    print(f"Response: {result.test_output.get('output')}")
    print(f"Session: {result.test_output.get('session_id')}")

# Access metrics
if result.test_metrics:
    for metric_name, score in result.test_metrics.items():
        print(f"{metric_name}: {score}")

# Check status
if result.status:
    print(f"Status: {result.status.name}")

TestConfiguration

A TestConfiguration defines the settings for test execution, linking test sets to endpoints with specific parameters.

Properties

Property	Type	Description
`id`	`str`	Unique identifier
`endpoint_id`	`str`	Target endpoint (required)
`test_set_id`	`str`	Test set to execute
`category_id`	`str`	Filter by category
`topic_id`	`str`	Filter by topic
`prompt_id`	`str`	Specific prompt
`status_id`	`str`	Configuration status
`attributes`	`dict`	Custom settings

Creating Test Configurations

create_config.py
from rhesis.sdk.entities import TestConfiguration

config = TestConfiguration(
    endpoint_id="endpoint-123",
    test_set_id="test-set-456",
    attributes={
        "timeout": 30,
        "retry_count": 3,
    }
)

config.push()
print(f"Created configuration: {config.id}")

Getting Test Runs for a Configuration

config_runs.py
from rhesis.sdk.entities import TestConfigurations

config = TestConfigurations.pull(id="config-123")
runs = config.get_test_runs()

for run in runs:
    print(f"Run: {run['id']} - Status: {run['status_id']}")

For the full execution API — including execution modes, metric overrides, re-scoring, and test set metric management — see Test Execution.