Test Set

Back to Glossary Testing

A collection of tests that can be executed together against an endpoint, similar to test suites in traditional software development.

Also known as: test suite

Overview

Test sets group related tests together for organized execution and analysis. They function like test suites in traditional software testing, allowing you to run comprehensive evaluations with a single command.

Type and requirements

One per set: or (immutable after creation). MCP uses the parameter with the same values.
Requirements listed on the set determine what expectations generated tests target.
All metrics linked to those requirements must include the set's type in or they will not run.

Key Benefits

Test sets help you organize testing by grouping related tests around features, behaviors, or scenarios. Rather than running tests individually, you can execute an entire set with a single command, ensuring consistent evaluation where the same tests run identically every time. This makes it easy to track performance trends across multiple runs and spot regressions Test sets integrate smoothly into CI/CD pipelines, enabling automated quality checks as part of your deployment process.

Common Test Set Patterns

By Feature:

Customer Support Scenarios
Product Recommendation Tests
Search Functionality Tests

By Requirement:

Safety and Harm Prevention
Accuracy and Factuality
Tone and Professionalism

By Purpose:

Smoke Tests (quick validation)
Regression Tests (comprehensive coverage)
Performance Tests (stress testing)

Example Usage

python
from rhesis import TestSet

# Create a test set
safety_tests = TestSet(
      name="Safety Evaluation",
      tests=[
          harmful_content_tests,
          bias_detection_tests,
          privacy_tests
      ]
)

# Run against an endpoint
results = safety_tests.run(endpoint="production-bot")

Documentation

/platform/test-sets