Test Attributes

Test attribute entities provide structure and classification for tests. Categories, topics, and requirements help organize test sets and enable filtered evaluation.

These entities use the shared entity interface for fetching, filtering, updating, and deleting.

Requirement

Requirements define expected actions or responses from LLM applications. Each requirement can have associated metrics for evaluation.

Properties

Property	Type	Description
`id`	`str`	Unique identifier
`name`	`str`	Requirement name (e.g., “Refuses harmful requests”)
`description`	`str`	Detailed description

Working with Requirements

requirements.py
from rhesis.sdk.entities import Requirements, Requirement

# List all requirements
for requirement in Requirements.all():
    print(f"{requirement.name}: {requirement.description}")

# Get by name
requirement = Requirements.pull(name="Refuses harmful requests")

# Create new requirement
new_requirement = Requirement(
    name="Admits uncertainty",
    description="The AI acknowledges when it doesn't know something"
)
new_requirement.push()

Requirement Metrics

Requirements can have associated metrics for automatic evaluation:

requirement_metrics.py
from rhesis.sdk.entities import Requirements

requirement = Requirements.pull(name="Refuses harmful requests")

# Get associated metrics
metrics = requirement.get_metrics()
for metric in metrics:
    print(f"Metric: {metric['name']}")

# Add a metric to this requirement
requirement.add_metric(metric_id="metric-123")

# Remove a metric
requirement.remove_metric(metric_id="metric-456")

Category

Categories provide high-level classification for tests (e.g., “Safety”, “Accuracy”, “Performance”).

Properties

Property	Type	Description
`id`	`str`	Unique identifier
`name`	`str`	Category name
`description`	`str`	Category description

Working with Categories

categories.py
from rhesis.sdk.entities import Categories, Category

# List all categories
for category in Categories.all():
    print(f"{category.name}")

# Get by name
safety = Categories.pull(name="Safety")

# Create new category
new_category = Category(
    name="Compliance",
    description="Tests for regulatory compliance requirements"
)
new_category.push()

Filtering Tests by Category

filter_by_category.py
from rhesis.sdk.entities import Tests

# Get all tests in a category
safety_tests = Tests.all(filter="category eq 'Safety'")

for test in safety_tests:
    print(f"{test.topic}: {test.requirement}")

Topic

Topics provide fine-grained classification within categories (e.g., “Privacy”, “Harmful Content”, “Misinformation”).

Properties

Property	Type	Description
`id`	`str`	Unique identifier
`name`	`str`	Topic name
`description`	`str`	Topic description

Working with Topics

topics.py
from rhesis.sdk.entities import Topics, Topic

# List all topics
for topic in Topics.all():
    print(f"{topic.name}: {topic.description}")

# Get by name
privacy = Topics.pull(name="Privacy")

# Create new topic
new_topic = Topic(
    name="Data Retention",
    description="Tests related to data retention policies"
)
new_topic.push()

Filtering Tests by Topic

filter_by_topic.py
from rhesis.sdk.entities import Tests

# Get all tests for a topic
privacy_tests = Tests.all(filter="topic eq 'Privacy'")

for test in privacy_tests:
    print(f"{test.category}: {test.requirement}")

Next Steps - Create Test Sets using these attributes

Associate Metrics with requirements for evaluation