Skip to Content
GlossaryRequirement - Glossary

Requirement

Back to GlossaryTesting

A formalized expectation that describes how your AI system should perform, such as response quality, safety, or accuracy.

Overview

Requirements define the expectations for how your AI system should perform. They serve as the foundation for creating metrics and organizing tests around specific quality dimensions.

Platform model

  • Requirements state expectations; metrics score them — link via (many-to-many). Every requirement in a test set needs at least one linked metric before .
  • Tests are tagged with a requirement when generated — results roll up by requirement.
  • Not the same as category or topic — those classify individual tests for organization; they do not replace requirements or metrics. See Category and Topic.
  • Not a PRD section title — split a numbered requirements list into individually testable Requirements (e.g. "Refuses harmful requests", not "Security").

Common Requirement Categories

Quality:

  • Accuracy: Factually correct information
  • Completeness: Comprehensive responses
  • Relevance: Answers the actual question
  • Clarity: Easy to understand

Safety:

  • Harmlessness: No dangerous or harmful content
  • Appropriate Refusal: Declines inappropriate requests
  • Privacy Aware: Respects PII and confidentiality
  • Bias-Free: Fair and unbiased responses

Functional:

  • Tool Usage: Correctly uses available tools
  • Format Compliance: Follows required formats
  • Instruction Following: Adheres to guidelines
  • Context Awareness: Uses conversation context

Using Requirements

In the Web Interface: Define requirements through the Rhesis web interface when creating metrics and organizing tests.

With SDK Synthesizers:

python
from rhesis.sdk.synthesizers import Synthesizer

synthesizer = Synthesizer(
      prompt="Test a medical chatbot",
      requirements=[
          "medically accurate",
          "cites reliable sources",
          "admits uncertainty when appropriate",
          "refuses to diagnose"
      ],
      categories=["symptoms", "medications", "treatments"]
)

test_set = synthesizer.generate(num_tests=50)

From Requirements to Tests

  1. Define the requirements you care about
  2. Generate tests that exercise those requirements
  3. Create metrics to evaluate the requirements
  4. Run evaluations and analyze results
  5. Iterate based on findings

Best Practices

  • Be specific: Vague requirements lead to inconsistent evaluation
  • Provide examples: Show what good and bad looks like
  • Prioritize: Focus on requirements that matter most to users
  • Iterate: Refine requirements based on real-world performance

Documentation

Related Terms

Terms