Skip to Content
ContributeWorkerTest Types

Test Types

The test type determines how a test is executed and evaluated. Rhesis supports single-turn and multi-turn tests.

Single-Turn Tests

Description

Traditional request-response tests with a single prompt. The endpoint receives one message and returns one response, which is then evaluated against configured metrics.

Use Cases

  • API Validation: Testing specific API endpoints and responses
  • Regression Testing: Ensuring consistent behavior across releases
  • Functional Testing: Verifying specific features or capabilities
  • Performance Benchmarks: Measuring response time and quality

How It Works

  1. Send single prompt to endpoint
  2. Receive response
  3. Evaluate response using configured metrics
  4. Store results

Configuration

Single-turn tests use the Test model with a Prompt containing:

  • content: The input text
  • expected_response: Optional expected output
  • Associated Requirement with metrics for evaluation

Example

single_turn_test.py
# Create a single-turn test
test = Test(
    prompt=prompt,
    requirement=requirement,  # Contains metrics
    test_type="Single-Turn",
    organization_id=org_id,
    user_id=user_id
)

Metric Evaluation

Metrics are evaluated by the worker using the MetricEvaluator:

  • Runs after endpoint responds
  • Each metric gets prompt, response, and optional context
  • Results stored with pass/fail status

Multi-Turn Tests

Description

Agentic conversation tests where Penelope orchestrates multi-turn interactions to achieve a specific goal. Instead of a single request-response, Penelope conducts an entire conversation strategy.

Use Cases

  • Conversational AI Testing: Testing chatbots and virtual assistants
  • Goal-Based Scenarios: Verifying complex user journeys
  • Context Maintenance: Testing conversation memory and coherence
  • Dialogue Flow Testing: Ensuring proper conversation handling
  • User Intent Testing: Verifying the system understands and handles user goals

How It Works

  1. On first execution, the test’s goal/instructions/restrictions/scenario are interpreted into an evaluation contract stating what the target must and must not do — see Evaluation Contract below. Cached on the test until its wording changes.
  2. Initialize Penelope agent with the test goal and, when present, the contract
  3. Penelope plans conversation strategy
  4. Agent interacts with endpoint over multiple turns
  5. Penelope evaluates compliance with the contract (or, if none, whether the goal was achieved)
  6. Complete trace stored (including all turns)

Configuration

Multi-turn tests store configuration in the test_configuration JSONB field:

multi_turn_config.json
{
  "goal": "Verify the chatbot can answer 2 questions about insurance coverage",
  "instructions": "Ask about coverage, then ask a follow-up question",
  "scenario": "You are a customer seeking information",
  "restrictions": "The chatbot must not mention competitor brands",
  "context": {
    "additional_info": "..."
  },
  "max_turns": 10
}

Configuration Fields

  • goal (required): What the test should achieve
  • instructions (optional): How to approach the goal
  • scenario (optional): Role/context for the agent
  • restrictions (optional): Boundaries the target must respect
  • context (optional): Additional metadata
  • max_turns (optional): Maximum conversation turns (default: 10)

Evaluation Contract

An author can phrase goal from either side of an adversarial test — “get the target to leak data” or “the target refuses to leak data” — and both describe the same test but score opposite outcomes as a pass if scored literally. rhesis.backend.app.services.test_interpretation derives an evaluation contract from goal, instructions, restrictions, and scenario, restating them as a fixed required_behavior / prohibited_behavior list plus a simulated_user_objective for Penelope to pursue. This runs lazily on first execution, keyed by a hash of those four fields, and is stored on Test.test_metadata so later runs reuse it without re-interpreting.

A contract that can’t be derived with enough confidence — an empty or ambiguous test — makes the run report Error rather than a scored verdict; see contract_usability().

Example

multi_turn_test.py
# Create a multi-turn test
test = Test(
    prompt=prompt,  # Used as fallback goal if not in config
    test_type="Multi-Turn",
    test_configuration={
        "goal": "Verify chatbot maintains context across 3 turns",
        "instructions": "Ask related questions building on previous responses",
        "max_turns": 5
    },
    organization_id=org_id,
    user_id=user_id
)

Metric Evaluation

Metrics are evaluated by Penelope during execution:

  • Goal Achievement: Primary metric — compliance with the evaluation contract when one was derived, otherwise whether the raw goal was achieved
  • Criteria/Behaviour Evaluation: Individual required/prohibited behaviours checked independently, each with its own verdict
  • Confidence Score: How confident Penelope is in the evaluation
  • Evidence: Conversation excerpts supporting the evaluation

The complete Penelope trace is stored, including:

  • All conversation turns
  • Agent reasoning at each step
  • Tool calls and responses
  • Goal evaluation details
  • Execution statistics

Penelope Trace Structure

penelope_trace.json
{
"status": "success",
"goal_achieved": true,
"turns_used": 3,
"findings": ["✓ All criteria met"],
"history": [
  {
    "turn_number": 1,
    "reasoning": "...",
    "assistant_message": {...},
    "tool_message": {...}
  }
],
"goal_evaluation": {
  "all_criteria_met": true,
  "criteria_evaluations": [...],
  "confidence": 1.0,
  "reasoning": "...",
  "evidence": [...]
},
"execution_stats": {...},
"metrics": {
  "Goal Achievement": {
    "score": 1.0,
    "is_successful": true,
    "confidence": 1.0
  }
}
}

Test Type Detection

The system automatically routes tests to the appropriate executor:

test_type_detection.py
from rhesis.backend.jobs.execution.modes import get_test_type, is_multi_turn_test
from rhesis.backend.app.constants import TestType

# Get test type
test_type = get_test_type(test)

# Check specific type
if is_multi_turn_test(test):
    # Multi-turn specific logic
    pass

# Or use enum
if test_type == TestType.MULTI_TURN:
    # ...
Test ExecutionExecution Modes