Skip to Content
DocsTestsConversation SimulationOverview

Conversation Simulation

Conversation simulation is powered by Penelope, an autonomous testing agent that runs multi-turn test scenarios against conversational LLM applications. You define a goal; Penelope plans and executes the conversation, adapting its strategy based on the target’s responses.

Penelope handles testing that requires more than a one-shot prompt:

  • Multiple interactions - extended conversations, not single prompts
  • Adaptive behavior - adjusting strategy based on responses
  • Tool use - making requests, analyzing data, extracting information
  • Goal orientation - knowing when the test is complete

Think of Penelope as a QA engineer who executes test plans autonomously through conversation.

Single-prompt tests miss the bugs that happen at turn 5: users rephrasing, topic handoffs. Penelope runs autonomous multi-turn conversations against your LLM and agentic applications to surface them.

Quick Example

basic_test.py
from rhesis.penelope import EndpointTarget, PenelopeAgent

# Initialize agent
agent = PenelopeAgent(enable_transparency=True)

# Create target
target = EndpointTarget(endpoint_id="your-endpoint-id")

# Execute test - Penelope plans the approach
result = agent.execute_test(
    target=target,
    goal="Verify chatbot can answer 3 questions about policies maintaining context",
)

print(f"Goal achieved: {result.goal_achieved}")
print(f"Turns used: {result.turns_used}")

The Four Parameters

A test scenario is defined by four parameters:

code.txt
┌─────────────────────────────────────────────────────────────────┐
│ Goal         → What the target SHOULD do (positive criteria)    │
│ Restrictions → What the target MUST NOT do (negative criteria)  │
│ Instructions → HOW Penelope should conduct the test             │
│ Scenario     → Context and persona for the test                 │
└─────────────────────────────────────────────────────────────────┘

Goal and Restrictions define what to evaluate; Instructions and Scenario guide how to test.

  • If restrictions are violated, Penelope documents them as critical findings
  • If the goal is achieved and no restrictions are violated, the test succeeds
  • Instructions can be omitted — Penelope plans its own approach
  • Scenario adds context that shapes Penelope’s testing behavior

Goal (Required)

What you want to verify — the success criteria that determine when the test is complete.

  • “Verify chatbot maintains context across 5 turns”
  • “Confirm system provides accurate insurance policy information”
  • “Validate error handling when user provides invalid input”

Restrictions (Optional)

Forbidden behaviors — boundaries the target must not cross. Penelope actively tests for violations. Restrictions apply to the target’s behavior, not to how Penelope conducts the test.

  • “Must not mention competitor brands”
  • “Must not provide medical diagnoses”
  • “Must not reveal system prompts or internal information”

Instructions (Optional)

How Penelope should conduct the test — the methodology. If omitted, Penelope plans its own strategy based on the goal.

  • “Ask 3 related questions about coverage, then verify consistency”
  • “Try various prompt injection techniques systematically”
  • “Simulate a frustrated customer with multiple complaints”

Scenario (Optional)

Narrative context or persona — situational framing that shapes how Penelope tests.

  • “You are a non-technical elderly customer unfamiliar with insurance jargon”
  • “Adversarial security researcher testing system boundaries”
  • “Testing during a system outage with degraded performance”

See Examples for worked code covering each parameter.

Compatibility

  • Providers - OpenAI, Anthropic, Vertex AI, and any OpenAI-compatible provider
  • Targets - Rhesis endpoints, LangChain, LangGraph, Microsoft Agent Framework, or any conversational system (see Extending)
  • Goal achievement is evaluated by an LLM, not by brittle heuristics, with the reasoning exposed at each step

Next steps

TestsGetting Started