Synthesizers

Generate test sets for evaluating LLM applications. Synthesizers use LLMs to create diverse, targeted test cases based on prompts, configurations, or source documents.

All synthesizers accept a model parameter to customize the LLM used for generation. See Models for available options and configuration.

Quick Start

quick_start.py
from rhesis.sdk.synthesizers import PromptSynthesizer

synthesizer = PromptSynthesizer(
    prompt="Generate tests for a customer support chatbot that handles refund requests"
)
test_set = synthesizer.generate(num_tests=10)

for test in test_set.tests:
    print(test.prompt.content)

With a specified model:

quick_start_gemini.py
from rhesis.sdk.synthesizers import PromptSynthesizer

synthesizer = PromptSynthesizer(
    prompt="Generate tests for a customer support chatbot that handles refund requests",
    model="gemini/gemini-2.0-flash",
)
test_set = synthesizer.generate(num_tests=10)

Adversarial-only generation

Set harmful=True when you want the synthesizer prompt to generate adversarial attack tests only. This removes the default harmless category mix and asks the model to focus on harmful, manipulative, or policy-violating inputs.

harmful_generation.py
from rhesis.sdk.synthesizers import PromptSynthesizer

synthesizer = PromptSynthesizer(
    prompt="Generate attacks against a travel booking chatbot",
    model="rhesis/polyphemus-default",
    harmful=True,
)

test_set = synthesizer.generate(num_tests=20)

OwaspSynthesizer enables adversarial-only generation by default. For general-purpose synthesizers, pass harmful=True explicitly.

Available Synthesizers

PromptSynthesizer

The simplest option. Provide a prompt describing what to test. When no model is provided, the default Rhesis model is used.

prompt_synthesizer.py
from rhesis.sdk.synthesizers import PromptSynthesizer

synthesizer = PromptSynthesizer(
    prompt="Generate adversarial tests for a medical advice chatbot",
)
test_set = synthesizer.generate(num_tests=20)

Synthesizer

Full control over generation with behaviors, categories, and topics.

synthesizer.py
from rhesis.sdk.synthesizers import Synthesizer

synthesizer = Synthesizer(
    prompt="Test an insurance claims assistant",
    behaviors=["helpful", "refuses harmful requests", "admits uncertainty"],
    categories=["auto claims", "home claims", "policy questions"],
    topics=["coverage limits", "deductibles", "filing process"],
)
test_set = synthesizer.generate(num_tests=30)

ConfigSynthesizer

Use a configuration object for reusable test generation settings.

config_synthesizer.py
from rhesis.sdk.synthesizers import ConfigSynthesizer, GenerationConfig

config = GenerationConfig(
    generation_prompt="Test a legal document assistant",
    behaviors=["accurate", "cites sources"],
    categories=["contracts", "compliance"],
    topics=["liability clauses", "termination terms"],
    additional_context="The assistant serves corporate lawyers",
)

synthesizer = ConfigSynthesizer(config=config)
test_set = synthesizer.generate(num_tests=15)

ContextSynthesizer

Generate tests grounded in specific context provided at runtime.

context_synthesizer.py
from rhesis.sdk.synthesizers import ContextSynthesizer

synthesizer = ContextSynthesizer(
    prompt="Generate questions a user might ask about this product"
)

product_description = """
The XR-500 is a wireless noise-canceling headphone with 40-hour battery life,
Bluetooth 5.2, and active noise cancellation with transparency mode.
"""

test_set = synthesizer.generate(num_tests=10, context=product_description)

OWASPSynthesizer

Generate red-team tests aligned with the OWASP LLM Top 10. Use this when you want security-focused prompts tailored to the purpose of your application instead of a generic adversarial prompt list.

owasp_synthesizer.py
from rhesis.sdk.synthesizers import OWASPSynthesizer

synthesizer = OWASPSynthesizer(
    purpose="Customer support chatbot for a bank with access to account data",
    categories=["llm01", "llm02", "llm07"],  # Omit to cover all 10 categories
    batch_size=10,
    model="gemini/gemini-2.0-flash",
)

test_set = synthesizer.generate(num_tests=30)

for test in test_set.tests:
    print(test.prompt.content)
    print(test.metadata["owasp_category"], test.metadata["owasp_name"])

OWASPSynthesizer always creates harmful single-turn tests. The generator spreads num_tests as evenly as possible across the selected OWASP categories and tags each generated test with metadata["owasp_category"] and metadata["owasp_name"].

Category ID	OWASP category	Rhesis behavior
`llm01`	Prompt Injection	Robustness
`llm02`	Sensitive Information Disclosure	Compliance
`llm03`	Supply Chain Vulnerabilities	Reliability
`llm04`	Data and Model Poisoning	Reliability
`llm05`	Improper Output Handling	Compliance
`llm06`	Excessive Agency	Robustness
`llm07`	System Prompt Leakage	Compliance
`llm08`	Vector and Embedding Weaknesses	Reliability
`llm09`	Misinformation	Reliability
`llm10`	Unbounded Consumption	Robustness

You can inspect the registry programmatically with OWASP_LLM_TOP_10:

owasp_categories.py
from rhesis.sdk.synthesizers import OWASP_LLM_TOP_10

for category_id, category in OWASP_LLM_TOP_10.items():
    print(category_id, category.name, category.topic)

OWASP generation uses an LLM to create fresh attacks for each run. For high-risk adversarial testing, pair these tests with a suitable generation model and review outputs before running them against production systems.

Using Source Documents

Synthesizers can extract content from documents, websites, text snippets, and images to generate contextually relevant tests.

with_sources.py
from rhesis.sdk.services.extractor import SourceSpecification, SourceType
from rhesis.sdk.synthesizers import PromptSynthesizer

sources = [
    SourceSpecification(
        type=SourceType.WEBSITE,
        name="API Docs",
        metadata={"url": "https://example.com/docs/api-reference"},
    ),
    SourceSpecification(
        type=SourceType.DOCUMENT,
        name="Knowledge Base",
        metadata={"path": "./knowledge_base.pdf"},
    ),
    SourceSpecification(
        type=SourceType.IMAGE,
        name="Product Screenshot",
        metadata={"path": "./screenshot.png"},
    ),
]

synthesizer = PromptSynthesizer(
    prompt="Generate tests based on the provided documentation",
    sources=sources,
)
test_set = synthesizer.generate(num_tests=50)

Image sources

SourceType.IMAGE supports local image paths and image URLs. Without a vision-capable model, extraction falls back to metadata available through MarkItDown. Pass a model to enable vision-based image descriptions.

image_sources.py
from rhesis.sdk.models.factory import get_language_model
from rhesis.sdk.services.extractor import SourceSpecification, SourceType
from rhesis.sdk.synthesizers import PromptSynthesizer

vision_model = get_language_model("openai/gpt-4o")

sources = [
    SourceSpecification(
        type=SourceType.IMAGE,
        name="Checkout Screenshot",
        metadata={"path": "./checkout.png"},
    ),
    SourceSpecification(
        type=SourceType.IMAGE,
        name="Remote Diagram",
        metadata={"url": "https://example.com/flow.png"},
    ),
]

synthesizer = PromptSynthesizer(
    prompt="Generate UI validation tests from these images",
    sources=sources,
    model=vision_model,
)
test_set = synthesizer.generate(num_tests=12)

For uploaded test attachments, the execution pipeline uses the shared extract_with_vision_fallback path: images go through image extraction first, documents use their text layer first, and image-heavy documents can fall back to a vision model when one is configured.

Chunking strategies for source-based generation (v0.6.11)

When you pass sources to a synthesizer, Rhesis chunks extracted content before generation. In v0.6.11, chunking is powered by chonkie and defaults to RecursiveChunker(chunk_size=1500) when no custom strategy is provided.

Choose a chunking strategy

Strategy	Best for	Key behavior
`TokenChunker`	Uniform chunk sizes and strict token budgets	Splits by token count with optional overlap
`SentenceChunker`	Readability-focused chunks	Preserves sentence boundaries while respecting token limits
`RecursiveChunker`	Long structured content (docs, markdown, policies)	Splits on large-to-small delimiters recursively

Example: pass a custom chunker

custom_chunking.py
from rhesis.sdk.services.chunker import SentenceChunker
from rhesis.sdk.services.extractor import SourceSpecification, SourceType
from rhesis.sdk.synthesizers import PromptSynthesizer

sources = [
    SourceSpecification(
        type=SourceType.WEBSITE,
        name="Product docs",
        metadata={"url": "https://example.com/docs"},
    )
]

synthesizer = PromptSynthesizer(
    prompt="Generate edge-case tests from this documentation",
    sources=sources,
    chunking_strategy=SentenceChunker(
        chunk_size=800,
        chunk_overlap=80,
        min_sentences_per_chunk=2,
    ),
)

test_set = synthesizer.generate(num_tests=20)

Recipe-based recursive chunking

RecursiveChunker also supports recipe presets:

recursive_chunker_recipe.py
from rhesis.sdk.services.chunker import RecursiveChunker

chunker = RecursiveChunker.from_recipe(recipe="markdown", lang="en")

SemanticChunker is deprecated and now aliases RecursiveChunker for backward compatibility. Prefer RecursiveChunker for new code.

Pushing Test Sets to Rhesis

Push generated test sets to the Rhesis platform for analysis, tracking, and collaboration.

Requirements: A Rhesis account and API key. Set your credentials via environment variables or configuration.

Call test_set.push() to upload. Your test set will appear in Testing → Test Sets.

push_test_set.py
import os

from rhesis.sdk.synthesizers import PromptSynthesizer

os.environ["RHESIS_BASE_URL"] = "https://api.rhesis.ai"
os.environ["RHESIS_API_KEY"] = "YOUR_API_KEY"

synthesizer = PromptSynthesizer(
    prompt="Generate safety tests",
)
test_set = synthesizer.generate(num_tests=10)

test_set.push()

Next Steps - Learn about Models to configure LLMs for generation

Use Metrics to evaluate generated test results