Synthesizers

Generate test sets for evaluating LLM applications. Synthesizers use an LLM to create test cases from prompts, configurations, or source documents.

All synthesizers accept a model parameter to customize the LLM used for generation. See Models for available options and configuration.

Quick Start

quick_start.py
from rhesis.sdk.synthesizers import PromptSynthesizer

synthesizer = PromptSynthesizer(
    prompt="Generate tests for a customer support chatbot that handles refund requests"
)
test_set = synthesizer.generate(num_tests=10)

for test in test_set.tests:
    print(test.prompt.content)

Adversarial-only generation

Set harmful=True when you want the synthesizer prompt to generate adversarial attack tests only. This removes the default harmless category mix and asks the model to focus on harmful, manipulative, or policy-violating inputs.

harmful_generation.py
from rhesis.sdk.synthesizers import PromptSynthesizer

synthesizer = PromptSynthesizer(
    prompt="Generate attacks against a travel booking chatbot",
    model="rhesis/polyphemus-default",
    harmful=True,
)

test_set = synthesizer.generate(num_tests=20)

OWASPSynthesizer enables adversarial-only generation by default. For general-purpose synthesizers, pass harmful=True explicitly.

Available Synthesizers

PromptSynthesizer

The simplest option. Provide a prompt describing what to test. When no model is provided, the default Rhesis model is used.

prompt_synthesizer.py
from rhesis.sdk.synthesizers import PromptSynthesizer

synthesizer = PromptSynthesizer(
    prompt="Generate adversarial tests for a medical advice chatbot",
)
test_set = synthesizer.generate(num_tests=20)

Synthesizer

Full control over generation with requirements, categories, and topics.

synthesizer.py
from rhesis.sdk.synthesizers import Synthesizer

synthesizer = Synthesizer(
    prompt="Test an insurance claims assistant",
    requirements=["helpful", "refuses harmful requests", "admits uncertainty"],
    categories=["auto claims", "home claims", "policy questions"],
    topics=["coverage limits", "deductibles", "filing process"],
)
test_set = synthesizer.generate(num_tests=30)

ConfigSynthesizer

Use a configuration object for reusable test generation settings.

config_synthesizer.py
from rhesis.sdk.synthesizers import ConfigSynthesizer, GenerationConfig

config = GenerationConfig(
    generation_prompt="Test a legal document assistant",
    requirements=["accurate", "cites sources"],
    categories=["contracts", "compliance"],
    topics=["liability clauses", "termination terms"],
    additional_context="The assistant serves corporate lawyers",
)

synthesizer = ConfigSynthesizer(config=config)
test_set = synthesizer.generate(num_tests=15)

ContextSynthesizer

Generate tests grounded in specific context provided at runtime.

context_synthesizer.py
from rhesis.sdk.synthesizers import ContextSynthesizer

synthesizer = ContextSynthesizer(
    prompt="Generate questions a user might ask about this product"
)

product_description = """
The XR-500 is a wireless noise-canceling headphone with 40-hour battery life,
Bluetooth 5.2, and active noise cancellation with transparency mode.
"""

test_set = synthesizer.generate(num_tests=10, context=product_description)

OWASPSynthesizer

Generate red-team tests aligned with the OWASP LLM Top 10. Use this when you want security-focused prompts tailored to the purpose of your application instead of a generic adversarial prompt list.

owasp_synthesizer.py
from rhesis.sdk.synthesizers import OWASPSynthesizer

synthesizer = OWASPSynthesizer(
    purpose="Customer support chatbot for a bank with access to account data",
    categories=["llm01", "llm02", "llm07"],  # Omit to cover all 10 categories
    batch_size=10,
    model="gemini/gemini-2.0-flash",
)

test_set = synthesizer.generate(num_tests=30)

for test in test_set.tests:
    print(test.prompt.content)
    print(test.metadata["owasp_category"], test.metadata["owasp_name"])

OWASPSynthesizer always creates harmful single-turn tests. It spreads num_tests across the selected OWASP categories and tags each test with metadata["owasp_category"] and metadata["owasp_name"].

The synthesizer downloads the selected report when generate() first runs, splits the PDF into risk sections, and uses the official section text as generation context. It does not use a hardcoded attack list.

Parameter	Default	Description
`purpose`	Required	What the system under test does; used to tailor every attack
`report_url`	OWASP LLM Top 10 v2025 PDF	Direct URL to a compatible OWASP Top 10 report
`categories`	All report sections	Section IDs to include, such as `["llm01", "llm07"]`
`subsection_exclusions`	`DEFAULT_SUBSECTION_EXCLUSIONS`	Headings to remove from each section; pass an empty collection to retain all subsections
`batch_size`	`10`	Maximum attacks generated per model call for each section
`model`	Default Rhesis model	Model name or `BaseLLM` used for generation
`requirement`	`"OWASP LLM Top 10"`	Requirement label stored on generated tests for analytics

Use the Agentic Top 10 report by changing both the report and the requirement label:

owasp_agentic_synthesizer.py
from rhesis.sdk.synthesizers import (
    DEFAULT_OWASP_AGENTIC_PDF_URL,
    OWASPSynthesizer,
)

synthesizer = OWASPSynthesizer(
    purpose="Autonomous coding agent with shell access",
    report_url=DEFAULT_OWASP_AGENTIC_PDF_URL,
    categories=["asi01", "asi03"],
    requirement="OWASP Agentic Top 10",
)

test_set = synthesizer.generate(num_tests=20)

Category ID	OWASP category
`llm01`	Prompt Injection
`llm02`	Sensitive Information Disclosure
`llm03`	Supply Chain Vulnerabilities
`llm04`	Data and Model Poisoning
`llm05`	Improper Output Handling
`llm06`	Excessive Agency
`llm07`	System Prompt Leakage
`llm08`	Vector and Embedding Weaknesses
`llm09`	Misinformation
`llm10`	Unbounded Consumption

OWASP generation uses an LLM to create fresh attacks for each run. For high-risk adversarial testing, pair these tests with a suitable generation model and review outputs before running them against production systems.

Using Source Documents

Synthesizers can extract content from documents, websites, text snippets, and images to generate contextually relevant tests.

with_sources.py
from rhesis.sdk.services.extractor import SourceSpecification, SourceType
from rhesis.sdk.synthesizers import PromptSynthesizer

sources = [
    SourceSpecification(
        type=SourceType.WEBSITE,
        name="API Docs",
        metadata={"url": "https://example.com/docs/api-reference"},
    ),
    SourceSpecification(
        type=SourceType.DOCUMENT,
        name="Knowledge Base",
        metadata={"path": "./knowledge_base.pdf"},
    ),
    SourceSpecification(
        type=SourceType.IMAGE,
        name="Product Screenshot",
        metadata={"path": "./screenshot.png"},
    ),
]

synthesizer = PromptSynthesizer(
    prompt="Generate tests based on the provided documentation",
    sources=sources,
)
test_set = synthesizer.generate(num_tests=50)

Image sources

SourceType.IMAGE supports local image paths and image URLs. Without a vision-capable model, extraction falls back to metadata available through MarkItDown. Pass a model to enable vision-based image descriptions.

image_sources.py
from rhesis.sdk.models.factory import get_language_model
from rhesis.sdk.services.extractor import SourceSpecification, SourceType
from rhesis.sdk.synthesizers import PromptSynthesizer

vision_model = get_language_model("openai/gpt-4o")

sources = [
    SourceSpecification(
        type=SourceType.IMAGE,
        name="Checkout Screenshot",
        metadata={"path": "./checkout.png"},
    ),
    SourceSpecification(
        type=SourceType.IMAGE,
        name="Remote Diagram",
        metadata={"url": "https://example.com/flow.png"},
    ),
]

synthesizer = PromptSynthesizer(
    prompt="Generate UI validation tests from these images",
    sources=sources,
    model=vision_model,
)
test_set = synthesizer.generate(num_tests=12)

For uploaded test attachments, images are extracted first, documents use their text layer first, and image-heavy documents fall back to a vision model when one is configured.

Chunking strategies for source-based generation

When you pass sources to a synthesizer, Rhesis chunks extracted content before generation. Chunking defaults to RecursiveChunker(chunk_size=1500) when no custom strategy is provided.

Choose a chunking strategy

Strategy	Best for	Key behavior
`TokenChunker`	Uniform chunk sizes and strict token budgets	Splits by token count with optional overlap
`SentenceChunker`	Readability-focused chunks	Preserves sentence boundaries while respecting token limits
`RecursiveChunker`	Long structured content (docs, markdown, policies)	Splits on large-to-small delimiters recursively

Example: pass a custom chunker

custom_chunking.py
from rhesis.sdk.services.chunker import SentenceChunker
from rhesis.sdk.services.extractor import SourceSpecification, SourceType
from rhesis.sdk.synthesizers import PromptSynthesizer

sources = [
    SourceSpecification(
        type=SourceType.WEBSITE,
        name="Product docs",
        metadata={"url": "https://example.com/docs"},
    )
]

synthesizer = PromptSynthesizer(
    prompt="Generate edge-case tests from this documentation",
    sources=sources,
    chunking_strategy=SentenceChunker(
        chunk_size=800,
        chunk_overlap=80,
        min_sentences_per_chunk=2,
    ),
)

test_set = synthesizer.generate(num_tests=20)

Recipe-based recursive chunking

RecursiveChunker also supports recipe presets:

recursive_chunker_recipe.py
from rhesis.sdk.services.chunker import RecursiveChunker

chunker = RecursiveChunker.from_recipe(recipe="markdown", lang="en")

SemanticChunker is deprecated and now aliases RecursiveChunker for backward compatibility. Prefer RecursiveChunker for new code.

Pushing Test Sets to Rhesis

Push generated test sets to the Rhesis platform for analysis, tracking, and collaboration.

Requirements: A Rhesis account and API key. Set your credentials via environment variables or configuration.

Call test_set.push() to upload. Your test set will appear in Testing → Test Sets.

push_test_set.py
import os

from rhesis.sdk.synthesizers import PromptSynthesizer

os.environ["RHESIS_BASE_URL"] = "https://api.rhesis.ai"
os.environ["RHESIS_API_KEY"] = "YOUR_API_KEY"

synthesizer = PromptSynthesizer(
    prompt="Generate safety tests",
)
test_set = synthesizer.generate(num_tests=10)

test_set.push()