Models

Models generate test data for LLM applications and act as LLM judges during evaluation. A single function, get_model, creates instances across every supported provider.

Using the `get_model` function

Call get_model to create a model instance. With no arguments it returns the default Rhesis model.

default_model.py
from rhesis.sdk.models import get_model

model = get_model()

To use a different provider, you can pass the provider name as an argument. This will use the default model for that provider.

provider_model.py
from rhesis.sdk.models import get_model

# Use default Gemini model
model = get_model("gemini")

Supported providers (highlights):

rhesis - Rhesis-hosted default models
openai, anthropic, gemini, vertex_ai - major hosted providers
azure_ai - Azure AI Studio deployments via LiteLLM
azure - Azure OpenAI deployments via LiteLLM
litellm_proxy - OpenAI-compatible LiteLLM Proxy endpoint
openrouter, mistral, cohere, groq, perplexity, replicate, together_ai, meta_llama
ollama, huggingface, vllm - local/self-hosted options

To use a specific model, provide its name in the format provider/model_name:

specific_model.py
from rhesis.sdk.models import get_model

model = get_model("gemini/gemini-2.0-flash")

The above code is equivalent to:

specific_model_alt.py
from rhesis.sdk.models import get_model

model = get_model(provider="gemini", model_name="gemini-2.0-flash")

Provider-specific connection options

Some providers accept extra connection parameters:

Provider	Required fields	Optional fields	Notes
`litellm_proxy`	`model_name`	`api_base`, `api_key`	`api_base` defaults to `LITELLM_PROXY_BASE_URL` env var or `http://0.0.0.0:4000`
`azure_ai`	`model_name`, `api_base`, `api_key`	—	`api_base`/`api_key` can be supplied via `AZURE_AI_API_BASE`/`AZURE_AI_API_KEY` env vars
`azure`	`model_name`, `api_base`, `api_key`	`api_version`	`api_base`/`api_key`/`api_version` can be supplied via `AZURE_API_BASE`/`AZURE_API_KEY`/`AZURE_API_VERSION` env vars

litellm_proxy.py
from rhesis.sdk.models import get_model

model = get_model(
    provider="litellm_proxy",
    model_name="gpt-4o-mini",
    api_base="http://localhost:4000",
    api_key="proxy-key",  # optional
)

azure_ai.py
from rhesis.sdk.models import get_model

model = get_model(
    provider="azure_ai",
    model_name="command-r-plus",
    api_base="https://your-endpoint.inference.ai.azure.com/",
    api_key="your-azure-ai-key",
)

azure_openai.py
from rhesis.sdk.models import get_model

model = get_model(
    provider="azure",
    model_name="my-gpt4o-deployment",
    api_base="https://your-resource.openai.azure.com/",
    api_key="your-azure-openai-key",
    api_version="2024-08-01-preview",
)

Direct import

Alternatively, you can access models by importing the model class directly. When you provide a model name as an argument, that specific model will be used. If no model name is provided, the default model for that provider will be used.

direct_import.py
from rhesis.sdk.models import AzureOpenAILLM, GeminiLLM

# Use specific Gemini model
gemini_model = GeminiLLM("gemini-2.0-flash")

# Use Azure OpenAI deployment
azure_model = AzureOpenAILLM(
    model_name="my-gpt4o-deployment",
    api_base="https://your-resource.openai.azure.com/",
    api_key="your-azure-openai-key",
)

Generating text

All models share the same interface. The core method is generate, which takes a prompt and an optional Pydantic schema for structured output. Language-model providers are async-first:

Method	Execution style	Typical use
`generate(...)`	Synchronous wrapper (bridges to `a_generate`)	Scripts and notebooks
`a_generate(...)`	Native async call	Async services, workers, concurrent pipelines
`generate_batch(...)`	Multi-prompt batch call	High-throughput generation

If your application is already async, call a_generate(...) directly.

a_generate(...) retries transient failures — connection errors, timeouts, and HTTP 429/500/502/503/504 from either requests or the provider SDK — with one initial attempt plus three retries and exponential backoff with jitter. Client errors (400, 401, 403, 404) are never retried.

Pass provider-specific keyword arguments through generate(...) or a_generate(...): stream=True enables token streaming on providers that support it, and schema requests structured output.

Multi-turn messages

Pass a pre-built messages list to a_generate(...) when you already have conversation history. When messages is provided, prompt and system_prompt are ignored because the full chat payload is supplied by the caller.

multi_turn_messages.py
import asyncio

from rhesis.sdk.models import get_model

model = get_model("openai/gpt-4o-mini")

messages = [
    {"role": "system", "content": "You are a concise support assistant."},
    {"role": "user", "content": "Can I change my delivery address?"},
    {"role": "assistant", "content": "Yes, before the package ships."},
    {"role": "user", "content": "Where do I do that?"},
]

async def main():
    response = await model.a_generate(messages=messages)
    print(response)

asyncio.run(main())

Use either messages or prompt plus system_prompt for a single call. Keeping the two modes separate avoids accidentally dropping system instructions when a multi-turn history is already assembled.

Generate text using prompt only:

generate_text.py
# Use default Rhesis model
model = get_model()
output = model.generate(prompt="What is the capital of France?")
# Output: "The capital of France is Paris."

Generate structured output using schemas:

generate_structured.py
from pydantic import BaseModel
from rhesis.sdk.models import get_model

class City(BaseModel):
    name: str
    population: int

class CityResponse(BaseModel):
    biggest_cities: list[City]

# Use default Rhesis model
model = get_model()
output = model.generate(
    prompt="The list of 5 biggest cities in Germany?",
    schema=CityResponse
)

Batch Processing

generate_batch processes multiple prompts in parallel.

Basic batch generation:

batch_basic.py
from rhesis.sdk.models import get_model

model = get_model("openai/gpt-4o-mini")

# Process multiple prompts in a single batch call
prompts = [
    "What is the capital of France?",
    "What is the capital of Germany?",
    "What is the capital of Spain?",
]

results = model.generate_batch(prompts=prompts)
# Returns: ["Paris is the capital of France.", "Berlin is...", "Madrid is..."]

Batch generation with structured output:

batch_structured.py
from pydantic import BaseModel
from rhesis.sdk.models import get_model

class CityInfo(BaseModel):
    name: str
    country: str
    population: int

model = get_model("openai/gpt-4o-mini")

prompts = [
    "Provide info about Paris",
    "Provide info about Tokyo",
    "Provide info about New York",
]

results = model.generate_batch(
    prompts=prompts,
    schema=CityInfo
)
# Returns list of validated dicts matching CityInfo schema

Generate multiple completions per prompt:

batch_multiple.py
from rhesis.sdk.models import get_model

model = get_model("openai/gpt-4o-mini")

prompts = ["Generate a creative product name for a coffee brand"]

# Generate 3 different responses for each prompt
results = model.generate_batch(
    prompts=prompts,
    n=3
)
# Returns 3 different product name suggestions

Use batch generation for large test sets, multi-input evaluations, or generating variations. It runs on LiteLLM-based providers and rhesis language models.

async_generation.py
import asyncio
from rhesis.sdk.models import get_model

model = get_model("rhesis")

async def main():
    single = await model.a_generate("Summarize why unit tests matter.")
    # generate_batch is synchronous; use asyncio.to_thread to avoid blocking the event loop
    batch = await asyncio.to_thread(
        model.generate_batch,
        prompts=[
            "Give one CI best practice.",
            "Give one code review best practice.",
        ],
    )
    print(single)
    print(batch)

asyncio.run(main())

Using models with synthesizers and metrics

Pass a model to a synthesizer or metric to control which LLM does the work:

models_with_tools.py
from rhesis.sdk.models import get_model
from rhesis.sdk.synthesizers import PromptSynthesizer
from rhesis.sdk.metrics import RhesisPromptMetricNumeric

# With synthesizers
model = get_model("gemini")
synthesizer = PromptSynthesizer(
    prompt="Generate tests for the car selling chatbot",
    model=model,
)

# With metrics
metric = RhesisPromptMetricNumeric(
    name="answer_quality_evaluator",
    evaluation_prompt="Evaluate the answer for accuracy, completeness, clarity, and relevance.",
    model="gemini",
)

Saving models to the platform

You can save an LLM configuration to the Rhesis platform as a Model entity. This allows you to:

Store model configurations centrally for team sharing
Set default models for test generation and evaluation
Retrieve configurations across different scripts

save_model.py
from rhesis.sdk.models import get_model

# Create an LLM instance
llm = get_model("openai", "gpt-4", api_key="sk-...")

# Save to platform as a Model entity
model = llm.push(name="GPT-4 Production")

# Set as default for generation or evaluation
model.set_default_generation()
model.set_default_evaluation()

You can also retrieve saved configurations and convert them back to LLM instances:

load_model.py
from rhesis.sdk.entities import Models

# Pull saved model from platform
model = Models.pull(name="GPT-4 Production")

# Convert to LLM instance
llm = model.get_model_instance()

# Use for generation
response = llm.generate("Hello, how are you?")

Tracking token usage

Every provider reports token counts after each call. Counts are normalized across providers into three keys — input_tokens, output_tokens, total_tokens — so you never have to know whether a given API calls them prompt_tokens, promptTokenCount, or something else.

Pass on_usage to watch one model:

track_one_model.py
from rhesis.sdk.models import get_model

totals = []
llm = get_model("openai/gpt-4o", api_key="sk-...", on_usage=totals.append)

llm.generate("Hello")
print(totals)  # [{'input_tokens': 8, 'output_tokens': 12, 'total_tokens': 20}]

To count tokens across an application, register a process-wide sink instead. It receives usage from every model in the process, including models built by code that knows nothing about usage tracking, so there is no per-call-site wiring to forget:

track_everything.py
from rhesis.sdk.models import get_model, set_default_usage_callback

def record(usage, model):
    print(f"{model.model_name} used {usage['total_tokens']} tokens")

set_default_usage_callback(record)

# No on_usage argument, and no reference to the sink.
get_model("openai/gpt-4o", api_key="sk-...").generate("Hello")

The sink fires in addition to any on_usage, not instead of it, so attaching a listener to one model does not detach it from your accounting. Batch calls report one summed total rather than one emission per prompt. Both callbacks run inside the generate call, and anything they raise is logged and swallowed — a broken counter never breaks the LLM call that produced the count.

BaseLLM.usage_metered is an optional flag for applications that bill for tokens. The SDK never sets or reads it; stamp it where you resolve models to record whose credentials paid for a given call, since the same provider can be billable or not depending on whose API key it uses.

Writing a custom provider? Call self._emit_usage(raw_usage) wherever you parse the provider’s response, passing the payload unchanged — normalization happens for you. Use self._emit_usage_batch(...) for batch methods. A provider that skips this reports zero tokens forever, and nothing else will fail to tell you.

Embedders

Embedders generate vector representations (embeddings) of text, which are useful for semantic search, similarity comparison, and clustering tasks.

Using the `get_model` function

Call get_model with an embedding model name; it detects the type and returns an embedder.

default_embedder.py
from rhesis.sdk.models import get_model

embedder = get_model("openai/text-embedding-3-small")

To use a specific model, provide the model name:

specific_embedder.py
from rhesis.sdk.models import get_model

# Use a specific embedding model (auto-detected from name)
embedder = get_model("openai/text-embedding-3-large")

The above code is equivalent to:

specific_embedder_alt.py
from rhesis.sdk.models import get_model

embedder = get_model(provider="openai", model_name="text-embedding-3-large", model_type="embedding")

Direct import

You can also import the embedder class directly:

direct_embedder_import.py
from rhesis.sdk.models import OpenAIEmbedder

# Use default model (text-embedding-3-small)
embedder = OpenAIEmbedder()

# Use specific model with custom dimensions
embedder = OpenAIEmbedder(model_name="text-embedding-3-large", dimensions=1024)

Generate embeddings

All embedders share a consistent interface with two main methods:

Generate embedding for a single text:

generate_embedding.py
from rhesis.sdk.models import get_model

embedder = get_model("openai/text-embedding-3-small")
embedding = embedder.generate("What is machine learning?")
# Returns: [0.0123, -0.0456, 0.0789, ...]  (list of floats)

Generate embeddings for multiple texts:

generate_embeddings_batch.py
from rhesis.sdk.models import get_model

embedder = get_model("openai/text-embedding-3-small")
texts = [
    "What is machine learning?",
    "How does deep learning work?",
    "Explain neural networks",
]

embeddings = embedder.generate_batch(texts)
# Returns: list of embedding vectors, one per input text

Configuring embedding dimensions

Some embedding models (like OpenAI’s text-embedding-3 family) support configurable output dimensions. Smaller dimensions reduce storage and computation costs while maintaining most of the semantic information.

embedder_dimensions.py
from rhesis.sdk.models import get_model

# Set dimensions at initialization
embedder = get_model("openai/text-embedding-3-small", dimensions=256)
embedding = embedder.generate("Hello world")
# Returns embedding with 256 dimensions

# Or override per call
embedding = embedder.generate("Hello world", dimensions=512)
# Returns embedding with 512 dimensions

See the Model entity documentation for more details on managing model configurations.

Models

Using the get_model function

Provider-specific connection options

Direct import

Generating text

Multi-turn messages

Batch Processing

Using models with synthesizers and metrics

Saving models to the platform

Tracking token usage

Embedders

Using the get_model function

Direct import

Generate embeddings

Configuring embedding dimensions

Using the `get_model` function

Using the `get_model` function