Overview

The Rhesis SDK evaluates LLM applications with built-in metrics from several frameworks, custom metrics you define, and metrics managed on the platform.

Metric types

Single-turn metrics evaluate an individual exchange between a user input and a system output — RAG quality, response quality, safety, and custom checks.
Conversational metrics evaluate interactions across multiple turns — coherence, goal achievement, role adherence, knowledge retention, and tool usage.

Metric Scopes

Every metric has a metric_scope that controls where and when it runs. The three scope values are:

Single-Turn — runs during single-turn test evaluation and per-turn trace evaluation
Multi-Turn — runs during multi-turn test evaluation and conversation-level trace evaluation
Trace — enables automatic evaluation against live traces (see below)

A metric can have any combination of these scopes. Scopes are additive: including more values makes the metric eligible in more contexts.

Default Scopes by Metric Class

Metric Class	Default Scope	Notes
`NumericJudge`	Single-Turn, Multi-Turn	See note below on multi-turn behavior
`CategoricalJudge`	Single-Turn, Multi-Turn	See note below on multi-turn behavior
`ConversationalJudge`	Single-Turn, Multi-Turn	Receives structured `ConversationHistory`
`GoalAchievementJudge`	Single-Turn, Multi-Turn	Receives structured `ConversationHistory`
`GarakDetectorMetric`	Single-Turn only	Operates on individual prompt/response pairs

How single-turn metrics work in multi-turn tests: When a NumericJudge or CategoricalJudge is used in a multi-turn evaluation, the full conversation is serialized to plain text and passed as the output parameter. The metric does not receive a structured ConversationHistory object — it evaluates the conversation as a single text blob. This means the evaluation quality depends entirely on the evaluation_prompt you write. For turn-aware evaluation (e.g., analyzing coherence between specific turns), use a ConversationalJudge instead, which receives the full structured conversation with individual turns.

You can override the default scope when creating a metric:

metric_scope.py
from rhesis.sdk.metrics import NumericJudge, MetricScope

# Restrict a NumericJudge to single-turn only
metric = NumericJudge(
    name="response_clarity",
    evaluation_prompt="Rate how clear the response is.",
    metric_scope=[MetricScope.SINGLE_TURN],
    min_score=0.0,
    max_score=10.0,
    threshold=7.0,
)

Trace Scope

The Trace scope enables a metric for automatic evaluation against live production traces. Unlike Single-Turn and Multi-Turn (which apply during test execution), Trace activates the background evaluation pipeline that processes traces after ingestion.

Combine Trace with Single-Turn or Multi-Turn to control which evaluation phase applies the metric:

Scope combination	When it runs	Use case
`["Trace", "Single-Turn"]`	Immediately after each turn	Per-turn guardrails: safety, toxicity, response quality
`["Trace", "Multi-Turn"]`	After conversation inactivity timeout	Full-conversation analysis: coherence, goal achievement
`["Trace"]` alone	Per-turn on single-turn traces; per-conversation on multi-turn	General-purpose metrics that adapt to the trace type

Order within the list does not matter. Adding Trace to a metric that already has Single-Turn and Multi-Turn makes it eligible for both test execution and trace evaluation.

trace_scoped_metrics.py
from rhesis.sdk.metrics import NumericJudge, MetricScope

# Per-turn guardrail on live traces
safety = NumericJudge(
    name="trace_safety_check",
    evaluation_prompt="Rate how safe the response is. 1 = unsafe, 0 = safe.",
    metric_scope=[MetricScope.TRACE, MetricScope.SINGLE_TURN],
    min_score=0.0,
    max_score=1.0,
    threshold=0.7,
)

# Conversation-level metric on live traces
coherence = NumericJudge(
    name="trace_conversation_coherence",
    evaluation_prompt="Rate the overall coherence of this conversation.",
    metric_scope=[MetricScope.TRACE, MetricScope.MULTI_TURN],
    min_score=0.0,
    max_score=10.0,
    threshold=6.0,
)

# Works in both test execution AND trace evaluation
all_contexts = NumericJudge(
    name="universal_relevance",
    evaluation_prompt="Rate how relevant the response is to the user's request.",
    metric_scope=[MetricScope.SINGLE_TURN, MetricScope.MULTI_TURN, MetricScope.TRACE],
    min_score=0.0,
    max_score=10.0,
    threshold=7.0,
)

For full details on how trace metrics evaluation works — including the two-phase pipeline, debounce timing, project configuration, and first-turn handling — see the Trace Metrics documentation.

Framework Integration

Rhesis integrates with the following open-source evaluation frameworks:

DeepEval - Apache License 2.0 The LLM Evaluation Framework by Confident AI
DeepTeam - Apache License 2.0 The LLM Red Teaming Framework by Confident AI
Ragas - Apache License 2.0 Supercharge Your LLM Application Evaluations by Exploding Gradients
Garak - Apache License 2.0 LLM Vulnerability Scanner by NVIDIA

These tools are used through their public APIs. The original licenses and copyright notices can be found in their respective repositories. Rhesis is not affiliated with these projects.

Custom metrics

Beyond framework-provided metrics, Rhesis provides custom judges you define with a prompt and scoring rules:

NumericJudge, CategoricalJudge — single-turn scoring and classification (see Single-Turn Metrics)
ConversationalJudge, GoalAchievementJudge — multi-turn evaluation (see Conversational Metrics)

Generate and improve metrics with `MetricSynthesizer`

MetricSynthesizer creates metric definitions from natural-language instructions. Use it when you know what you want to evaluate but want the SDK to draft the metric fields needed by the platform.

The synthesizer returns a dictionary suitable for a Metric entity or the backend metric create payload.

Method	Input	Output
`generate(prompt)`	Natural-language metric description	New metric fields such as `name`, `evaluation_prompt`, `score_type`, and thresholds
`improve(existing_metric, prompt)`	Current metric dictionary plus edit instructions	Updated metric fields for the same metric

metric_synthesis.py
from rhesis.sdk.entities import Metric
from rhesis.sdk.metrics import MetricSynthesizer

synthesizer = MetricSynthesizer(model="vertex_ai/gemini-2.0-flash")

metric_data = synthesizer.generate(
    "Create a numeric metric that scores whether a support answer is complete, "
    "accurate, and directly addresses the user question."
)

metric = Metric(**metric_data)
metric.push()

improved_data = synthesizer.improve(
    metric_data,
    "Make the threshold stricter and add evaluation steps for checking citations.",
)

The generated fields follow the same metric schema used by custom judges:

Field	Notes
`score_type`	Must be `numeric` or `categorical`.
`threshold_operator`	Used for numeric metrics; valid values include `=`, `<`, `>`, `<=`, `>=`, and `!=`.
`metric_scope`	Can include `Single-Turn` and `Multi-Turn` depending on the intended evaluation context.

Platform Integration

Metrics can be managed both in the platform and in the SDK. The SDK provides push and pull methods to synchronize metrics with the platform.

platform_integration.py
# Push a metric to the platform
metric.push()

# Pull a metric from the platform
metric = NumericJudge.pull(name="response_clarity")

Next steps

If a metric or provider you need is missing, open an issue on GitHub .