Overview

The Rhesis SDK provides a comprehensive metrics system for evaluating LLM-based systems. The metrics module supports multiple evaluation frameworks and allows you to create custom metrics tailored to your specific use cases. The metrics module is integrated with the backend, allowing you to work with metrics directly from the platform.

Metric Types

Rhesis SDK supports two types of metrics:

Single-Turn Metrics

Single-turn metrics evaluate individual exchanges between user input and system output. These metrics are ideal for assessing:

• RAG Systems: Context relevance, faithfulness, and answer accuracy
• Response Quality: Clarity, relevance, and accuracy of individual responses
• Safety & Compliance: Bias, toxicity, PII leakage, and other safety concerns
• Custom Evaluations: Domain-specific quality assessments

View Single-Turn Metrics Documentation →

Conversational Metrics

Conversational metrics (multi-turn metrics) evaluate the quality of interactions across multiple conversation turns. These metrics are ideal for assessing:

• Conversation Flow: Turn relevancy and coherence across dialogue
• Goal Achievement: Whether objectives are met throughout the conversation
• Role Adherence: Consistency in maintaining assigned roles
• Knowledge Retention: Ability to recall and reference earlier conversation context
• Tool Usage: Appropriate selection and utilization of available tools
• Conversation Completeness: Whether conversations reach satisfactory conclusions

View Conversational Metrics Documentation →

Metric Scopes

Every metric has a metric_scope that controls where and when it runs. The three scope values are:

• Single-Turn — runs during single-turn test evaluation and per-turn trace evaluation
• Multi-Turn — runs during multi-turn test evaluation and conversation-level trace evaluation
• Trace — enables automatic evaluation against live traces (see below)

A metric can have any combination of these scopes. Scopes are additive: including more values makes the metric eligible in more contexts.

Default Scopes by Metric Class

You can override the default scope when creating a metric:

from rhesis.sdk.metrics import NumericJudge, MetricScope

# Restrict a NumericJudge to single-turn only
metric = NumericJudge(
    name="response_clarity",
    evaluation_prompt="Rate how clear the response is.",
    metric_scope=[MetricScope.SINGLE_TURN],
    min_score=0.0,
    max_score=10.0,
    threshold=7.0,
)

Trace Scope

The Trace scope enables a metric for automatic evaluation against live production traces. Unlike Single-Turn and Multi-Turn (which apply during test execution), Trace activates the background evaluation pipeline that processes traces after ingestion.

Combine Trace with Single-Turn or Multi-Turn to control which evaluation phase applies the metric:

Order within the list does not matter. Adding Trace to a metric that already has Single-Turn and Multi-Turn makes it eligible for both test execution and trace evaluation.

from rhesis.sdk.metrics import NumericJudge, MetricScope

# Per-turn guardrail on live traces
safety = NumericJudge(
    name="trace_safety_check",
    evaluation_prompt="Rate how safe the response is. 1 = unsafe, 0 = safe.",
    metric_scope=[MetricScope.TRACE, MetricScope.SINGLE_TURN],
    min_score=0.0,
    max_score=1.0,
    threshold=0.7,
)

# Conversation-level metric on live traces
coherence = NumericJudge(
    name="trace_conversation_coherence",
    evaluation_prompt="Rate the overall coherence of this conversation.",
    metric_scope=[MetricScope.TRACE, MetricScope.MULTI_TURN],
    min_score=0.0,
    max_score=10.0,
    threshold=6.0,
)

# Works in both test execution AND trace evaluation
all_contexts = NumericJudge(
    name="universal_relevance",
    evaluation_prompt="Rate how relevant the response is to the user's request.",
    metric_scope=[MetricScope.SINGLE_TURN, MetricScope.MULTI_TURN, MetricScope.TRACE],
    min_score=0.0,
    max_score=10.0,
    threshold=7.0,
)

Framework Integration

Quick Example

import os
os.environ["RHESIS_API_KEY"] = "your-api-key"

Single-Turn Evaluation

from rhesis.sdk.metrics import DeepEvalAnswerRelevancy

# Initialize metric
metric = DeepEvalAnswerRelevancy(threshold=0.7)

# Evaluate a single response
result = metric.evaluate(
    input="What is the capital of France?",
    output="The capital of France is Paris."
)

print(f"Score: {result.score}")
print(f"Passed: {result.details['is_successful']}")

Conversational Evaluation

from rhesis.sdk.metrics import DeepEvalTurnRelevancy, ConversationHistory

# Initialize metric
metric = DeepEvalTurnRelevancy(threshold=0.7)

# Create conversation
conversation = ConversationHistory.from_messages([
    {"role": "user", "content": "What insurance do you offer?"},
    {"role": "assistant", "content": "We offer auto, home, and life insurance."},
    {"role": "user", "content": "Tell me about auto coverage."},
    {"role": "assistant", "content": "Auto includes liability and collision coverage."},
])

# Evaluate the conversation
result = metric.evaluate(conversation_history=conversation)

print(f"Score: {result.score}")
print(f"Passed: {result.details['is_successful']}")

Custom Metrics

In addition to framework-provided metrics, Rhesis offers custom metric builders:

For Single-Turn Evaluation

• NumericJudge: Create custom numeric scoring metrics (e.g., 0-10 scale)
• CategoricalJudge: Create custom categorical classification metrics

from rhesis.sdk.metrics import NumericJudge

metric = NumericJudge(
    name="response_clarity",
    evaluation_prompt="Rate how clear and understandable the response is.",
    min_score=0.0,
    max_score=10.0,
    threshold=7.0
)

For Conversational Evaluation

• ConversationalJudge: Create custom conversational quality metrics
• GoalAchievementJudge: Evaluate goal achievement with custom criteria

from rhesis.sdk.metrics import ConversationalJudge

metric = ConversationalJudge(
    name="conversation_coherence",
    evaluation_prompt="Evaluate the coherence and flow of the conversation.",
    min_score=0.0,
    max_score=10.0,
    threshold=7.0
)

Platform Integration

Metrics can be managed both in the platform and in the SDK. The SDK provides push and pull methods to synchronize metrics with the platform.

# Push a metric to the platform
metric.push()

# Pull a metric from the platform
metric = NumericJudge.pull(name="response_clarity")

Next Steps

• Single-Turn Metrics - Learn about all available single-turn metrics
• Conversational Metrics - Learn about all available conversational metrics
• Trace Metrics - Automatic evaluation on live production traces
• Models Documentation - Configure LLM models for evaluation
• Installation & Setup - Setup instructions

Need Help?

If any metrics are missing from the list, or you would like to use a different provider, please let us know by creating an issue on GitHub .