Single-Turn Metrics

Overview

Single-turn metrics evaluate individual exchanges between user input and system output. These metrics are ideal for assessing the quality of standalone responses, RAG systems, and classification tasks.

API Key Required: All examples in this documentation require a valid Rhesis API key. Set your API key using:

setup.py
import os
os.environ["RHESIS_API_KEY"] = "your-api-key"

For more information, see the Installation & Setup guide.

Rhesis integrates with the following open-source evaluation frameworks:

DeepEval - Apache License 2.0
The LLM Evaluation Framework by Confident AI
DeepTeam - Apache License 2.0
The LLM Red Teaming Framework by Confident AI
Ragas - Apache License 2.0
Supercharge Your LLM Application Evaluations by Exploding Gradients

These tools are used through their public APIs. The original licenses and copyright notices can be found in their respective repositories. Rhesis is not affiliated with these projects.

Supported Metrics

DeepEval Metrics

Metric	Description	Requires Context	Requires Ground Truth	Reference
`DeepEvalAnswerRelevancy`	Measures answer relevance to the question	No	No	Docs
`DeepEvalFaithfulness`	Checks if answer is grounded in context	Yes	No	Docs
`DeepEvalContextualRelevancy`	Evaluates context relevance to question	Yes	No	Docs
`DeepEvalContextualPrecision`	Measures precision of retrieved context	Yes	Yes	Docs
`DeepEvalContextualRecall`	Measures recall of retrieved context	Yes	Yes	Docs
`DeepEvalBias`	Detects biased content in responses	No	No	Docs
`DeepEvalToxicity`	Detects toxic content in responses	No	No	Docs
`DeepEvalPIILeakage`	Detects personally identifiable information	No	No	Docs
`DeepEvalRoleViolation`	Detects when assistant violates assigned role	No	No	Docs
`DeepEvalMisuse`	Detects potential misuse of the system	No	No	Docs
`DeepEvalNonAdvice`	Ensures assistant doesn’t give restricted advice	No	No	Docs

DeepTeam Metrics

Metric	Description	Requires Context	Requires Ground Truth	Reference
`DeepTeamSafety`	Detects safety violations	No	No	Docs
`DeepTeamIllegal`	Detects illegal content or requests	No	No	Docs

Ragas Metrics

Metric	Description	Requires Context	Requires Ground Truth	Reference
`RagasContextRelevance`	Evaluates context relevance to question	Yes	No	Docs
`RagasAnswerAccuracy`	Measures answer accuracy against ground truth	No	Yes	Docs
`RagasFaithfulness`	Checks if answer is grounded in context	Yes	No	Docs
`RagasAspectCritic`	Custom aspect-based evaluation	No	No	Docs

Rhesis Custom Metrics

Metric	Description	Configuration
`NumericJudge`	LLM-based numeric scoring (e.g., 0-10 scale)	Min/max score, threshold, custom prompts
`CategoricalJudge`	LLM-based categorical classification	Categories, passing categories, custom prompts

If any metrics are missing from the list, or you would like to use a different provider, please let us know by creating an issue on GitHub .

Quick Start

Using DeepEval Metrics

deepeval_example.py
from rhesis.sdk.metrics import DeepEvalAnswerRelevancy

# Initialize metric
metric = DeepEvalAnswerRelevancy(threshold=0.7)

# Evaluate
result = metric.evaluate(
    input="What is the capital of France?",
    output="The capital of France is Paris."
)

print(f"Score: {result.score}")
print(f"Passed: {result.details['is_successful']}")

Using Ragas Metrics

ragas_example.py
from rhesis.sdk.metrics import RagasFaithfulness

# Initialize metric
metric = RagasFaithfulness(threshold=0.8)

# Evaluate with context
result = metric.evaluate(
    input="What is photosynthesis?",
    output="Photosynthesis is the process by which plants convert light into energy.",
    context=[
        "Photosynthesis occurs in chloroplasts...",
        "Plants use sunlight to produce glucose..."
    ]
)

print(f"Score: {result.score}")
print(f"Passed: {result.details['is_successful']}")

Creating Custom Metrics

You can create custom metrics using the NumericJudge and CategoricalJudge classes.

Numeric Judge

NumericJudge returns a numeric score (e.g., from 0 to 10) and requires four specific parameters: min_score, max_score, threshold, and threshold_operator.

numeric_judge.py
from rhesis.sdk.metrics import NumericJudge

# Define custom numeric metric
metric = NumericJudge(
    name="response_clarity",
    evaluation_prompt="Rate how clear and understandable the response is.",
    evaluation_steps="""
                    1. Check sentence structure
                    2. Evaluate word choice
                    3. Assess overall clarity""",
    min_score=0.0,
    max_score=10.0,
    threshold=7.0,
)

# Evaluate
result = metric.evaluate(
    input="Explain quantum computing",
    output="Quantum computers use qubits to process information...",
    expected_output="A quantum computer uses quantum mechanics...",
)

Categorical Judge

CategoricalJudge returns a categorical value and requires you to specify categories and passing_categories.

categorical_judge.py
from rhesis.sdk.metrics import CategoricalJudge

# Define custom categorical metric
metric = CategoricalJudge(
    name="tone_classifier",
    evaluation_prompt="Classify the tone of the response.",
    categories=["professional", "casual", "technical", "friendly"],
    passing_categories=["professional", "technical"]
)

# Evaluate
result = metric.evaluate(
    input="Describe machine learning",
    output="Machine learning is a subset of AI...",
    expected_output="ML enables systems to learn from data...",
)

print(f"Category: {result.score}")
print(f"Passed: {result.details['is_successful']}")

Understanding Results

All metrics return a MetricResult object:

metric_results.py
result = metric.evaluate(input="...", output="...")

# Access score
# Numeric score or categorical value
print(result.score)

# Access details
print(result.details)
# {
#     'score': 0.85,
#     'reason': 'The response is highly relevant...',
#     'is_successful': True,
#     'threshold': 0.7,
#     'score_type': 'numeric'
# }

Configuring Models

All metrics require an LLM model to perform the evaluation. If no model is specified, the default model will be used. You can specify the model using the model argument.

For more information about models, see the Models Documentation.

model_config.py
from rhesis.sdk.metrics import DeepEvalAnswerRelevancy
from rhesis.sdk.models import get_model

# Use specific model
model = get_model("gemini")
metric = DeepEvalAnswerRelevancy(threshold=0.7, model=model)

# Or pass model name directly
metric = DeepEvalAnswerRelevancy(threshold=0.7, model="gpt-4")

Advanced Configuration

Serialization

Custom metrics can be serialized and deserialized using the from_config/to_config or from_dict/to_dict methods.

serialization.py
metric = NumericJudge(
    name="response_clarity",
    evaluation_prompt="Rate how clear and understandable the response is.",
    evaluation_steps="""
                    1. Check sentence structure
                    2. Evaluate word choice
                    3. Assess overall clarity""",
    min_score=0.0,
    max_score=10.0,
    threshold=7.0,
)

config = metric.to_config()
metric = NumericJudge.from_config(config)

Platform Integration

Metrics can be managed both in the platform and in the SDK. The SDK provides push and pull methods to synchronize metrics with the platform.

Pushing Metrics

To push a metric to the platform:

push_metric.py
metric = NumericJudge(
    name="response_clarity",
    description="Rate how clear and understandable the response is.",
    metric_type="classification",
    requires_ground_truth=True,
    requires_context=False,
    evaluation_prompt="Rate how clear and understandable the response is.",
    evaluation_steps="""
                    1. Check sentence structure
                    2. Evaluate word choice
                    3. Assess overall clarity""",
    min_score=0.0,
    max_score=10.0,
    threshold=7.0,
)
metric.push()

Pulling Metrics

To pull metrics from the platform, use the pull method and specify the metric name. If the name is not unique, you must also specify the metric ID.

pull_metric.py
metric = NumericJudge.pull(name="response_clarity")