Metrics

Define and manage evaluation criteria for testing AI responses with LLM-based grading.

Metrics at a glance

Metrics are organized by Behaviors, which are atomic expectations you have for the output of your application.

In short: Behaviors define what you expect, metrics measure how well you meet those expectations.

Offline Tests vs Online Trace Metrics

Metrics can be used in two different contexts:

1. Offline Evaluation (Testing): Grade deterministic outputs against known ground-truths or predefined test sets.
2. Online Evaluation (Trace Metrics): Automatically evaluate live production traces in real-time as users interact with your AI endpoint.

To learn how to enable metrics for live production traffic, read the Trace Metrics Guide.

You can add an existing metric to a behavior by clicking the ”+” icon in the top-right corner of the chip on the Metrics Overview page.

Using Existing Metrics

The platform includes pre-built metrics from multiple providers:

• DeepEval  - The LLM Evaluation Framework by Confident AI
• DeepTeam  - Team-based evaluation framework by Confident AI
• Ragas  - Supercharge Your LLM Application Evaluations by Exploding Gradients
• Rhesis - Custom metrics developed by the Rhesis team

Here are some examples:

Creating Custom Metrics

You can create custom metrics tailored to your specific evaluation needs.

This section presents what needs to be configured to create a custom metric using a Judge-as-Model appraoch.

1. Evaluation Process

Define how the LLM evaluates responses by providing four key components:

Example: "Evaluate whether the response is accurate, complete, and relevant.
Use the following criteria: accuracy of facts, coverage of required points,
clarity of explanation."

Example:
1. Check Accuracy: Identify any factual errors or unsupported statements
2. Check Coverage: Determine if all required elements are addressed
3. Check Clarity: Assess whether the response is clear and well-structured

Example: "Extract key claims, assess completeness and correctness,
weigh issues by importance, then determine the final score."

Best Practices:

• Focus on one evaluation dimension (accuracy, tone, safety, etc.)
• Use concrete, measurable criteria
• Provide clear examples when possible

Evaluating Context, Metadata, and Tool Calls

Metrics can also take into account context, metadata, and tool_calls. These are Rhesis-managed keys providing additional information for a given interaction alongside the standard text response.

• context: Background information, retrieved documents, or system prompts provided to the model.
• metadata: Structured data returned by your endpoint (e.g., token usage, latency, model version, confidence scores, or custom fields).
• tool_calls: External functions or tools invoked by the AI during the interaction.

How it works:

1. Configure your endpoint’s response mapping to extract a metadata field from the API response (e.g., "metadata": "$.usage")
2. Reference the metadata in your metric’s Evaluation Prompt — the JSON object will be available to the evaluation model automatically

Example 1: Evaluating Context Groundedness

Evaluation Prompt:

Evaluate whether the response is fully grounded in the provided context.
Check the `context` array. Any claim in the output that is not supported
by the `context` should result in a lower score.

Example 2: Evaluating Token Efficiency (Metadata)

Endpoint response mapping:

{
  "output": "$.choices[0].message.content",
  "metadata": "$.usage"
}

Evaluation Prompt:

Evaluate whether the response is efficient in token usage.
Check the `metadata` for token counts. A response that uses more than
500 tokens for a simple factual question should score lower.

Common use cases:

• RAG Groundedness (context) — check that the model’s answer relies only on retrieved knowledge
• Tool Correctness (tool_calls) — verify the AI chose the right tool and passed the correct arguments
• Token usage (metadata) — evaluate cost efficiency by checking prompt_tokens and completion_tokens
• Confidence scores (metadata) — verify the model’s self-reported confidence aligns with response quality

2. Score Configuration

Choose your scoring approach:

3. Metric Scope

Select applicable scopes (at least one required):

• Single-Turn: Individual question-answer pairs
• Multi-Turn: Multi-exchange conversations
• Trace: Live production traffic evaluated in real-time (see Trace Metrics)

You can find more information about test types in the Tests Section.

4. Result Explanation

Provide instructions for how the LLM should explain its scoring rationale.

Example: "Explain which specific parts were accurate or inaccurate, citing evidence from the provided context."

See Also

• Learn how to use metrics programmatically in the SDK Metrics Documentation