Trace Metrics

Trace Metrics let you run evaluation metrics against live production traces automatically. Every time your endpoint handles a request, Rhesis can evaluate the response for safety, relevance, coherence, or any custom criteria you define — with no extra test cases.

When Evaluation Runs

Trace metrics evaluate at two levels depending on the type of trace.

Per-turn evaluation runs immediately after each request, with the project’s Single-Turn metrics. This is where you catch issues in real time: safety violations, off-topic responses, hallucinations. Every trace gets per-turn evaluation.

Conversation evaluation runs after a multi-turn conversation goes quiet (default: 5 minutes of inactivity). This is where you assess the conversation as a whole: whether the assistant stayed coherent, achieved the user’s goal, and retained context across turns. It uses the project’s Multi-Turn metrics. Only multi-turn traces get conversation evaluation.

Note: The inactivity timeout is centrally configurable in the backend via the DEFAULT_CONVERSATION_DEBOUNCE_SECONDS environment variable.

The “First Turn” Edge Case

Often, the very first message in a chat does not yet have a conversation_id attached to it. When this happens:

1. The backend treats it as a single-turn trace and runs the Single-Turn metrics immediately.
2. When the user replies (Turn 2) and a conversation_id is established, the backend recognizes it as a multi-turn conversation.
3. It stops running “Conversation” metrics on individual turns and starts the debounce timer to wait for the conversation to finish.

Setting Up Trace Metrics

1. Pick or create metrics

Any metric can evaluate traces. You don’t need a special scope. The metric’s scope decides which evaluation phase uses it on conversation traces:

A trace without a conversation_id only gets per-turn evaluation, so a Multi-Turn-only metric never runs on it.

from rhesis.sdk.metrics import NumericJudge, MetricScope

# Runs immediately after each turn. Good for guardrails.
safety = NumericJudge(
    name="safety_check",
    evaluation_prompt="Rate how safe and appropriate the response is.",
    metric_scope=[MetricScope.SINGLE_TURN],
    min_score=0.0,
    max_score=1.0,
    threshold=0.7,
)
safety.push()

# Runs after the conversation ends. Good for overall quality.
coherence = NumericJudge(
    name="conversation_coherence",
    evaluation_prompt="Rate the overall coherence of this conversation.",
    metric_scope=[MetricScope.MULTI_TURN],
    min_score=0.0,
    max_score=10.0,
    threshold=6.0,
)
coherence.push()

2. Add the metrics to your project

A trace is only evaluated with the metrics its project lists. A project with no trace metrics evaluates nothing.

1. Open Projects and select your project
2. In the Trace Metrics section, click Add Metric
3. Select one or more metrics
4. Use row selection + Remove metrics for bulk removal

2b. Advanced project attributes (API / internal)

The project stores its trace metrics in attributes.trace_metrics, either as a list of metric IDs or as a config object:

{
  "trace_metrics": {
    "enabled": true,
    "metric_ids": [
      "uuid-1",
      "uuid-2"
    ],
    "sampling_rate": 1
  }
}

3. Deploy your endpoint

No changes needed in your application code. If your endpoint is already sending traces to Rhesis, evaluation starts as soon as the project has trace metrics.

Viewing Results

Traces Table

The Traces dashboard includes an Evaluation column showing the overall status for each trace: Pass, Fail, or a dash when no evaluation has run yet.

Trace Drawer

When you click on a trace that has evaluation results, a Trace Metrics tab appears in the detail drawer with two sections:

• Turn Metrics — per-turn results for the selected span, showing each metric’s score, pass/fail status, and the evaluator’s reasoning
• Conversation Metrics — full-conversation results shared across all spans, shown only for multi-turn traces

Trace Annotations tab

Trace details also include an Annotations tab for human overrides on live traces:

• Trace target — review the overall trace verdict
• Metric target — review a specific metric result
• Turn target — review a specific conversation turn

The review drawer enforces:

• Pass/Fail selection
• Comment validation (minimum comment length)
• Optional @ mentions for metrics and turns to infer review target

When a review is saved, Rhesis stores both:

• the original automated outcome, and
• the human override metadata

For each target, the UI surfaces latest review status and conflict markers when human and automated verdicts differ.

How Overall Status Is Determined

Each trace receives an overall status of Pass, Fail, or Error based on its metric results. This status appears in the Evaluation column of the Traces dashboard and powers the evaluation filter.

Single-turn traces

After per-turn evaluation completes, the status is derived from the turn metrics:

Multi-turn traces (conversation)

Multi-turn traces go through two evaluation phases. The final status reflects all metrics combined:

1. Phase 1 (per-turn): Turn metrics run immediately and set an initial status.
2. Phase 2 (conversation): After the inactivity timeout, conversation metrics run. The backend then merges turn metrics and conversation metrics together and re-derives the status from the combined set.

This means a single failing turn metric causes the overall trace to show Fail, even if all conversation-level metrics pass.

When status is not set

If the evaluation task itself fails (for example, the evaluation model is unavailable), the task retries up to three times. If all retries are exhausted, no status is written and the trace shows a dash in the Evaluation column. These traces are not returned by the Pass, Fail, or Error filters.

Using Trace Metrics with Test Metrics

The same metric can run in test execution and on live traces. A metric scoped Single-Turn and Multi-Turn runs on both kinds of tests, and once added to a project it also evaluates that project’s traces.