Code Metrics

Code Metrics let you write evaluation logic directly in Python — on your own machine or infrastructure — and have the Rhesis backend invoke it automatically during test execution. Instead of relying on an LLM-as-a-judge, you implement the scoring logic yourself: call a local model, apply business rules, query an internal service, or use any library available in your environment.

When to Use Code Metrics

Code metrics are a good fit when:

• You have deterministic rules or heuristics that don’t need an LLM to evaluate (e.g. checking for required keywords, format validation, regex patterns).
• You want to use a locally hosted model (e.g. a fine-tuned classifier or embedding model) without routing traffic through the Rhesis API.
• You need to call internal services or databases that are only accessible from your network.
• You want full control over scoring logic and don’t want to rely on prompt engineering for evaluation.

For general-purpose quality evaluation — coherence, relevance, helpfulness — LLM-based metrics are often faster to set up. Code metrics complement them for domain-specific or deterministic checks.

How It Works

Code metrics run inside your process. The Rhesis backend connects to your running script via the connector (client.connect()), sends it the test inputs, and collects the scores your function returns. No metric code leaves your machine.

Test Run (Backend)  ──► Connector (your machine)  ──► @metric function
                    ◄──                            ◄──  { "score": ... }

Defining a Code Metric

Use the @metric decorator from the SDK. The decorator registers the function with the connector so the backend can call it by name.

Basic Example

from dotenv import load_dotenv
from rhesis.sdk import RhesisClient, metric

load_dotenv()
client = RhesisClient.from_environment()


@metric(name="citation_check", score_type="binary")
def citation_check(input: str, output: str) -> dict:
    """Passes if the response includes at least one citation marker like [1] or (Source:...)."""
    import re
    has_citation = bool(re.search(r"\[\d+\]|\(Source:", output))
    return {
        "score": 1.0 if has_citation else 0.0,
        "details": {"reason": "Response contains a citation." if has_citation else "No citation found."},
    }


if __name__ == "__main__":
    client.connect()

Run the script, and when a test that uses citation_check executes, the backend invokes your function and records the result.

python metrics.py

Decorator Parameters

Function Signature

Your function declares any of four parameters, and receives only those:

No other parameter names are accepted. The decorator raises a TypeError that names the replacement for a parameter metrics used to take, such as trace.context for context or expectations.response for expected_output.

Checking what the endpoint did

With trace, a metric can check the steps behind an answer, not only the answer. Every evaluation has one: when the endpoint sent no spans, it holds the request and response. Each span has a kind ("llm.invoke", "tool.invoke", "retrieval", "agent.invoke", …), an input and an output, and tool spans a tool_name.

The trace also reads as a whole:

from rhesis.sdk import metric
from rhesis.sdk.entities import Trace


@metric(score_type="binary")
def looks_up_the_order_before_refunding(input: str, output: str, trace: Trace) -> dict:
    tools = [call["name"] for call in trace.tool_calls]
    if "issue_refund" not in tools:
        return {"score": 1.0}
    ok = "lookup_order" in tools[: tools.index("issue_refund")]
    return {"score": 1.0 if ok else 0.0, "details": {"reason": f"Tool calls: {tools}"}}

To compare the calls with the ones a test expects, the built-in Tool Call Trajectory metric does it without code.

Which spans a trace holds depends on the endpoint:

• SDK endpoints send their own spans: model calls, tool calls, retrievals and anything else they trace.
• REST and WebSocket endpoints have one span for the call. If their response mapping exposes tool_calls or context, each tool call becomes a tool span under it and the context a retrieval span.
• Services instrumented with OpenTelemetry receive a W3C traceparent header with each call, so the spans they send to Rhesis join the same trace. For a service that rejects unknown headers, set the endpoint’s endpoint_metadata.propagate_trace_context to false through the API.

Return Format

Return a dict with at least a "score" key. An optional "details" dict can carry structured metadata that appears in the Rhesis UI alongside the score.

# Minimal — score only
return {"score": 8.5}

# With details — shown in the platform result drawer
return {
    "score": 8.5,
    "details": {
        "reason": "Response was accurate and well-structured.",
        "flagged_phrases": [],
    },
}

For "binary" metrics, use 1.0 for pass and 0.0 for fail.

Viewing in the Platform

When your script is running and connected (client.connect()), the backend receives the metric registrations over WebSocket and makes them available in the Rhesis UI.

Open Metrics in the sidebar. Code metrics registered from the SDK appear in the list marked with an SDK button, which distinguishes them from LLM-based metrics configured directly in the platform.

You can assign them to Requirements and include them in test sets exactly like any other metric. The only difference is that the metric code runs on your machine — the connector must be active when a test run that uses the metric is executed.