Checking Tool Calls

The Tool Call Trajectory metric compares the tools an endpoint called with the tools a test expects it to call. It needs no model: it reads the tool calls from the run’s trace and the expected ones from the test.

Giving a test its expected tool calls

A test lists the calls it expects in test_configuration.expected_tool_calls, in order. Each call has a name, and optionally arguments with only the keys that matter. Both single-turn and multi-turn tests can carry them; on a multi-turn test they cover the whole conversation.

tool_tests.py
from rhesis.sdk.entities import Prompt, Test, TestSet

test = Test(
    category="Refunds",
    topic="Late orders",
    requirement="Follows the refund process",
    prompt=Prompt(content="My order 42 arrived late, I want my money back."),
    expected_tool_calls=[
        {"name": "lookup_order", "arguments": {"order_id": 42}},
        {"name": "issue_refund"},
    ],
)

TestSet(name="Refund flow", tests=[test]).push()

Through the API and MCP, send the same list as test_configuration.expected_tool_calls when creating or updating a test. An empty list expects no tool calls at all.

Adding the metric

Every organization has a built-in Tool Call Trajectory metric. Add it to a requirement or a test set like any other metric. Tests without expected tool calls are not judged by it: their cell in the results grid shows as not applicable, and the test’s result comes from its other metrics. A test with no other metric is Inconclusive. Production traces have no expectations, so the metric never runs on them.

The endpoint has to report its tool calls:

SDK endpoints send them as tool spans when the tools are traced.
REST and WebSocket endpoints report them through the tool_calls field of their response mapping.

When no tool calls were recorded, the result says so.

How calls are matched

The metric’s config sets how the calls must line up. The built-in metric uses in_order and partial.

`match`	Passes when
`exact`	The same calls were made in the same order, and no others
`in_order`	The expected calls were made in order; other calls may come between them
`any_order`	The same calls were made in any order, and no others
`subset`	Every call made was expected; some expected calls may be missing
`superset`	Every expected call was made; other calls are allowed

`arguments`	A call’s arguments match when
`partial`	Every listed key has the listed value; other keys are ignored
`exact`	They equal the listed arguments
`ignore`	Always; only the tool’s name is compared

A call listed without arguments matches any arguments. The result scores 1 on a match and 0 otherwise, names the first call that differs (“Expected lookup_order; it was never called.”), and reports precision and recall: the share of calls made that were expected, and the share of expected calls that were made.

Running it locally

trajectory.py
from rhesis.sdk.metrics import ToolCallTrajectory

metric = ToolCallTrajectory(match="in_order", arguments="partial")
result = metric.evaluate(
    tool_calls=[
        {"name": "lookup_order", "arguments": {"order_id": 42}},
        {"name": "issue_refund", "arguments": {"order_id": 42}},
    ],
    expectations={"tool_calls": [{"name": "lookup_order"}, {"name": "issue_refund"}]},
)
print(result.score, result.details["reason"])

For a check the modes can’t express, write a code metric that reads trace.tool_calls.