Annotations

An annotation is a person’s judgement recorded against a test result, a trace, or a test. It is its own entity, linked to what it judges by entity_type and entity_id.

A Pass or Fail annotation on a test result or a trace overrides that parent’s automated outcome, so creating one changes what the platform reports for the parent, not only what the annotation says. A trace without metrics has no automated outcome; see Traces for what an annotation does there.

Properties

Leave target_type and target_reference unset for a judgement on the parent as a whole; the backend fills in the right entity-level target for the parent type.

Reading annotations

Start from whatever you already have in hand:

# From the parent entity
test_result.get_annotations()
test.get_annotations()
test_run.get_annotations()     # everything in the run: its results and its traces
test_set.get_annotations()
endpoint.get_annotations()
requirement.get_annotations()  # on the results of tests linked to it
trace.get_annotations()        # on the trace and on every span in it

# A metric reaches the judgements people left about it
metric = NumericJudge.pull(name="Answer Relevancy")
metric.get_annotations()

There is no Metric entity class: a platform metric in the SDK is the judge itself, from rhesis.sdk.metrics, and get_annotations hangs off that.

Or query the collection directly when you have an id rather than an object:

from rhesis.sdk.entities import Annotations

Annotations.for_entity("TestResult", test_result_id)
Annotations.for_trace_id(trace_id)          # the OTEL hex; covers every span
Annotations.for_trace(trace_db_id)          # one span, by its row id
Annotations.for_test_run(test_run_id)
Annotations.for_test_set(test_set_id)
Annotations.for_endpoint(endpoint_id)
Annotations.for_metric("Answer Relevancy")  # by name or id; see the note below
Annotations.for_requirement(requirement_id)
Annotations.for_annotator(user_id)
Annotations.for_date_range(date_from="2026-01-01")

for annotation in Annotations.for_test_run(test_run_id):
    print(annotation.status.name, annotation.user.name, annotation.comments)

These read every page rather than the first one, so a parent with more annotations than a single page returns whole.

for_metric takes a name or an id and returns both kinds of judgement about that metric: the ones left on a metric within a test result or trace, which name the metric, and the metric tuning ones, which name its id so that renaming the metric does not orphan them. The tuning ones are the judgements about whether the metric itself is any good, so asking by name and getting only the first kind would be the wrong answer.

Recording a judgement

Annotate the thing you have in hand. The verdict is named, not looked up.

# On the whole result
test_result.annotate("fail", "The answer cites a policy that does not exist.")

# On one metric within it, leaving the others on their automated verdicts
test_result.annotate("pass", "Relevant after all.", metric="Answer Relevancy")

# On one turn of a multi-turn conversation
test_result.annotate("fail", "Went off script here.", turn=2)

# Labelling a test, the way a person labels one in the Explorer
test.annotate("fail", "Not a fair question.")

"pass" and "fail" are resolved to the organization’s status rows for you, and cached. Verdict and AnnotatableEntity hold the same values as constants if you would rather not type strings.

Metric tuning judgements (accepting or rejecting how a metric scored a test) are made in the platform’s tuning flow, which records the verdict each judgement was made against. The SDK refuses to create one, since without that record tuning would treat it as out of date straight away.

A turn is given as its number, from test_output.conversation_summary, and stored as the label the platform groups judgements by:

test_result.annotate("fail", "...", turn=2)          # stored as "Turn 2"
test_result.annotate("fail", "...", turn="Turn 2")    # the same thing

Both land on the same target. Passing a bare "2" would not: judgements are grouped by that reference, so it would sit beside the turn’s other annotations instead of superseding them, which is why the number is normalised here rather than sent as given.

A trace is annotated the same way, from the Trace object. See Traces.

When you have an id rather than an object, use the collection. Constants are available where a literal would be easy to mistype:

from rhesis.sdk.entities import AnnotatableEntity, Annotations, Verdict

Annotations.create(AnnotatableEntity.TEST_RESULT, test_result_id, Verdict.FAIL, "Off topic.")

Annotating a trace from the application that produced it

An instrumented application knows a trace by its OTEL trace id, the 32-character hex its tracer generated. annotate_trace takes that id, and the platform finds the trace for you.

from rhesis.sdk.telemetry import annotate_current_trace, annotate_trace

# Inside the instrumented call, or after it returns
annotate_current_trace("fail", "Cited a document that does not exist.")

# Or naming the trace, e.g. from a thumbs-down handler that stored the id
annotate_trace(trace_id, "fail", "User reported the answer was wrong.")

This is how feedback from outside the platform gets recorded: a thumbs-down from an end user, a QA harness, an eval script. It lands as an ordinary annotation and appears wherever annotations already do. A Pass/Fail verdict on the whole trace overrides its automated outcome, or becomes its verdict when the trace has no metrics. metric and turn work the same way as above.

annotate_current_trace reads the trace id from the tracer, so it works after the span has closed, which is when a human verdict usually arrives. It raises if there is no trace in context rather than filing the verdict against nothing.

Spans are ingested asynchronously, so a trace recorded moments ago may not be queryable yet; that case raises too, and retrying is the right response. Waiting for the request to finish first leaves more time for the spans to arrive but does not guarantee they have, so a call made close to the request should be ready to retry.

Outside an instrumented app, the collection takes the same hex, for writing and for reading back:

from rhesis.sdk.entities import Annotations

Annotations.create_for_trace_id(trace_id, "fail", "Retrieved the wrong document.")
Annotations.for_trace_id(trace_id)

for_trace_id covers annotations on any span of the trace, not only the root, since the question is about the trace rather than one operation inside it.

From an annotation back to its trace

get_trace returns the trace an annotation sits on, whether it judges the whole trace or one span in it. It returns None for an annotation on a test result or a test.

trace = annotation.get_trace()
if trace is not None:
    print(trace.root_operation, trace.duration_ms)
    trace.span("ai.llm.invoke")

Each call reads the trace with its full span tree, which is the large response described in Traces. Avoid calling it for every annotation in a run.

Resolving and deleting

annotation.resolve()   # the disagreement has been handled
annotation.reopen()    # it needs attention again

# Deleting reverts the override it applied to the parent
annotation.delete()

A resolve sends only that change, so a local edit to a comment cannot ride along with it.

Building an annotation by hand

annotate and create cover the common cases. Construct an Annotation directly when you need a field they do not take, such as attributes:

from rhesis.sdk.entities import Annotation, Statuses

Annotation(
    entity_type="TestResult",
    entity_id=test_result_id,
    status_id=Statuses.pull(name="Fail").id,
    comments="Recorded by the nightly eval job.",
    attributes={"source": "nightly"},
).push()

The write endpoints take the target nested under target, while responses return it flat as target_type and target_reference. push handles that remap, so set the flat pair and it arrives correctly.