Annotation
A person's verdict on an automated evaluation, recorded against a test result, trace, or test, which overrides the automated pass/fail where one exists.
Overview
An annotation is a person's verdict on something the platform evaluated automatically. On a Test Result or a Trace, a Pass or Fail annotation becomes the effective verdict and the original automated outcome is kept alongside it, so there is always a record of what the system scored before a human weighed in.
Annotations are how domain knowledge gets back into the platform: a metric can be confidently wrong for your domain, and an annotation is what records that, with the reasoning attached.
You can annotate a test result, a trace, a test in the Explorer, or a metric tuning case. An annotation can name a target: the entity as a whole, one Metric within it, or one turn of a multi-turn conversation. See Annotations for what each surface supports and what an override does.
Conflicts and resolution
When a human verdict disagrees with the automation, the platform flags the conflict rather than hiding it. Mark the annotation Resolved once the disagreement has been handled, and reopen it if it needs attention again.
Annotations are per person and timestamped, so several people can judge the same result and each judgement is kept and attributed.
Reading them back
Every annotation in a project is listed under Improve → Annotations. They are also a first-class entity in the SDK and over MCP, so an evaluation script or a coding agent can read what the team decided and act on it.
Best Practices
- Always add a comment. The verdict says what you decided; the comment is what a teammate, and metric tuning, can act on
- Target the metric, not the result, when only one criterion was scored wrongly
- Resolve once handled, so the open list stays a list of real work
- Treat repeated corrections as a metric problem, not a labelling chore: the same override appearing across runs is a sign the metric needs tuning