Annotations
Browse human reviews across test results and traces from Improve → Annotations.
Annotations are the same judgements you create on a test result or trace — Pass/Fail with a comment — collected in one list so you can find and follow up on them without hunting through individual runs.
Why they matter
Automated metrics are a useful signal, but they miss context only humans have: brand tone, policy nuance, or a “failure” that was actually the right refusal. Annotations turn those judgements into durable feedback so the team can improve the agent instead of rediscovering the same issues.
Use them to:
- Correct automated Pass/Fail when the metric was wrong for your domain
- Capture why a response succeeded or failed for teammates and future tuning
- Keep disagreements visible (Open) until the model, prompt, or metric is fixed (Resolved)
- Spot patterns across runs and live traces — not only inside a single result drawer
The hub is the Improve entry point for that work: every open annotation is a candidate improvement.
Using the hub
- Open Improve → Annotations
- Use All / Open / Resolved to filter by resolution status
- Search by comment, annotator, target, behavior, or rating
- Open the filter drawer for type (test result / trace), rating, and target
- Click a row to open the source review in a new tab
| Column | Meaning |
|---|---|
| Type | Test result or trace |
| Target | Output, metric, turn, or trace |
| Rating | Human Pass / Fail |
| Annotator | Who wrote the review |
| Behavior | Linked test behavior (test results only) |
| Status | Open or Resolved |
| Comment | Review text (mentions render as chips) |
Resolve
Mark a review Resolved when the disagreement is handled (for example after fixing the model or accepting the override). Use Reopen if it needs attention again.
Resolve and reopen from the Reviews tab on the test result or trace. Conflicts between automated and human verdicts are hidden once the conflicting review is resolved.
Permissions
You need test result read and/or telemetry read. The list only includes sources you can access.
Related
- Test Reviews — how to add and target reviews on test results
- Test Runs — open a result and use the Reviews tab
- Tracing — live traces and review workflows
- Trace Metrics — automated evaluation on traces