Running and Analyzing
After you approve the plan, Architect runs the tests and presents results in a structured format. You can also ask it to analyze existing runs or compare two runs directly.
Running tests
When creation is complete, Architect offers to run the tests. Confirm, and it submits the test set to your endpoint. The mode chip switches to executing while it waits.
You don’t need to create a test configuration manually. Architect handles the setup and monitors the run automatically — there is no polling or waiting on your part.
When the run completes, Architect presents the results.
Results format
Results come in three layers, each more specific than the last:
Overall — a single pass/fail percentage for the full run. Sets the context before diving in.
Per-requirement — pass rate for each requirement you tested. Requirements with a rate below threshold are flagged immediately.
Per-metric — within each requirement, how each metric scored. Lets you see whether a requirement failed on one specific criterion (e.g., tone) while passing others (e.g., accuracy).
Notable failures — the worst-performing tests, with the evaluator’s stated reason for marking them as failing. This tells you why something failed, not just that it did.
Interpreting failure patterns
Architect categorizes failure patterns and surfaces likely causes:
| Pattern | What it means | What to do |
|---|---|---|
| All tests fail (0%) | Endpoint unreachable, response format changed, or auth issue | Check connectivity and endpoint response format |
| Most tests fail in one requirement | That specific requirement needs attention — the endpoint may be inconsistent in this area | Tighten the requirement description or review the endpoint’s handling |
| Single metric fails across requirements | The evaluation criteria may be miscalibrated, or there’s a genuine weakness | Review the metric threshold; check if failures cluster on a topic |
| Tests fail on a narrow topic cluster | The endpoint has a knowledge gap | Consider targeted improvements or additional training data |
| Borderline pass rates (50–70%) | The endpoint is inconsistent — working sometimes but not reliably | Re-run with more tests to confirm; look for input patterns that trigger the inconsistency |
Comparing two runs
Ask Architect to compare any two runs to see what changed:
"How does the latest run compare to the previous one?"
"Compare run A to run B."Architect retrieves both run results and shows:
- Overall trend — improved, regressed, or unchanged
- Per-requirement changes — which requirements got better or worse
- Per-metric changes — which specific metrics drove the change
- Notable regressions — tests that were passing and now fail
- Notable improvements — tests that were failing and now pass
Analyzing an existing run
You don’t need to run a test yourself to get an analysis. Ask Architect about any past run:
"Analyze run 42."
"What do the results of the last test run look like?"Architect fetches the results and presents the same three-layer summary — overall, requirement breakdown, metric breakdown — plus actionable suggestions based on what it finds.
Architect refers to runs by name or index. You don’t need to supply a run ID.
What people already flagged
Automated scores are not the last word. Architect reads the annotations people left on a run before it reports on it, and treats a human Pass or Fail as ground truth when it contradicts a metric:
"What did people flag on the last run?"
"Any open annotations on this test set?"Two things this changes. A “failure” someone marked Pass is a metric problem rather than an endpoint problem, and Architect says so instead of reporting the raw failure. And a run where every metric passed is not necessarily clean, because a person can fail a test the metrics passed — which is why Architect checks before concluding that nothing is wrong.
It can also record a verdict you give it, attributed to you. It will not annotate on its own initiative: an annotation is your judgement, not the agent’s reading of a result.
Why something failed
The result tells you a test failed and what the evaluator made of it. When that does not explain the failure, Architect opens the trace — what your application actually did to produce the response, as a tree of LLM calls, retrievals and tool invocations with their own durations and statuses:
"Why did that test error?"
"Why was the last run so slow?"
"What did this run cost?"The span that failed, or the one whose duration dominates the rest, is usually the answer. Architect can also report cost and latency percentiles across a run’s traces.
This is a diagnostic step, not part of every analysis. Pass rates, breakdowns and comparisons are answered without traces, and Architect does not open one to produce a run summary — a trace per result would be slow and would tell you nothing the summary does not.