Skip to Content
DocsArchitectScenarios

Scenarios

Common requests and what to expect from each one.

In this page:

In the examples below, U is the user and is Architect.

Full workflow: design and run a test suite

The most common entry point. Describe your endpoint and what you care about.

Example
Test my travel booking chatbot. Focus on accuracy and safety.
U
I’ll explore the endpoint first. Would you prefer a Quick scan (domain only) or Comprehensive (full capability and boundary mapping)?
Quick is fine.
U
Done. The chatbot handles flight search, booking, and cancellations. It refuses travel advice outside its scope. I noticed some uncertainty around partial cancellations — should I include refund edge cases in the test suite?
Yes, include those.
U
Here's my plan: • 3 requirements: Handles Flight Search, Refuses Off-Topic, Handles Cancellations • 2 test sets (30 tests total) • 2 metrics: Accuracy, Refusal Correctness Shall I proceed?
Go ahead.
U

→ See Workflow for what happens after approval.

Direct operations

You can skip the full workflow and ask Architect to do specific things with existing entities.

List test sets:

Show me my test sets.
U
You have 4 test sets: • Flight Search Core — 32 tests • Refusal Suite — 15 tests • Edge Cases — 8 tests • Cancellation Flow — 12 tests

Update a metric:

Make the Accuracy metric stricter — raise the threshold to 0.85.
U
Updated Accuracy — threshold is now 0.85.

Link a requirement to a metric:

Link Handles Cancellations to the Refusal Correctness metric.
U
Done — Handles Cancellations is now evaluated by Refusal Correctness.

Describe a requirement:

What does the Refuses Off-Topic requirement cover?
U
Refuses Off-Topic checks that the chatbot declines requests outside travel booking — such as weather, restaurant advice, or general knowledge questions.

Testing with a knowledge source

Ground test generation in your own documentation. Reference the source by name.

Use our product FAQ to generate tests for the support chatbot.
U
Found “Product FAQ” in your knowledge library. I’ll use it to generate grounded single-turn tests. Shall I plan the full suite?

→ Planning — Knowledge sources covers how grounding works and its single-turn limitation.

Attaching a spec or requirements document

Upload a document and let Architect read it before planning. Works well for OpenAPI specs, requirements docs, or policy PDFs.

[attaches openapi-spec.yaml] Build a test suite based on this API spec.
U
I've read the spec. It defines 8 endpoints across 3 domains. Here's what I'd focus on for testing...

→ Chat Features — File attachments lists supported formats and size limits.

Comparing two test runs

Compare the last two runs.
U
Run 14 vs Run 13 — overall: 78% → 82% (improved) • Handles Flight Search: 90% → 90% (unchanged) • Refuses Off-Topic: 60% → 75% (improved) • Handles Cancellations: 83% → 80% (slight regression) The Refusal Correctness metric drove most of the Off-Topic improvement. Cancellation handling regressed on Accuracy — two tests that were passing now fail.

Analyzing a past run

Analyze the last test run.
U
Run 14 — 78% overall. By requirement: • Handles Flight Search: 90% • Refuses Off-Topic: 60% ← below threshold • Handles Cancellations: 83% Off-Topic refusals are failing primarily on Refusal Correctness. The evaluator notes that the chatbot gives partial answers rather than clean refusals on 6 of the 10 failing tests. Tightening the system prompt for out-of-scope topics should address this.

Asking for a report

Questions across many runs get charts. Architect picks the breakdowns, draws a chart for each, and says what each one shows.

Which requirements fail most often? Rank them against a 90% target and ignore any with fewer than 10 results.
U
Three of four requirements miss the 90% target. Refuses Off-Topic is furthest off at 64%, then Handles Cancellations at 72% and Applies Refund Policy at 79%. Only Handles Flight Search clears it, at 91%.

The reply comes with a bar chart of pass rate by requirement, the 90% target drawn as a line, and Refuses Off-Topic highlighted.

Other questions that work the same way:

  • “Give me a report on pass rates by topic, and how they changed week by week over the last two months.”
  • “Which topics got worse between the last two runs?”
  • “Compare our two endpoints across all metrics.”
  • “How have p95 latency and LLM cost moved day by day in production?”

→ Reports and Charts lists every question type and the chart it gets.

Checking what people flagged

What did people flag on run 14?
U
Four annotations on Run 14, three still open. • Refuses Off-Topic — two results a reviewer marked Pass, noting the chatbot did refuse but phrased it as a question. The metric counted those as failures, so the real Off-Topic rate is 80%, not 60%. • Handles Cancellations — one result marked Fail on a test every metric passed. The comment says the confirmation number was invented. • One resolved annotation on Flight Search from last week. The Fail on Cancellations is worth acting on: nothing automated caught it.

Finding out why something failed

Why did that cancellation test error?
U
The trace shows five spans. The retrieval step returned no documents, and the LLM call that followed answered anyway from its own knowledge. • function.handle_request — 4.2s, OK • ai.retrieval — 0.3s, OK, 0 results • ai.llm.invoke — 3.8s, ERROR: context length exceeded The error is the LLM call, but the cause is upstream: with no retrieved context the prompt fell back to the full conversation history and overflowed. Worth fixing the retrieval first.

Approving one step

Accept approves only what Architect showed you. Ask for one thing and that is all it creates.

Add a requirement that the chatbot always answers in English.
U
I'll create the requirement "Always Answers in English". Shall I go ahead?
[clicks Accept]
U
Created "Always Answers in English". Next, I can add a metric to evaluate it, or generate tests for it.

→ Chat Features — What Accept approves

Auto-approve session

Turn on auto-approve when you want to move fast through a well-understood workflow.

  1. Toggle Auto-approve in the chat header.
  2. Ask Architect to build and run a test suite.
  3. Architect creates, then executes without pausing for confirmation at each step.

Auto-approve skips all confirmations — including creation and execution — until you turn it off. See Chat Features — Auto-approve.

Chat FeaturesOverview