Tests
Tests are individual cases that validate specific inputs and expected outputs for your LLM application, evaluated using assigned metrics.
Test Types
There are two types of tests:
- Single-turn tests check how the LLM application responds to a single prompt with no follow-up (Q&A).
- Multi-turn tests check how it behaves over multiple messages in a conversation.
Single-Turn Tests
A single prompt sent to your LLM application, evaluated against expected outputs and metrics.
Properties:
| Field | Description |
|---|---|
| Test Prompt | The input text sent to your LLM application |
| Files | (Optional) File attachments (images, PDFs, or audio) sent alongside the prompt. See Multi-modal Testing for supported types and size limits, and file format filters for how files are mapped to provider formats. |
| Category | High-level classification (e.g., Harmful, Harmless) |
| Topic | Specific subject matter (e.g., healthcare, financial advice) |
| Behavior | Type of behavior to validate (e.g., Compliance, Reliability, Robustness) |
| Expected Output | (Optional) What the LLM application should respond with |
Multi-Turn Tests
Goal-based conversations that test your LLM application across multiple turns. Powered by Penelope, an autonomous testing agent that adapts its strategy based on responses. Ideal for testing conversational workflows.
Properties:
| Field | Description |
|---|---|
| Goal | What the target should do - the success criteria for this test |
| Instructions | (Optional) How to conduct the test - if not provided, the agent plans its own approach |
| Restrictions | (Optional) What the target must not do - forbidden behaviors or boundaries |
| Scenario | (Optional) Context and persona for the test - narrative setup or user role |
| Files | (Optional) File attachments (images, PDFs, or audio) included with the test. Files are available on each conversation turn via the files variable in the endpoint request template. See Multi-modal Testing for supported types and size limits. |
| Min. Turns | Minimum turns before early stopping is allowed. If omitted, defaults to 80% of max turns. |
| Max. Turns | Maximum number of conversation turns allowed (max_turns) |
| Category | High-level classification (e.g., Harmful, Harmless) |
| Topic | Specific subject matter (e.g., healthcare, financial advice) |
| Behavior | Type of behavior to validate (e.g., Compliance, Reliability, Robustness) |
Creating Tests
Create tests manually or generate them automatically from behaviors and requirements. See Test Generation for automated generation.
Manual Test Writer
Use Manual Test Writer to enter or paste multiple tests without running AI generation. It provides a spreadsheet-style workflow for single-turn and multi-turn test rows.
| Test type | Fields |
|---|---|
| Single-turn | Prompt, category, topic, behavior, expected output, and optional file attachments |
| Multi-turn | Goal, instructions, restrictions, scenario, min turns, max turns, category, topic, behavior, and optional file attachments |
From the Tests page, choose manual creation, select the test type, add or remove rows, attach files when needed, then save the rows as tests or as a test set. Use the single-test drawer when you only need to create or edit one test.
Running Tests
Run individual tests from the Tests page , or execute multiple tests together as Test Sets.
Viewing Execution History
Open a test and select Execution History to see how it performed across test runs. The table shows the effective Pass or Fail status, test run, number of passing metrics, and execution time. Select a test run to open the matching result in a new tab.
The history loads up to 100 recent results and shows one entry per test run. It remains empty until the test has been executed as part of a test set.