# Rhesis Documentation (full) > Structured feedback and evals for AI agents. Connect the agent you are building, share it with your team, and turn their feedback into tests and metrics that run on every change. > This file contains the complete documentation corpus for LLM ingestion. > For a curated index with links, see https://docs.rhesis.ai/llms.txt --- url: https://docs.rhesis.ai/docs title: Rhesis Documentation --- # Rhesis Documentation **Structured feedback and evals for AI agents.** Rhesis is the collaboration layer between the people building an AI agent and the people who know what it should answer. Connect your agent, share the link with your team, and turn their feedback into tests and metrics that run on every change. [ [Rhesis Product Demo – Watch on YouTube] ](https://youtu.be/QmiUd7j4fL0) ## Getting Started ### Setup Environment Run Rhesis as a managed service, locally with Docker, or self-hosted. [Setup Environment Guide →](/docs/getting-started/setup-environment) ### Connect Application Use the SDK connector and your process opens an outbound WebSocket, so the agent stays on your laptop or inside your VPC with no public URL. If it already serves a public REST endpoint, register that URL instead. [Connect Application Guide →](/docs/getting-started/connecting-application) ### Run Evaluations Share the link with domain experts and product managers. They chat with the live agent, turn conversations into tests, and leave pass/fail verdicts and comments down to the individual metric or conversation turn — then you read it all back from the SDK, the API, or MCP. [Run Evaluations Guide →](/docs/getting-started/run-evaluations) ## Community & Support --- url: https://docs.rhesis.ai/docs/acknowledgments title: Acknowledgments --- # Acknowledgments Rhesis includes and depends on the open-source projects below. Thanks to their developers and maintainers. ## Evaluation Frameworks ### garak - LLM Vulnerability Scanner Rhesis uses garak for LLM security testing and vulnerability detection probes. - **Website**: [garak.ai](https://garak.ai) - **Repository**: [github.com/NVIDIA/garak](https://github.com/NVIDIA/garak) - **License**: Apache License 2.0 - **Copyright**: Copyright (c) 2023 Leon Derczynski, Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES ### ragas - RAG Assessment Framework Rhesis uses ragas for Retrieval-Augmented Generation (RAG) evaluation metrics. - **Repository**: [github.com/explodinggradients/ragas](https://github.com/explodinggradients/ragas) - **License**: Apache License 2.0 - **Copyright**: Copyright 2023 Vibrant Labs ### deepeval - LLM Evaluation Framework Rhesis uses deepeval for comprehensive LLM evaluation metrics including single-turn and conversational assessments. - **Website**: [confident-ai.com](https://confident-ai.com) - **Repository**: [github.com/confident-ai/deepeval](https://github.com/confident-ai/deepeval) - **License**: Apache License 2.0 - **Copyright**: Copyright 2024 Confident AI Inc. ### deepteam - LLM Red Teaming Framework Rhesis uses deepteam for red teaming and adversarial testing of LLM systems. - **Repository**: [github.com/confident-ai/deepteam](https://github.com/confident-ai/deepteam) - **License**: Apache License 2.0 - **Copyright**: Copyright 2024 Confident AI Inc. ## Infrastructure ### OpenTelemetry Python Rhesis uses OpenTelemetry for distributed tracing and observability. - **Website**: [opentelemetry.io](https://opentelemetry.io) - **Repository**: [github.com/open-telemetry/opentelemetry-python](https://github.com/open-telemetry/opentelemetry-python) - **License**: Apache License 2.0 - **Copyright**: Copyright The OpenTelemetry Authors (CNCF) ### Transformers - Hugging Face Rhesis optionally uses the Transformers library for machine learning model integration. - **Website**: [huggingface.co/transformers](https://huggingface.co/docs/transformers) - **Repository**: [github.com/huggingface/transformers](https://github.com/huggingface/transformers) - **License**: Apache License 2.0 - **Copyright**: Copyright 2018 The Hugging Face team ## License Information The full text of the Apache License, Version 2.0 is available at: [apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) For complete attribution details, see the [NOTICE](https://github.com/rhesis-ai/rhesis/blob/main/NOTICE) file in the repository root. --- url: https://docs.rhesis.ai/docs/agent-skill title: Agent Skill --- # Agent Skill Rhesis ships one **agent skill** (`rhesis`) that covers the full workflow via the Rhesis MCP server — from spec to test execution. | Workflow | When to use | | --- | --- | | [**Platform**](/docs/agent-skill/platform) | Explore endpoints, design and run tests, analyze results | | [**Spec → test foundation**](/docs/agent-skill/spec) | Paste a Product Requirements Document (PRD) or product spec; create behaviors, metrics, tags, and test sets | One install. Invoke with `/rhesis` for a **four-path menu** when intent is unclear, or speak naturally and the skill routes automatically. > **This is different from Rhesis's inbound MCP connector** (where the platform imports content from Notion, GitHub, or Jira). The agent skill lets an external AI interface call *into* Rhesis to drive testing workflows. ## Four paths On an ambiguous `/rhesis` (no endpoint, spec, or run request yet), the skill offers: 1. **Quick exploration** — fast endpoint scan 2. **Comprehensive exploration** — full capability and boundary analysis 3. **Spec → test foundation** — behaviors, metrics, tags, test sets from a spec 4. **Run or analyze** — execute test sets or compare past runs (returning users) If you already paste a spec or name an endpoint, the menu is skipped. → **AI agents:** [Agent reference](/docs/agent-skill/for-agents) and [llms.txt](https://docs.rhesis.ai/llms.txt) → **Terms:** [Glossary](/glossary) · [Metric scope](/docs/metrics/metric-scope) ## Quick install ```bash npx skills add rhesis-ai/rhesis -g ``` Connect the MCP server in your agent — see [Install in Cursor](/docs/agent-skill/platform#install-in-cursor). ## Examples ```text # Ambiguous — shows four-path menu /rhesis # Platform — explore and test an endpoint "I want to test my travel chatbot. The endpoint is called 'travel-agent-v2'." # Spec — build a test foundation first "Here's our agent PRD: [paste your product spec]" # Returning user — run or analyze "Compare my last two test runs for the safety test set" ``` ## Telemachus (native Architect) The in-app **Architect** agent mirrors the same four paths. See [Architect workflow](/docs/architect/workflow). ## Source [`skills/rhesis/`](https://github.com/rhesis-ai/rhesis/tree/main/skills/rhesis) — skill instructions and reference docs (`references/prd/` for spec design guides). --- url: https://docs.rhesis.ai/docs/annotations title: Annotations --- # Annotations Browse human reviews across test results and traces from **Improve → Annotations**. Annotations are the same judgements you create on a [test result](/docs/tasks/test-reviews) or [trace](/docs/tracing) — Pass/Fail with a comment — collected in one list so you can find and follow up on them without hunting through individual runs. ![Annotations hub — human judgements on test results and traces in one list](/screenshots/rhesis-ai-annotations.webp) ## Why they matter Automated metrics are a useful signal, but they miss context only humans have: brand tone, policy nuance, or a “failure” that was actually the right refusal. Annotations turn those judgements into durable feedback so the team can improve the agent instead of rediscovering the same issues. Use them to: - Correct automated Pass/Fail when the metric was wrong for your domain - Capture *why* a response succeeded or failed for teammates and future tuning - Keep disagreements visible (Open) until the model, prompt, or metric is fixed (Resolved) - Spot patterns across runs and live traces — not only inside a single result drawer The hub is the Improve entry point for that work: every open annotation is a candidate improvement. ## Using the hub 1. Open **Improve → Annotations** 2. Use **All / Open / Resolved** to filter by resolution status 3. Search by comment, annotator, target, behavior, or rating 4. Open the filter drawer for type (test result / trace), rating, and target 5. Click a row to open the source review in a new tab | Column | Meaning | |--------|---------| | **Type** | Test result or trace | | **Target** | Output, metric, turn, or trace | | **Rating** | Human Pass / Fail | | **Annotator** | Who wrote the review | | **Behavior** | Linked test behavior (test results only) | | **Status** | Open or Resolved | | **Comment** | Review text (mentions render as chips) | ## Resolve Mark a review **Resolved** when the disagreement is handled (for example after fixing the model or accepting the override). Use **Reopen** if it needs attention again. Resolve and reopen from the Reviews tab on the test result or trace. Conflicts between automated and human verdicts are hidden once the conflicting review is resolved. ## Permissions You need **test result read** and/or **telemetry read**. The list only includes sources you can access. ## Related - [Test Reviews](/docs/tasks/test-reviews) — how to add and target reviews on test results - [Test Runs](/docs/test-runs) — open a result and use the Reviews tab - [Tracing](/docs/tracing) — live traces and review workflows - [Trace Metrics](/docs/metrics/trace-metrics) — automated evaluation on traces --- url: https://docs.rhesis.ai/docs/api-tokens title: API Tokens --- # API Tokens API tokens authenticate programmatic access to Rhesis via the REST API or Python SDK. Generate and manage them at [app.rhesis.ai/tokens](https://app.rhesis.ai/tokens). The token is shown only once at creation — copy it immediately. ![Token Overview](/screenshots/rhesis-ai-tokens.png) ## Using API Tokens ### With REST API Include the token in the Authorization header: ```bash curl -H "Authorization: Bearer YOUR_TOKEN_HERE" \\ https://api.rhesis.ai/tests ``` ### With Python SDK Initialize the SDK with your token: ```python from rhesis.sdk import RhesisClient client = RhesisClient(api_key="YOUR_TOKEN_HERE") ``` See [Python SDK](/sdk/installation) for complete usage documentation. --- url: https://docs.rhesis.ai/docs/architect title: Rhesis Architect --- # Rhesis Architect Architect (also known as Telemachus) is a conversational agent that designs, builds, and runs test suites for your AI endpoints. Describe what you want to test — it explores your endpoint, proposes a plan, waits for your approval, then creates everything on the platform. [Watch on YouTube](https://www.youtube.com/watch?v=M-x-Bjbgl2g) ## What you can do - **Explore** an endpoint to understand its domain, capabilities, and refusal behavior - **Design** a complete test suite with behaviors, metrics, and test sets — reusing what already exists on the platform - **Generate** AI-written tests or import your own verbatim - **Ground tests in your docs** by telling Architect to use a specific knowledge source - **Execute** tests against any endpoint and get a structured pass/fail summary - **Compare** two test runs to spot regressions and improvements - **Analyze** any existing test run with behavior and metric breakdowns - **Do direct operations** — update a metric, link a behavior, list test sets — without going through the full planning flow ![Architect chat — what the endpoint exploration found, the proposed plan, and the plan progress bar](/screenshots/rhesis-ai-architect-chat.webp) > Architect refers to all entities by name. You never need to supply an ID. ## Get started Architect needs at least one [endpoint configured](/docs/endpoints) in your organization. Open **Architect** in the sidebar. Type what you want to test, or pick one of the suggested prompts on the welcome screen. ```text "Test my travel chatbot — focus on safety and accuracy." ``` Architect responds in real time. You'll see the current phase, streaming tool activity, and a plan panel that tracks progress as work happens. ## Pages in this section | Page | What it covers | | --- | --- | | [Workflow](/docs/architect/workflow) | The four phases and what you see in each one | | [Endpoint Exploration](/docs/architect/exploration) | Quick vs Comprehensive modes and what each reveals | | [Planning Test Suites](/docs/architect/planning) | Plan structure, reuse logic, knowledge sources | | [Running and Analyzing](/docs/architect/execution-and-analysis) | Test execution, result analysis, run comparison | | [Chat Features](/docs/architect/chat-features) | File attachments, auto-approve, plan panel, mode chip | | [Scenarios](/docs/architect/scenarios) | Request cookbook for common workflows | ## For developers The Architect system spans four layers, each documented in the contributing section: - [Backend — session API, WebSocket, MCP](/contribute/backend/architect-chat-system) - [Frontend — chat UI, hook, streaming indicators](/contribute/frontend/architect-chat-ui) - [Worker — Celery task, async resume](/contribute/worker/architect-background-tasks) - [SDK — ArchitectAgent class, plan model, prompt templates](/contribute/sdk/architect-agent) --- url: https://docs.rhesis.ai/docs/behaviors title: Behaviors --- # Behaviors Behaviors are the expectations for how your LLM application should perform — response quality, safety, brand voice, and so on. You attach metrics to a behavior to measure it, and tag tests with a behavior so results roll up by expectation. ![Behaviors page — each behavior card lists the metrics that measure it](/screenshots/rhesis-ai-behaviors.webp) ## Behaviors and metrics Rhesis uses a two-layer model: **behaviors** state what you expect; **metrics** verify it. One behavior can carry multiple metrics, so you can measure the same expectation from different angles. See [Tests](/docs/tests) and [Test Results](/docs/test-results). ## Behavior tags Behavior tags are optional labels for organizing and filtering behaviors — use them when several behaviors share a feature area, risk theme, or team workflow. Add them while creating or editing a behavior: type in the Tags field and press Enter, or pick an existing tag from the suggestions. Tags are organization-scoped and reusable across behaviors, and the behavior grid can filter by them. Behavior tags are distinct from a test's category and topic: tags organize behavior definitions, while category and topic classify individual tests. ## Next steps - Create [Metrics](/docs/metrics) for your behaviors - Generate [Tests](/docs/tests) - Review results grouped by behavior --- url: https://docs.rhesis.ai/docs/concepts title: Core Concepts --- # Core Concepts Rhesis evaluates whether your LLM application meets the quality bar you care about. This page has two parts: the **workflow** (what you do and in what order) and the **platform structure** (what each object is and how they connect). ## The workflow A normal workflow in Rhesis looks like this: 1. **Connect your endpoint** — link Rhesis to the system under test. See [Connect your application](/docs/getting-started/connecting-application). 2. **Define behaviors** — decide what good looks like for your application. 3. **Set up metrics** — assign judges to each behavior so responses can be scored automatically. 4. **Create tests** — write, import, or generate inputs; tag each test with a behavior. 5. **Run a test set** — execute a batch of tests against your endpoint. 6. **Review results** — inspect individual failures, then track trends across runs. > **New to Rhesis?** The [Getting Started guide](/docs/getting-started) walks through environment setup and your first run. ## Platform structure --- > **Ready to start?** Create a [project](/docs/getting-started/projects), [connect an endpoint](/docs/getting-started/connecting-application), and [generate tests](/docs/test-sets/tests-generation) to run your first evaluation. --- url: https://docs.rhesis.ai/docs/deployment title: Deployment --- # Deployment You can deploy Rhesis on your own infrastructure. We provide rich guides from very easy quick start with one command, to more advanced configurations like Kubernetes and cloud environments. > **Want the easiest path?** > Skip the infrastructure management and start testing immediately with [Rhesis Cloud](https://app.rhesis.ai). We handle the hosting, scaling, and updates so you can focus on building and testing your AI apps. ## Available Options ## Architecture Overview ### Services The Rhesis platform consists of several interconnected services: ### Service Dependencies ```mermaid graph LR Frontend["Frontend
Port 3000
Next.js"] Backend["Backend
Port 8080
FastAPI"] PostgreSQL["PostgreSQL
Port 5432
Database"] Redis["Redis
Port 6379
Cache & Queue"] Worker["Worker
Port 8081
Celery"] Frontend --> Backend Backend --> PostgreSQL Backend --> Redis Worker --> Redis ``` --- url: https://docs.rhesis.ai/docs/endpoints title: Endpoints --- # Endpoints An endpoint is how Rhesis connects to your AI model or application to send test inputs and evaluate responses. There are two ways to set one up depending on how your model is exposed: - **REST or WebSocket** — your model is behind an HTTP API. You configure the URL, authentication, and request/response mappings [directly in the platform](/docs/endpoints/creating-endpoints). This is the most common case. - **SDK** — your model is a Python function with no HTTP layer. You register it as an endpoint using the `@endpoint` decorator from the Rhesis SDK. See [SDK Endpoints](/docs/endpoints/sdk-endpoints). Once configured, an endpoint can be reused across any number of tests and test sets. You can also run the same tests against multiple endpoints to compare models, environments, or configurations. ![Endpoints page — connection type, environment, and project for each endpoint](/screenshots/rhesis-ai-endpoints.webp) ## Platform-Managed Variables Rhesis populates these variables at runtime; you reference them in request templates and extract them in response mappings. ### Request Variables Available in request body templates via Jinja2 syntax (`}`). | Variable | Description | |---|---| | `input` | The user query or test prompt. This is the primary text sent to your AI endpoint. | | `files` | Array of file attachments included with the test. Each file is an object with `filename`, `content_type`, and `data` (base64-encoded). Use [file format filters](/docs/endpoints/request-mapping#file-format-filters) to convert files into provider-specific formats. | | `conversation_id` | Conversation tracking identifier for multi-turn endpoints (both stateful and stateless). Rhesis uses this to track conversations across turns. | | `messages` | Full conversation history as an array of message objects. Used for stateless endpoints that require the entire history with every request. | | `system_prompt` | System prompt text. Rhesis prepends it to the `messages` array and strips it from the final request body before sending. | | `params` | Resolved [experiment parameters](/docs/experiments) for the current test run. Access individual values with dot notation: `}`, `}`. Only populated when the test run is associated with an experiment; empty dict otherwise. See [Using Experiment Parameters](#using-experiment-parameters) below. | | `test_id` | Unique identifier of the test being executed. Only populated during test runs; renders as an empty string otherwise. Place it wherever your API expects it — see [Test Execution Context](/docs/endpoints/mapping-examples#test-execution-context) for examples. | | `test_run_id` | Unique identifier of the current test run. Only populated during test runs; renders as an empty string otherwise. | | `test_configuration_id` | Unique identifier of the test configuration driving the run. Only populated during test runs; renders as an empty string otherwise. | > Provider-specific Jinja filters such as `to_anthropic`, `to_openai`, and `to_gemini` are documented in > [Request Mapping](/docs/endpoints/request-mapping#file-format-filters). ### Response Variables Extracted from your API response using JSONPath or Jinja2 expressions in the response mapping. | Variable | Description | |---|---| | `output` | The main response text from your API. This is what Rhesis evaluates against your metrics. | | `context` | Additional context or reasoning provided by the response (e.g., retrieved documents, sources). | | `metadata` | Structured data about the response (e.g., model version, token counts, confidence scores). Stored with the test result and available to [custom metrics](/docs/metrics#using-metadata-in-evaluation) during evaluation. | | `tool_calls` | Tool or function calls made by the API during response generation. Stored with the test result and available to metrics that evaluate tool use. | | Conversation ID fields | Any of the recognized conversation identifiers: `conversation_id`, `session_id`, `thread_id`, `chat_id`, `dialog_id`, `dialogue_id`, `context_id`, `interaction_id`. Map one of these for multi-turn endpoints so Rhesis can track conversations across turns. | > You can include additional custom fields in both request templates and response > mappings. Custom fields are passed through and stored, but Rhesis does not > actively use them for evaluation or conversation management. **Request mapping** uses Jinja2 templates to format platform variables into your API's request body. **Response mapping** uses JSONPath expressions to extract values from the response. For full details, see [Request Mapping](/docs/endpoints/request-mapping) and [Response Mapping](/docs/endpoints/response-mapping). ## Using Experiment Parameters When you run tests against an [experiment](/docs/experiments), Rhesis resolves the experiment's parameter values and makes them available in your request mapping under the `params` variable. This lets you drive model selection, temperature, system prompts, and any other configuration directly from the experiment -- without changing your endpoint definition. ### Accessing parameters in request mappings Use Jinja2 dot notation to reference individual parameter values: ```json { "model": "{{ params.model }}", "messages": [ { "role": "system", "content": "{{ params.system_prompt }}" }, { "role": "user", "content": "{{ input }}" } ], "temperature": {{ params.temperature }}, "max_tokens": {{ params.max_tokens }} } ``` > Numeric and boolean parameters (like `temperature` or `max_tokens`) should be > referenced **without quotes** so they render as JSON numbers or booleans rather > than strings. When no experiment is associated with a test run, `params` is an empty object and unresolved references render as empty strings. You can use Jinja2 defaults to handle this gracefully: ```json { "model": "{{ params.model | default('gpt-4') }}", "temperature": {{ params.temperature | default(0.7) }} } ``` ### How it works Parameter injection follows the same flow for both REST and SDK connector endpoints: 1. When a test run is created, the experiment's parameter values are **resolved once** and stored as an immutable snapshot on the test run. 2. During execution, the snapshot is injected into the Jinja2 template context as `params`. 3. The template renderer evaluates `\\}` (or any other parameter name) against the snapshot. This means every test in a run uses the exact same parameter values, and those values are recorded alongside the test results for full reproducibility. ### REST vs. SDK connector Both endpoint types receive the same resolved parameters: - **REST endpoints** access parameters through `\\}` in request mappings, as shown above. Your external API receives the rendered values in the HTTP request body. - **SDK connector endpoints** should access parameters through `\\}` in `request_mapping`, or read the test-run snapshot through `Parameters.get()` from inside the function. The legacy `@endpoint(parameters=...)` kwarg-merging path is deprecated. See [Connector Injection](/docs/experiments/connector-injection) for details. ## Managing endpoints After creation, endpoints can be edited, duplicated, deleted, and tested from the detail page, and reused across tests. See [Management](/docs/endpoints/management) for the detail-page tabs, grid actions, and environment organization. ## Next steps - [Create an Endpoint](/docs/endpoints/creating-endpoints) — wizard walkthrough and Auto-Configure - [Request Mapping](/docs/endpoints/request-mapping) — Jinja2 syntax, filters, multi-turn patterns - [Response Mapping](/docs/endpoints/response-mapping) — JSONPath and Jinja2 extraction - [Examples](/docs/endpoints/mapping-examples) — provider configurations - [Management](/docs/endpoints/management) — testing, detail tabs, and environments --- url: https://docs.rhesis.ai/docs/experiments title: Experiments --- # Experiments An **experiment** is one attempt at a parameter configuration. **Versions** are its immutable history; **environments** are how running code and test runs find a version. ## The mental model Experiments follow a specific lifecycle from private drafting to production deployment: ```mermaid flowchart LR Schema["Project schema
declares typed slots"] Author["Create experiment
private to me"] Commit["Save values
= new immutable version"] Share["Flip to shared
visible to project"] Promote["Promote to environment
(e.g. 'default')"] Resolve["SDK / test runs
resolve via environment"] Schema --> Author Author --> Commit Commit -. iterate .-> Commit Commit --> Share Share --> Promote Promote --> Resolve ``` 1. **Schema:** The project's [Parameter Schema](/docs/experiments/parameter-schema) defines the typed slots available to configure. 2. **Author:** You create a new experiment. By default, it is private to you, providing a safe sandbox. 3. **Commit:** You fill in values for the parameter slots and save. Every save mints a new, immutable *Version* with a sequential identifier (v1, v2, v3, ...). 4. **Share:** Once the experiment is ready, you flip its visibility to shared, making it visible to your team. 5. **Promote:** You promote a specific version of your shared experiment to an *Environment* (like `default` or `production`). 6. **Resolve:** Application code and test runs request parameters by environment, automatically receiving the newly promoted configuration. ## Vocabulary - **Parameter Schema** — A project-scoped, typed *contract*. Declares what knobs exist (slot name + type + default). - **Parameter** (or *slot*) — One named entry in the schema. - **Experiment** — A named, owned attempt at a configuration. Holds values for the schema's slots. - **Version** — An immutable snapshot of an experiment's values at a moment in time, identified by a sequential label (v1, v2, ...). - **Environment** — A movable, project-scoped pointer at a specific `(experiment, version)` pair. - **Visibility** — A per-experiment flag (`private` vs `shared`). - **Promoting** — Moving an environment to point at a different version. - **Pinning** — Fetching configuration by `version` instead of by environment. - **Snapshotting** — Capturing resolved values at run-queue time to guarantee test reproducibility. ## Why immutable versions? Versions are immutable so a test run can be re-run six months from now and produce bit-identical inputs. If versions were mutable, changing a parameter would silently invalidate historical test results. Because each version is immutable and sequentially numbered, the audit trail is exact: "this run used `v3`". ## Why movable environments? Production code shouldn't hardcode version identifiers — it asks for `environment="production"`. Environments separate *what's running* from *what was running*. Promoting an environment is the deployment step. Tying environments to versions (not to experiments) means you can promote the same experiment to multiple environments (for example staging a change before rolling it out to production). ## Visibility: private vs shared Private experiments are your personal sandboxes — your colleagues can't accidentally use yours, and you can't accidentally affect theirs. Sharing an experiment is the explicit act of saying "this is ready to be a team artifact". As a safety mechanism, only `shared` experiments can be promoted onto an environment, ensuring a stray draft can't accidentally hit production. ## Promoting (the deployment primitive) *Promoting* is the act of moving an environment. The mechanics are simple: pick a `shared` experiment, select a version, and click "Promote to default" (or any other environment name). > **Deployment Impact:** Moving an environment immediately impacts all consumers resolving against that environment (after their short TTL cache expires). Promoting the `production` environment is a true deployment and should be treated as a deliberate action. ## Well-known environments Rhesis provides four well-known environment names out of the box: `default`, `development`, `staging`, and `production`. - `default` is the implicit pointer. It's what the SDK returns when no environment is explicitly requested. - `development`, `staging`, and `production` mirror a typical promotion path so you can fan out deployments without copying experiments between projects. You can also create custom environment names at will for your own workflows. ## Reproducibility: snapshotting When a test run is queued, Rhesis *snapshots* the resolved parameter values. This means that if an environment is moved *after* a run is queued, the test run is not perturbed. The worker only reads from the snapshot. This guarantees that test runs remain reproducible even as your environments point to newer configurations over time. For the most reproducible production deployments, you can *pin* a version in the SDK (`version="v3"`) to bypass environment resolution entirely. ## UI Walkthrough ### Running a test set against an experiment When executing a Test Set, you can optionally target it against an experiment: 1. Open the Execute drawer for a Test Set. 2. Select an **Experiment** from the dropdown. 3. (Optional) Pin to a specific version. 4. Review the resolved-values preview (secret references are masked). 5. Click Run. ### The Latest Results panel The bottom of the Experiment detail page features the Latest Results panel. This fetches recent test runs that executed against versions of this experiment. It offers two toggleable views: 1. **By run:** A flat list providing a pass/fail summary per run with deep links to the test run details. 2. **By version (config-diff):** Runs grouped by `parameter_version`, ordered newest first. For each pair of adjacent versions, a compact diff row shows: - The changed parameter slots and their before/after values (for example `temperature: 0.7 → 1.4`). - The pass-rate before vs after, and the delta (for example `82% → 70% (-12pp)`). The config-diff view is your primary regression-finder: it surfaces exactly which edit broke things without requiring manual reasoning. ## Comparisons To help anchor the mental model, here is how Rhesis parameter management compares to systems you may already know: | Rhesis | Git | Feature Flags | MLflow | | --- | --- | --- | --- | | Version | Commit (immutable) | Rule Snapshot | Run | | Environment | Branch Tip / Tag | Environment | Tag | | Experiment | Branch | Flag | Experiment | --- url: https://docs.rhesis.ai/docs/explorer title: Test Explorer --- # Test Explorer Test Explorer is an interactive workspace for building a test set. Draft or import tests, invoke your endpoint, and see responses and metric scores in the same place — so you can iterate until you find the gaps you care about. [Watch on YouTube](https://www.youtube.com/watch?v=4E1vD7UiHqU) ## Two benefits of Test Explorer Test Explorer surfaces two complementary capabilities: ### 1. Interactive endpoint play Write a test, invoke the endpoint, see metric scores — all in the same view. No test run setup and no waiting for a batch job. The loop is fast enough to tweak a single prompt and confirm the fix in seconds. ### 2. Exploration via suggestions Given the tests you already have, an LLM generates a batch of new inputs, ranked by diversity so the most varied prompts surface first, then invokes your endpoint and scores each row. You accept only the rows that add real coverage or expose real failures. For the full pipeline — sampling, diversity ranking, generation guide, accept flows — see [Building and Evaluating — Suggestions](/docs/explorer/building-and-evaluating#suggestions-explore-the-space-automatically) and [Workflow](/docs/explorer/workflow). [Test Explorer] ## What you can do - **Metrics at a glance** — each test and topic shows pass/fail and per-metric scores - **Organize by topic** — group findings (for example off-topic or competitor cases) for export and regression runs ## Prerequisites - At least one [endpoint configured](/docs/endpoints) in your organization - At least one metric available under [Metrics](/docs/metrics) ## Get started Open **Testing → Test Explorer** in the sidebar. - Click **New session** to start from scratch - Click **Load Test Set** to seed an explorer session from an existing test set ## Pages in this section | Page | What it covers | | --- | --- | | [Workflow](/docs/explorer/workflow) | Setup, build, suggestions pipeline, review, export — with a workflow diagram | | [Building and Evaluating](/docs/explorer/building-and-evaluating) | Sessions, settings, topic tree, suggestions in depth, scoring, export | | [Scenarios](/docs/explorer/scenarios) | Cookbook workflows | --- url: https://docs.rhesis.ai/docs/getting-started title: Getting Started with Rhesis --- # Getting Started This guide takes you from a fresh environment to your first automated evaluation of your LLM application. ## Quick install (Python SDK) Prefer code first? Install the SDK and list your test sets in a few lines: ```bash pip install rhesis-sdk ``` ```python import os from rhesis.sdk.entities import TestSets os.environ["RHESIS_API_KEY"] = "rh-your-api-key" # from app.rhesis.ai for ts in TestSets.all(): print(ts.name) ``` For full SDK docs, see [SDK installation](/sdk/installation). ## What are you building? Pick a path that matches your goal — each card links to deeper docs. > **Prefer programmatic access?** Use the [Python SDK](/sdk/installation) for test sets, runs, and the connector. Getting started consists of three core steps: ### Setup Environment Choose how you want to run Rhesis: Cloud (fastest), Local, or Self-hosted. [Setup Environment Guide →](/docs/getting-started/setup-environment) ### Connect Application Connect your LLM application to Rhesis via REST Endpoints or our Python SDK. [Connect Application Guide →](/docs/getting-started/connecting-application) ### Run Evaluations Generate test sets interactively and run automated evaluations against your application. [Run Evaluations Guide →](/docs/getting-started/run-evaluations) ## Next Steps ## Need Help? - **In-app support drawer**: Click **Support** in the sidebar footer to open links to the documentation site, support email, GitHub, and Discord. - **GitHub Issues**: [Report bugs or request features](https://github.com/rhesis-ai/rhesis/issues) --- url: https://docs.rhesis.ai/docs/integrations title: Integrations --- # Integrations Rhesis plugs into your LLM stack across four layers, each addressing a different concern. This page is the single map: pick the layer you care about and follow the link into the matching guide. | Layer | What it covers | Where to start | |---|---|---| | **LLM providers** | The model that runs your test generation and LLM-as-Judge evaluation. | [Models](/docs/models) | | **Tracing** | Streaming OpenTelemetry spans from your application to Rhesis. | [Tracing](/docs/tracing) · [Auto-instrumentation](/docs/tracing/auto-instrumentation) | | **Test execution** | Letting Rhesis invoke entry points in your application remotely to run test cases. | [Connector](/sdk/connector) | | **REST API** | Programmatic access to test sets, runs, and platform resources. | [api.rhesis.ai/docs](https://api.rhesis.ai/docs) | Beyond those four layers, Rhesis also bundles **evaluation frameworks** (DeepEval, Ragas, Garak) and **tool connections** (Notion, GitHub, Jira, and more) as additional sources of metrics, tests, and knowledge context. ## Tracing your application Your application emits OpenTelemetry spans through the Rhesis SDK. How you attach them depends on the framework: - **LangChain**, **LangGraph**, **Microsoft Agent Framework**, and **Pydantic AI** auto-instrument with a single `auto_instrument()` call. - Other Python frameworks (CrewAI, OpenAI Agents SDK, LlamaIndex, …) use the `@observe.*` decorators on the functions, tools, or agents you want to capture. - Any OpenTelemetry-compatible exporter can target the Rhesis ingestion endpoint directly. See: [Tracing overview](/docs/tracing) · [Auto-instrumentation](/docs/tracing/auto-instrumentation) · [Decorators](/docs/tracing/decorators) · [Multi-agent tracing](/docs/tracing/multi-agent) ## Test execution: the connector To run test cases against your application, Rhesis needs to call your code. Register an entry point with the `@endpoint` decorator; the SDK holds a persistent outbound WebSocket to Rhesis, so test runs invoke your function locally without exposing a public URL. The same call path serves single-turn test cases and multi-turn conversations. See: [Connector](/sdk/connector) · [Parameter binding](/sdk/connector/binding) · [Connector examples](/sdk/connector/examples) ## LLM providers Choose any provider for the LLMs that drive test synthesis and LLM-as-Judge evaluation. Provider routing is powered by [LiteLLM](https://github.com/BerriAI/litellm), giving you a single interface to 100+ models — cloud (OpenAI, Anthropic, Google Gemini, Mistral, Cohere, Groq, Together AI) or local/self-hosted (Ollama, vLLM, LiteLLM proxy). See: [Models](/docs/models) · [API tokens](/docs/api-tokens) ## Evaluation frameworks In addition to Rhesis-native metrics, you can use metrics from DeepEval and Ragas, and import Garak probes as test sets for adversarial scanning. See: [DeepEval](/docs/metrics/deepeval) · [Ragas](/docs/metrics/ragas) · [Import from Garak](/docs/test-sets/import-from-garak) · [Metrics](/docs/metrics) ## Tools Pull knowledge into Rhesis from external services like Notion or Github, or create a Jira issue directly from a Rhesis task. See: [Tools](/docs/tools) ## REST API Direct API access for custom integrations and CI/CD pipelines: manage test sets, trigger test runs, fetch results, and inspect traces programmatically. Language-agnostic — call from Python, TypeScript, Go, shell scripts, or anywhere else. See: [OpenAPI spec](https://api.rhesis.ai/docs) · [API tokens](/docs/api-tokens) Tip: New to Rhesis? Start with [Getting started](/docs/getting-started). --- url: https://docs.rhesis.ai/docs/knowledge title: Knowledge --- # Knowledge Knowledge is domain-specific content you add to Rhesis to generate relevant, context-aware test cases. It is made up of **Sources** — documents or pages imported from external systems that serve as context for test generation. ![Knowledge Page](/screenshots/rhesis-ai-knowledge.png) ## How to Add a Source ### Upload Documents Upload files directly from your computer. Rhesis accepts `.txt`, `.md`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.json`, `.xml`, `.html`, `.htm`, `.epub`, and `.zip` (up to 5 MB per file). ### Import from Tools Import pages from connected tools such as Notion. See the [Tools documentation](/docs/tools) for connecting tools. Once you have Sources, use them to [generate context-aware tests](https://app.rhesis.ai/generation). --- url: https://docs.rhesis.ai/docs/metrics title: Metrics --- # Metrics Metrics are the quantifiable measurements that evaluate an AI response and decide whether it meets your requirements. They are organized by [**Behaviors**](/docs/behaviors): behaviors define what you expect, metrics measure how well you meet it. ![Metrics page — each metric card shows the behaviors it measures and its backend](/screenshots/rhesis-ai-metrics.webp) ## Offline Tests vs Online Trace Metrics Metrics can be used in two different contexts: 1. **Offline Evaluation (Testing):** Grade deterministic outputs against known ground-truths or predefined test sets. 2. **Online Evaluation (Trace Metrics):** Automatically evaluate live production traces in real-time as users interact with your AI endpoint. To learn how to enable metrics for live production traffic, read the [**Trace Metrics Guide**](/docs/metrics/trace-metrics). You can add an existing metric to a behavior by clicking the "+" icon in the top-right corner of the chip on the Metrics Overview page. ![Metrics Assign](/screenshots/rhesis-ai-metrics-assign.png) ## Using Existing Metrics The platform includes pre-built metrics from multiple providers: - **[DeepEval](/docs/metrics/deepeval)** — single-turn and conversational metrics by Confident AI - **[DeepTeam](https://github.com/confident-ai/deepteam)** — team-based red teaming by Confident AI - **[Ragas](/docs/metrics/ragas)** — RAG evaluation metrics by Exploding Gradients - **Rhesis** - Custom metrics developed by the Rhesis team Here are some examples: | Metric Name | Provider | Score Type | Scope | Description | |-------------|----------|------------|-------|-------------| | Role Adherence | DeepEval | Numeric | Multi-Turn | Evaluates whether the assistant maintains its assigned role throughout the conversation. | | Knowledge Retention | Rhesis | Numeric | Multi-Turn | Measures memory consistency and correct use of information introduced earlier in the conversation. | | Answer Accuracy | Ragas | Numeric | Single-Turn | Measures accuracy of the generated answer against ground truth. | ## Creating Custom Metrics You can create custom metrics tailored to your specific evaluation needs. ![Create Metrics](/screenshots/rhesis-ai-create-metrics.png) This section presents what needs to be configured to create a custom metric using a Judge-as-Model approach. ### 1. Evaluation Process Define how the LLM evaluates responses by providing four key components: Evaluation Model Select the [Model](https://app.rhesis.ai/models) you want to use for the evaluation. For more information on how to configure models, see the [Models documentation](/docs/models). Note: The evaluation model can also be overridden per-run via the Model Settings section in the [execution drawer](/docs/test-runs/execution#model-settings). Evaluation Prompt Write clear instructions specifying what to evaluate and the criteria to use. ```text Example: "Evaluate whether the response is accurate, complete, and relevant. Use the following criteria: accuracy of facts, coverage of required points, clarity of explanation." ``` Evaluation Steps Break down the evaluation into clear steps. These guide the LLM when producing a score and reasoning. ```text Example: 1. Check Accuracy: Identify any factual errors or unsupported statements 2. Check Coverage: Determine if all required elements are addressed 3. Check Clarity: Assess whether the response is clear and well-structured ``` Reasoning Instructions Explain how to reason about the evaluation and weight different aspects. ```text Example: "Extract key claims, assess completeness and correctness, weigh issues by importance, then determine the final score." ``` **Best Practices:** - Focus on one evaluation dimension (accuracy, tone, safety, etc.) - Use concrete, measurable criteria - Provide clear examples when possible ### Evaluating Context, Metadata, and Tool Calls Metrics can also take into account `context`, `metadata`, and `tool_calls`. These are Rhesis-managed keys providing additional information for a given interaction alongside the standard text response. - **`context`**: Background information, retrieved documents, or system prompts provided to the model. - **`metadata`**: Structured data returned by your endpoint (e.g., token usage, latency, model version, confidence scores, or custom fields). - **`tool_calls`**: External functions or tools invoked by the AI during the interaction. **How it works:** 1. Configure your endpoint's [response mapping](/docs/endpoints) to extract a `metadata` field from the API response (e.g., `"metadata": "$.usage"`) 2. Reference the metadata in your metric's **Evaluation Prompt** -- the JSON object will be available to the evaluation model automatically **Example 1: Evaluating Context Groundedness** Evaluation Prompt: ```text Evaluate whether the response is fully grounded in the provided context. Check the `context` array. Any claim in the output that is not supported by the `context` should result in a lower score. ``` **Example 2: Evaluating Token Efficiency (Metadata)** Endpoint response mapping: ```json { "output": "$.choices[0].message.content", "metadata": "$.usage" } ``` Evaluation Prompt: ```text Evaluate whether the response is efficient in token usage. Check the `metadata` for token counts. A response that uses more than 500 tokens for a simple factual question should score lower. ``` **Common use cases:** - **RAG Groundedness (`context`)** -- check that the model's answer relies only on retrieved knowledge - **Tool Correctness (`tool_calls`)** -- verify the AI chose the right tool and passed the correct arguments - **Token usage (`metadata`)** -- evaluate cost efficiency by checking `prompt_tokens` and `completion_tokens` - **Confidence scores (`metadata`)** -- verify the model's self-reported confidence aligns with response quality ### 2. Score Configuration Choose your scoring approach: Numeric Scoring Define a numeric scale with a pass/fail threshold: - **Min/Max Score**: e.g., 0-10 - **Threshold**: Passing score (e.g., 7) - **Operator**: `>=`, `>`, `<=`, `<`, or `=` *Example: Min 0, Max 10, Threshold ≥7 means scores 7-10 pass, 0-6 fail.* Categorical Scoring Define custom categories for classification: - Add categories: "Excellent", "Good", "Fair", "Poor" - The LLM classifies responses into one category *Example: "Safe", "Borderline", "Unsafe" for safety evaluation.* ### 3. Metric Scope Select at least one scope: - **Single-Turn**: individual question-answer pairs - **Multi-Turn**: multi-exchange conversations - **Trace**: live production traffic evaluated in real-time (see [Trace Metrics](/docs/metrics/trace-metrics)) Scope must include the test's type or the metric is silently skipped at run time — see [Metric scope](/docs/metrics/metric-scope). ### 4. Result Explanation Provide instructions for how the LLM should explain its scoring rationale. ```text Example: "Explain which specific parts were accurate or inaccurate, citing evidence from the provided context." ``` ## Improve Metrics with Natural-Language Instructions You can refine an existing metric without rewriting every field manually. The improve endpoint reads the current metric definition, applies your natural-language edit instructions with the platform generation model, and updates the metric in place. Use this workflow when a metric is close to what you need but the threshold, prompt, scoring criteria, or scope needs to change. ```bash curl -X POST "$RHESIS_BASE_URL/metrics/$METRIC_ID/improve" \\ -H "Authorization: Bearer $RHESIS_API_TOKEN" \\ -H "Content-Type: application/json" \\ -d '{ "prompt": "Make this metric stricter about hallucinated facts and require citations from the provided context." }' ``` The request body has one field: | Field | Type | Required | Description | | --- | --- | --- | --- | | `prompt` | `string` | Yes | Natural-language instructions for how to change the existing metric. | > The improved metric replaces the existing metric definition. Review the updated evaluation prompt, > score type, threshold, categories, and metric scope before using it in a critical test run. ## Next steps - Assign metrics to [behaviors](/docs/behaviors) and use them in [test generation](/docs/test-sets/tests-generation) - View metric performance in [Test Results](/docs/test-results) - Use metrics programmatically with the [SDK](/sdk/metrics) --- url: https://docs.rhesis.ai/docs/models title: Models --- # Models Models are the LLM providers Rhesis uses for three purposes: - **Test generation**: create test cases from your requirements (see [Tests](/docs/tests)) - **Evaluation (LLM-as-judge)**: score responses with LLM-powered [Metrics](/docs/metrics) - **Execution (multi-turn)**: drive conversation simulations with [Penelope](/docs/tests/conversation-simulation) against your endpoints Configure a default model per purpose in the model settings, or override it per run (see [Test Execution](/docs/test-runs/execution)). ![Models Overview](/screenshots/rhesis-ai-models.png) ## Default Model **Rhesis Default** is a Rhesis-hosted model, pre-configured for generation, evaluation, and execution with no API key required — so you can start testing without connecting a provider. ## Supported Providers Rhesis connects to major cloud providers and to local or self-hosted runtimes (Ollama, vLLM, LiteLLM proxy). A selection: Provider list - **Anthropic** - Claude provider - **Azure AI Studio** - Azure-hosted model endpoints (`azure_ai`) - **Azure OpenAI** - Azure OpenAI deployments (`azure`) - **Cohere** - Command R provider - **Google** - Gemini provider - **Groq** - LPU-based model hosting - **LiteLLM Proxy** - OpenAI-compatible proxy gateway - **Meta** - Llama provider - **Mistral** - Mistral provider - **OpenAI** - OpenAI provider - **Perplexity** - Labs model API - **Polyphemus** - Adversarial testing model (restricted access workflow) - **Replicate** - Model hosting provider - **Together AI** - Multi-model provider ## Connecting a Model Provider Click **Add Model** to open the model connection drawer. The drawer has two steps: ### Select a provider Choose whether you are adding a language model or an embedder, then select the provider from the provider list. Providers display icons and badges so you can quickly distinguish hosted, cloud, and restricted-access options. ### Connect to the provider Fill in the connection details and defaults: | Field | Purpose | | --- | --- | | **Connection Name** | Unique display name for this connection | | **Model Name** | Provider-specific model or deployment name, such as `gemini-1.5-pro` or `gpt-4-turbo` | | **API Endpoint** | Endpoint URL, for providers that need one (see the table below) | | **API Key** | Provider API key, when required | | **Default for Test Generation** | Use this model when generating tests | | **Default for Evaluation (LLM as Judge)** | Use this model for LLM-as-judge metrics | | **Default for Execution (Multi-Turn)** | Use this model for conversation simulation with Penelope | Click **Test Connection** before saving. ## Provider-Specific Connection Fields Some providers require endpoint configuration in addition to model name: | Provider | Provider value | API Endpoint required | API key required | Notes | |----------|----------------|-----------------------|------------------|-------| | LiteLLM Proxy | `litellm_proxy` | Yes | Optional | Pre-filled with `http://host.docker.internal:4000` by default | | Azure AI Studio | `azure_ai` | Yes | Yes | Use your Azure AI inference endpoint URL | | Azure OpenAI | `azure` | Yes | Yes | Use your Azure OpenAI resource endpoint URL | > If a model connection fails, use **Test Connection** in the connection dialog before saving. ## Polyphemus Access Control Polyphemus uses an explicit request-and-review workflow. If access is not granted yet, the model card shows **Access Required** and a **Request Access** action. ### Request Workflow 1. Open **Models** and click **Request Access** on the Polyphemus card. 2. Submit request details in the access modal. 3. The request is recorded and review notification is sent. 4. After approval, your account is marked as verified and the model becomes available. ### Request Fields | Field | Rules | |-------|-------| | `justification` | Required, 10-2000 characters | | `expected_monthly_requests` | Required, integer from 0 to 10,000 | ## Next steps - Build [Metrics](/docs/metrics) that use your connected models as the judge. - Generate [Tests](/docs/tests) with a model set as the generation default. --- url: https://docs.rhesis.ai/docs/organizations title: Organizations & Team --- # Organizations & Team Manage your organization's profile, settings, and team members. ![Rhesis team management](/screenshots/rhesis-ai-team.png) ## Organization Settings You can configure your organization's information and contact details in [Organization Settings](https://app.rhesis.ai/organizations/settings). ## Organization menu and sidebar The sidebar brand block shows the active project and organization. Open it to access organization settings, team management, projects, and **Switch project**. The main sidebar is organized into product areas: | Section | Pages | | --- | --- | | Define | Knowledge, Behaviors, Metrics | | Generate | Playground, Explorer, Tests, Test Sets | | Improve | Insights, Test Runs, Experiments, Traces, Tasks | | CONNECT | Endpoints, Models, Tools, API | Sidebar items you don't have permission to access are hidden rather than shown locked. Enterprise installations can also enable: - [Roles & Permissions](/docs/organizations/roles) for role-based access control - [Single Sign-On](/docs/organizations/sso) for per-organization OIDC login - [API Clients](/docs/organizations/api-clients) for machine-to-machine token exchange ## Team Management ### Inviting Team Members Send an invitation email from the [Team Page](https://app.rhesis.ai/organizations/team). Each invite has a status: - **Invited**: the email has been sent, but the user has not accepted yet. - **Active**: the user has accepted and is part of your organization. > Users cannot belong to multiple organizations simultaneously. ### Leaving an Organization 1. Navigate to Organization Settings 2. Scroll to "Danger Zone" 3. Click "Leave Organization" and type the organization name to confirm You'll immediately lose access and be signed out. A new invitation is required to rejoin. ## Next steps - Create [Projects](/docs/getting-started/projects) to organize your work - Configure [Integrations](/docs/integrations) for your workflow - Enable [Single Sign-On](/docs/organizations/sso) for enterprise identity --- url: https://docs.rhesis.ai/docs/playground title: Playground --- # Playground Interact with any endpoint — send messages, attach files, view traces, and capture test cases, no code required. [Playground] ## Getting Started Open the Playground and select the **Endpoints** action to open the endpoint drawer. Choose an endpoint, click **Select**, and start typing in the message box. The active project, endpoint, and environment appear in the chat header. Use the same action again when you want to switch endpoints. ### Attaching files For endpoints that accept file inputs, attach files directly in the chat composer. Playground supports images (JPEG, PNG, GIF, WebP), PDF, JSON, spreadsheets (XLSX, XLS), and CSV, up to 10 MB per message. For multi-modal endpoints, see [Multi-modal Testing](/docs/tests/multimodal-testing). ## Side-by-Side Comparison The Playground supports a split-screen mode where you can run two conversations in parallel against the same or different endpoints. This is useful for: - Comparing responses from different model configurations - Evaluating prompt changes before committing them - Checking how two endpoints handle the same inputs Each side operates independently — you can send different messages or share the same input to see how responses diverge. ## Viewing Traces After each response, click on the agent's reply to open the associated trace. The trace shows the full span tree for that turn — including any tool calls, retrieval steps, or agent handoffs triggered by your message. This gives you immediate feedback on what happened inside your endpoint without leaving the Playground. See [Tracing](/docs/tracing) for more on how traces are structured. ## Creating Tests Any conversation can be saved as a test directly from the Playground: - Click the **lab icon** on the conversation window to create a test from the current conversation. - Click the **plus icon** to open a new chat window and start a fresh conversation. You'll be prompted to assign a category, topic, behavior, and optional expected output before saving. Tests capture the full message context, including any attached files, so they can be replayed exactly during test execution. --- url: https://docs.rhesis.ai/docs/results-overview title: Results Overview --- # Results Overview The Test Results page gives an overview of the test results across test runs and across projects. Analyze aggregated [Test Run](/docs/test-runs) performance across multiple executions. In the Overview page, you can see a list of all test runs with summary metrics and pass rates. ![Results Overview](/screenshots/rhesis-ai-results-overview.png) You can also view statistics and graphs for: - **Pass Rate**: Pass/fail distribution, monthly trends, and run comparisons - **Dimensions**: Performance grouped by behavior, category, and topic - **Metrics**: Individual metric performance across test runs ## Next steps - View individual [Test Runs](/docs/test-runs) - Configure [Metrics](/docs/metrics) --- url: https://docs.rhesis.ai/docs/tasks title: Tasks --- # Tasks Tasks help teams coordinate testing work within Rhesis — tracking test failures, planning test development, and organizing reviews. Each task can be linked to a specific test, test set, or test result to provide full context for the work that needs to be done. ![Tasks page — work items with status and assignee, filtered by state](/screenshots/rhesis-ai-tasks.webp) ## Task Properties Each task includes the following properties to help organize and track work: | Property | Options | Description | |----------|---------|-------------| | **Status** | Open, In Progress, Completed, Cancelled | The current state of the task | | **Priority** | Low, Medium, High | The urgency level for completing the task | | **Creator** | User | Person who created the task (automatically set) | | **Assignee** | User (optional) | Team member responsible for completing the task | | **Title** | Text | Short, descriptive name for the task | | **Description** | Text | Detailed information about the task | ### Linked Resources Tasks can be associated with specific resources to provide context: | Resource Type | Purpose | |---------------|---------| | **Test** | Link to a specific test that needs attention or development | | **Test Set** | Link to a collection of tests for batch operations or reviews | | **Test Result** | Link to a test execution result that requires follow-up | ## Jira Integration Create Jira issues directly from tasks. First, [set up a Jira tool connection](/docs/tools#jira) with a configured project key. Once connected, use the **Create Jira Issue** button on any task to automatically create a linked issue in your Atlassian workspace. ## Creating Tasks You can create tasks from the [Tasks](https://app.rhesis.ai/tasks) page, or across the platform in [Test Results](https://app.rhesis.ai/test-results), [Tests](https://app.rhesis.ai/tests) or [Test Sets](https://app.rhesis.ai/test-sets). ## Next steps - Create tasks from [Test Results](/docs/test-results) - Link tasks to [Tests](/docs/tests) - Organize work by [Projects](/docs/getting-started/projects) --- url: https://docs.rhesis.ai/docs/test-results title: Test Results --- # Test Results > **This page has moved.** Test result analytics now live under [Insights](/docs/results-overview). --- url: https://docs.rhesis.ai/docs/test-runs title: Test Runs --- # Test Runs A test run is created when you execute a test set against an endpoint. It captures all individual test results with detailed metrics, conversation history, execution metadata, and performance data for analysis. ![Test Runs page — pass rate, status, executor, and review count per run](/screenshots/rhesis-ai-test-runs.webp) ## Test Runs List Use the list to compare status, pass rate, executor, and review activity across runs. The **Reviews** column shows how many tests received human reviews. Hover the value to see how many reviewed tests were corrected because the human verdict changed the automated result. Open the filter drawer and set **Reviews** to **With** or **Without** to include only runs that have reviews or runs that do not. By clicking on a test run, you'll see the overview of how the test run went. From here you can: - **Review**: Manually revise the automated test evaluation - **Compare**: Compare against a baseline test run - **Re-run**: Execute the test set again with the same configuration - **Download**: Export test run data as CSV ![Test run summary — pass rate, tests executed, duration and review count, with pass rates broken down by behavior and by metric](/screenshots/rhesis-ai-test-run-details.webp) ## Test Run Detail View The test run detail page includes an expanded **Overview** and **Conversation** workflow for debugging failures faster: - **Go to Test**: Open the original test definition directly from a result detail - **Context panel**: Expand/collapse captured context items when available - **Metadata panel**: Expand/collapse structured metadata for the result - **Files / Output Files panels**: Inspect and download both input attachments and generated output files - **JSON-aware rendering**: Response content that is valid JSON is pretty-printed in the overview tab ![Overview tab of a test result — status, prompt, response, expected response, retrieved context, and result tags](/screenshots/rhesis-ai-test-results-overview.webp) For multi-turn tests, the **Conversation** tab supports trace-driven debugging: - Click assistant responses to open the trace drawer - Navigate turn-by-turn while keeping test result context visible ![Conversation tab of a test result — each turn with its message, the agent reply, and a per-turn pass or fail status](/screenshots/rhesis-ai-test-results-conversation.webp) ## Parameters Used When a test run targets a parameter experiment, it snapshots the experiment's resolved values at queue time. This guarantees that test runs remain reproducible even if the experiment or environment changes later. The Test Run detail page displays an **Experiment** card showing the experiment name, version (e.g., `v3`), the source environment (if any), and an expandable **Resolved Values** panel. (Values for `secret_ref` slots are masked so credentials never leak into the test run record.) Because Test Results inherit this snapshot from their parent test run, there is no need to track configurations on a per-result basis. The test-runs list page also includes an Experiment filter to help you slice historical results by configuration. ## Multi-turn Result Details For multi-turn tests, the run detail includes conversation-specific execution data: - **Turns used**: How many turns were consumed during execution - **Configured max turns**: The max turn budget from test configuration - **Goal achieved flag**: Whether the multi-turn goal was met - **Conversation-linked trace access**: Open the trace drawer to inspect tree, sequence, graph, and conversation views for the related trace ## Cancel In-Progress Test Runs You can cancel runs that are still waiting or running: 1. Open the **Test Runs** list view 2. Select one or more runs with status **Queued** or **Progress** 3. Click **Cancel Test Run** or **Cancel Test Runs** 4. Confirm the action The API route used by the UI is `POST /test_runs/{id}/cancel`. | Current status | Cancel allowed | Result | |---|---|---| | `Queued` | Yes | Run is marked `Cancelled` immediately | | `Progress` | Yes | Run is marked `Cancelled`; in-flight work stops cooperatively | | `Completed` / `Failed` | No | API returns `409` | | Missing run ID | No | API returns `404` | ```bash curl -X POST "$RHESIS_BASE_URL/test_runs/$TEST_RUN_ID/cancel" \\ -H "Authorization: Bearer $RHESIS_API_TOKEN" ``` > Cancellation sets the run status to `Cancelled` immediately so dashboards and > run lists update without waiting for all worker tasks to finish. ## Review Test Runs Reviews allow human evaluators to validate or override automated test evaluations. 1. Select a Test Run from the [Test Runs](https://app.rhesis.ai/test-runs) overview page 2. Click on the **Reviews** tab, then **Add Your Review** 3. Select a **Review Status**: - **Pass**: The test passes the metric - **Fail**: The test fails the metric 4. Add a comment explaining your review decision ### Review targets and override behavior Reviews can now target different parts of a test result, not only the overall result. | Target type | What it reviews | Reference format | |---|---|---| | `test_result` | Overall pass/fail decision for the test result | `null` | | `metric` | A specific metric result in `test_metrics` | Metric name | | `turn` | A specific turn in multi-turn conversation output | Turn label such as `Turn 3` | When a review targets a metric or turn, Rhesis stores an override marker on the affected object and recomputes effective status in the test run views. > In review comments, type `@` to mention metrics or turns. The first metric or turn mention determines the > review target automatically. On a test run detail page, open the filter drawer and use **Review Status** to show reviewed, unreviewed, or conflicting results. **Conflicting** shows results where a human review disagrees with the automated verdict. ```text @[Goal Achievement](metric:goal-achievement) @[Turn 2](turn:turn-2) ``` ## Compare Test Runs Compare test runs to identify regressions and improvements between executions. **How to Compare:** 1. Click the **Compare** button (top right, next to Download) 2. Select a baseline test run to compare against 3. View the test-by-test comparison of the test set **Comparison Filters:** Use filters to focus on specific changes: - **All Tests**: Show all tests from both runs - **Improved**: Tests that now pass but failed in baseline - **Regressed**: Tests that now fail but passed in baseline - **Unchanged**: Tests with the same pass/fail status ## Test Run Metrics Each test result records the metrics it was evaluated against. Which metrics run is resolved from a priority hierarchy (execution-time → test set → behavior) configured when you start the run. See [Test Execution](/docs/test-runs/execution#test-run-metrics) for how to set metrics at each level. The **Metrics** tab of a result lists every metric with its verdict and the reason the judge gave. Filter to **Passed** or **Failed** to narrow a long list. ![Metrics tab of a test result — overall performance, the behavior it belongs to, and a pass or fail verdict with a reason per metric](/screenshots/rhesis-ai-test-results-metrics.webp) ## Next steps - Review failed tests to understand issues - Compare against baseline runs to detect regressions - Add human reviews to test results - Export results for reporting or analysis --- url: https://docs.rhesis.ai/docs/test-sets title: Test Sets --- # Test Sets Test sets group related tests into collections for batch execution. When you [generate tests](/docs/test-sets/tests-generation), all created tests are automatically grouped into a single test set. You can also manually assign tests to test sets or remove them as needed. Test sets inherit shared types, behaviors, categories, topics and sources from their tests. ![Test Sets page — each set with its behaviors, categories, type, and test count](/screenshots/rhesis-ai-test-sets.webp) ## Test Sets Page The [Test Sets page](https://app.rhesis.ai/test-sets) is the central place to manage all your test sets. It displays a summary of test set activity in charts at the top, followed by a searchable, filterable grid of all test sets in your organization. From this page you can: - **Create** a new empty test set - **Import** tests from a file or from Garak (see below) - **Execute** one or more test sets against an endpoint - **Delete** test sets you no longer need Click any test set name to open its detail page, where you can inspect individual tests, view metrics, and manage tags. ## Importing Test Sets Rhesis provides two ways to bring existing tests into a test set without generating them from scratch. ### From a File Import tests from a **CSV**, **Excel**, **JSON**, or **JSONL** file. Rhesis analyses your file's structure, suggests a column mapping to Rhesis test fields, and lets you review and adjust before committing. Both Single-Turn and Multi-Turn tests are supported. [Import from File →](/docs/test-sets/import-from-file) ### From Garak [Garak](https://github.com/NVIDIA/garak) is an open-source LLM vulnerability scanner. Rhesis integrates its full probe library directly into the platform — select the probes you want, and Rhesis creates one test set per probe, pre-populated with prompts. Garak **detectors are automatically mapped as Rhesis metrics**, so imported test sets are ready to evaluate immediately. Probes come in two types: **static** (prompts bundled with the probe) and **dynamic** (prompts generated at runtime by an LLM). Both are supported. [Import from Garak →](/docs/test-sets/import-from-garak) ## Test Set Types Every test set has a type that determines how its tests are executed: | Type | Description | | --- | --- | | **Single-Turn** | Tests that evaluate individual prompt/response exchanges. Each test sends a single input and evaluates the response. Ideal for RAG systems, classification tasks, and standalone response quality. | | **Multi-Turn** | Tests that evaluate conversational interactions across multiple turns. Each test defines a goal and the system conducts an automated multi-turn conversation to assess the endpoint's behavior. Ideal for chatbots, agents, and dialogue systems. | The test set type is set when the test set is created and determines which metrics can be applied during evaluation. When generating tests or importing from files, the type is inferred automatically from the tests: if any test is multi-turn, the test set is classified as Multi-Turn. ## Executing Test Sets Executing a test set runs all its tests against your LLM application endpoint to see how your application responds. This creates a [Test Run](/docs/test-runs) that captures all results. To execute a test set, select it from the [Test Sets page](https://app.rhesis.ai/test-sets) and configure: **Execution Target** - **Project**: The current project - **Endpoint**: The endpoint within the project to execute tests against **Execution Mode** - **Parallel** (default): Tests run simultaneously for faster execution - **Sequential**: Tests run one after another, better for rate-limited endpoints **Model Settings** - **Evaluation Model**: Override the default evaluation model for this run - **Execution Model** (multi-turn only): Override the default execution model used by Penelope For details on all execution options, see [Test Execution](/docs/test-runs/execution). **Tags**: Optional tags to categorize and find this test run ## Next steps - [Import from File](/docs/test-sets/import-from-file) to create a test set from CSV, Excel, JSON, or JSONL - [Import from Garak](/docs/test-sets/import-from-garak) to import Garak vulnerability probes - [Generate tests](/docs/test-sets/tests-generation) to create test sets automatically - View execution progress in [Results Overview](/docs/results-overview) - Track historical performance in [Test Runs](/docs/test-runs) --- url: https://docs.rhesis.ai/docs/test-sets-runs title: Test Sets --- # Test Sets > **This page has moved.** Test sets are now documented under [Test Sets](/docs/test-sets). --- url: https://docs.rhesis.ai/docs/tests title: Tests --- # Tests Tests are individual cases that validate specific inputs and expected outputs for your LLM application, evaluated using assigned metrics. ![Tests page — prompts with their behavior, topic, category, and test type](/screenshots/rhesis-ai-tests.webp) ## Test Types There are two types of tests: - **Single-turn** tests check how the LLM application responds to a single prompt with no follow-up (Q&A). - **Multi-turn** tests check how it behaves over multiple messages in a conversation. ### Single-Turn Tests A single prompt sent to your LLM application, evaluated against expected outputs and metrics. **Properties:** | Field | Description | |-------|-------------| | **Test Prompt** | The input text sent to your LLM application | | **Files** | (Optional) File attachments (images, PDFs, or audio) sent alongside the prompt. See [Multi-modal Testing](/docs/tests/multimodal-testing) for supported types and size limits, and [file format filters](/docs/endpoints/request-mapping#file-format-filters) for how files are mapped to provider formats. | | **Category** | High-level classification (e.g., Harmful, Harmless) | | **Topic** | Specific subject matter (e.g., healthcare, financial advice) | | **Behavior** | Type of behavior to validate (e.g., Compliance, Reliability, Robustness) | | **Expected Output** | (Optional) What the LLM application should respond with | ### Multi-Turn Tests Goal-based conversations that test your LLM application across multiple turns. Powered by [Penelope](/docs/tests/conversation-simulation), an autonomous testing agent that adapts its strategy based on responses. Ideal for testing conversational workflows. **Properties:** | Field | Description | |-------|-------------| | **Goal** | What the target should do - the success criteria for this test | | **Instructions** | (Optional) How to conduct the test - if not provided, the agent plans its own approach | | **Restrictions** | (Optional) What the target must not do - forbidden behaviors or boundaries | | **Scenario** | (Optional) Context and persona for the test - narrative setup or user role | | **Files** | (Optional) File attachments (images, PDFs, or audio) included with the test. Files are available on each conversation turn via the `files` variable in the endpoint request template. See [Multi-modal Testing](/docs/tests/multimodal-testing) for supported types and size limits. | | **Min. Turns** | Minimum turns before early stopping is allowed. If omitted, defaults to 80% of max turns. | | **Max. Turns** | Maximum number of conversation turns allowed (`max_turns`) | | **Category** | High-level classification (e.g., Harmful, Harmless) | | **Topic** | Specific subject matter (e.g., healthcare, financial advice) | | **Behavior** | Type of behavior to validate (e.g., Compliance, Reliability, Robustness) | ## Creating Tests Create tests manually or generate them automatically from behaviors and requirements. See [Test Generation](/docs/test-sets/tests-generation) for automated generation. ### Manual Test Writer Use **Manual Test Writer** to enter or paste multiple tests without running AI generation. It provides a spreadsheet-style workflow for single-turn and multi-turn test rows. | Test type | Fields | | --- | --- | | Single-turn | Prompt, category, topic, behavior, expected output, and optional file attachments | | Multi-turn | Goal, instructions, restrictions, scenario, min turns, max turns, category, topic, behavior, and optional file attachments | From the Tests page, choose manual creation, select the test type, add or remove rows, attach files when needed, then save the rows as tests or as a test set. Use the single-test drawer when you only need to create or edit one test. ## Running Tests Run individual tests from the [Tests page](https://app.rhesis.ai/tests), or execute multiple tests together as [Test Sets](/docs/test-sets). ## Viewing Execution History Open a test and select **Execution History** to see how it performed across test runs. The table shows the effective Pass or Fail status, test run, number of passing metrics, and execution time. Select a test run to open the matching result in a new tab. The history loads up to 100 recent results and shows one entry per test run. It remains empty until the test has been executed as part of a test set. ## Next steps - Organize tests into [Test Sets](/docs/test-sets) - Generate tests from [Knowledge](/docs/knowledge) - View and filter execution results in [Test Runs](/docs/test-runs) --- url: https://docs.rhesis.ai/docs/tools title: Tools --- # Tools Tools connect external services to Rhesis so you can export their content into your [Knowledge](/docs/knowledge) base for use in test generation and other workflows. > This page covers importing knowledge into Rhesis from tools like Notion, GitHub, and Jira. If you want to call Rhesis from Cursor or Claude Code to explore endpoints and run tests, see [Agent Skill](/docs/agent-skill) instead. ![Tools Page](/screenshots/rhesis-ai-mcp.png) ## Setting Up a Tool Connection Every connection follows the same flow in Rhesis: on the **Tools** page, click **Add tool connection**, pick the provider, fill in the connection name, token, and any provider-specific fields listed below, then **Test the connection** and **Save**. Each provider below covers only how to generate its token and which extra fields it needs. [] Notion 1. Create a Notion integration at [notion.so/profile/integrations/internal](https://www.notion.so/profile/integrations/internal/). 2. In the "Content access" tab, grant access to the pages and teamspaces you want to export from — required for the integration to reach your content. 3. In Rhesis, paste the token (starts with `ntn...`). [] GitHub 1. On GitHub, go to Settings → Developer Settings → Personal Access Tokens → Tokens (classic). 2. Generate a classic token with the `repo` scope (full repository access). 3. In Rhesis, add the repository URL and paste the token (starts with `ghp...`). [] Jira 1. Go to [id.atlassian.com/manage-profile/security](https://id.atlassian.com/manage-profile/security). 2. Under API tokens, click "Create API token" (not "Create API token with scopes" — Rhesis needs the classic unscoped token). Verify your identity, name the token, and set an expiration. 3. In Rhesis, also enter your Atlassian email and workspace URL (`yourorg.atlassian.net`), then paste the token. [] GitLab 1. Go to avatar → Edit profile → Access → Personal access tokens. 2. Generate a "Legacy token" (not "Fine-grained token") with the `api` scope (or `read_api` for read-only). Name it and set an expiration. 3. In Rhesis, add the project namespace (`group/project`, or a full project URL) and paste the token (starts with `glpat-`). Self-managed instances also need the GitLab API URL. [] Asana 1. Go to profile photo → My Settings → Apps → "Manage Developer Apps". 2. Under "Personal access tokens", create a new token and copy it (shown only once). 3. In Rhesis, paste the token. Optionally add a workspace GID to scope search and import. [] Azure DevOps 1. Sign in at `https://dev.azure.com/{your_organization}`, then go to user settings → Personal access tokens. 2. Create a token with the required scopes (e.g. Work Items (Read)) or Full access, set an expiration, and copy it (shown only once). 3. In Rhesis, enter your organization name (not the full URL), your Azure DevOps email, and the project, then paste the token. [] Linear 1. In Linear, go to Settings → Account → Security & access. 2. Under "Personal API keys", create an API key with an access level and copy it (shown only once, starts with `lin_api_`). 3. In Rhesis, paste the token. [] Shortcut 1. In Shortcut, go to Settings → Your Account → API Tokens. 2. Generate a token and copy it (grants full workspace access). 3. In Rhesis, paste the token. Watch this video to see how to connect your Notion workspace and export pages as knowledge sources: