# Rhesis Documentation > Rhesis: AI-powered testing and evaluation platform for Gen AI applications. Collaborative testing that brings together developers, domain experts, and stakeholders. ## Docs - [Rhesis Documentation](https://docs.rhesis.ai/docs.md): **More than just evals. The testing platform for AI teams.** - [Acknowledgments](https://docs.rhesis.ai/docs/acknowledgments.md): Rhesis includes and depends on several excellent open-source projects. We are grateful to the developers and maintainers of these projects. - [Agent Skill](https://docs.rhesis.ai/docs/agent-skill.md): Use Rhesis from Claude Code, Cursor, or any compatible AI interface — without the Rhesis web UI. - [API Tokens](https://docs.rhesis.ai/docs/api-tokens.md): Generate and manage API tokens for programmatic access to Rhesis. You can manage them at [app.rhesis.ai/tokens](https://app.rhesis.ai/tokens). - [Rhesis Architect](https://docs.rhesis.ai/docs/architect.md): Architect (also known as Telemachus) is a conversational agent that designs, builds, and runs test suites for your AI endpoints. Describe what you want to test ... - [Behaviors](https://docs.rhesis.ai/docs/behaviors.md): Define what good looks like for your AI system and attach metrics to measure it. - [Core Concepts – Rhesis](https://docs.rhesis.ai/docs/concepts.md): Rhesis evaluates whether your AI application meets the quality bar you care about. This page has two parts: the **workflow** (what you do and in what order) and... - [Deployment](https://docs.rhesis.ai/docs/deployment.md): We recommend using **Rhesis Cloud** for the fastest, most reliable experience without operational overhead. We also understand that some teams have strict data ... - [Endpoints – Rhesis](https://docs.rhesis.ai/docs/endpoints.md): An endpoint is how Rhesis connects to your AI model or application to send test inputs and evaluate responses. - [Experiments](https://docs.rhesis.ai/docs/experiments.md): An *Experiment* is one attempt at a configuration. *Versions* are its history. *Environments* are how the world finds it. - [Test Explorer](https://docs.rhesis.ai/docs/explorer.md): Test Explorer is an interactive workspace for building a test set. Draft or import tests, invoke your endpoint, and see responses and metric scores in the same ... - [Getting Started with Rhesis](https://docs.rhesis.ai/docs/getting-started.md): Welcome to Rhesis! This guide will help you get started with testing and evaluating your Generative AI applications. - [Integrations](https://docs.rhesis.ai/docs/integrations.md): Rhesis plugs into your LLM stack across four layers, each addressing a different concern. This page is the single map: pick the layer you care about and follow ... - [Knowledge](https://docs.rhesis.ai/docs/knowledge.md): Knowledge is domain-specific content that you add to Rhesis to generate relevant, context-aware test cases. Knowledge is made up of **Sources**. - [MCP](https://docs.rhesis.ai/docs/mcp.md): MCP (Model Context Protocol) servers allow you to export sources from external tools into your Rhesis Knowledge base. These sources can then be used in test gen... - [Metrics](https://docs.rhesis.ai/docs/metrics.md): Define and manage evaluation criteria for testing AI responses with LLM-based grading. - [Models](https://docs.rhesis.ai/docs/models.md): **What are Models?** Models are AI service providers that you can set up for your evaluation and testing workflows. - [Organizations & Team](https://docs.rhesis.ai/docs/organizations.md): Manage your organization's profile, settings, and team members. - [Playground](https://docs.rhesis.ai/docs/playground.md): Interact freely with any endpoint — send messages, attach files, view traces, and capture test cases — all without writing a line of code. - [Product Tour](https://docs.rhesis.ai/docs/product-tour.md): You spent weeks or months building something innovative. Don't let inadequate testing ruin the release. Traditional testing approaches fail with Gen AI because ... - [Results Overview](https://docs.rhesis.ai/docs/results-overview.md): The Test Results page gives an overview of the test results across test runs and across projects. Analyze aggregated [Test Run](/docs/test-runs) performance acr... - [Tasks](https://docs.rhesis.ai/docs/tasks.md): Organize and track testing work with integrated task management that connects directly to your tests, test sets, and results. - [Test Results](https://docs.rhesis.ai/docs/test-results.md): **This page has moved!** Test results are now available under [Results Overview](/docs/results-overview). - [Test Runs](https://docs.rhesis.ai/docs/test-runs.md): View execution results when you run a test set against an endpoint. Each test run contains all individual test results with detailed metrics, conversation histo... - [Test Sets](https://docs.rhesis.ai/docs/test-sets.md): **What are Test Sets?** Test Sets are groups of tests that can be executed together. - [Test Sets](https://docs.rhesis.ai/docs/test-sets-runs.md): **This page has moved!** Test sets are now available under [Test Sets](/docs/test-sets). - [Tests](https://docs.rhesis.ai/docs/tests.md): Individual test cases that evaluate specific aspects of your AI system's performance. - [Tracing – Rhesis](https://docs.rhesis.ai/docs/tracing.md): **OpenTelemetry-based observability for AI applications** - [Chat Features](https://docs.rhesis.ai/docs/architect/chat-features.md): You can attach documents to any message. Architect extracts the text and uses it as context for that turn — for example, to ground test generation in a spec you... - [Running and Analyzing](https://docs.rhesis.ai/docs/architect/execution-and-analysis.md): After you approve the plan, Architect runs the tests and presents results in a structured format. You can also ask it to analyze existing runs or compare two ru... - [Endpoint Exploration](https://docs.rhesis.ai/docs/architect/exploration.md): Before designing a test suite, Architect explores your endpoint — running real conversations against it to understand what it does and how it behaves. - [Planning Test Suites](https://docs.rhesis.ai/docs/architect/planning.md): After exploration, Architect proposes a structured plan. You review it, ask for changes, and approve before anything is created. - [Scenarios](https://docs.rhesis.ai/docs/architect/scenarios.md): Common requests and what to expect from each one. - [Workflow](https://docs.rhesis.ai/docs/architect/workflow.md): An Architect session follows five phases. You stay in control throughout — Architect proposes and waits; it never creates or modifies anything without your conf... - [Running Locally](https://docs.rhesis.ai/docs/deployment/running-locally.md): Get Rhesis running on your local machine in under 5 minutes without setup. - [Self-Hosting](https://docs.rhesis.ai/docs/deployment/self-hosting.md): Deploy Rhesis on your own infrastructure with proper security, authentication, and production configuration. - [Auto-Configure](https://docs.rhesis.ai/docs/endpoints/auto-configure.md): Let AI generate your request and response mappings automatically. Paste any reference material about your endpoint and Rhesis handles the rest. - [Creating Endpoints](https://docs.rhesis.ai/docs/endpoints/creating-endpoints.md): There are two ways to set up an endpoint: - [Managing Endpoints](https://docs.rhesis.ai/docs/endpoints/management.md): Open an endpoint and go to the **Test** tab to send a live request and verify your configuration before running full test suites. - [Mapping Examples](https://docs.rhesis.ai/docs/endpoints/mapping-examples.md): Practical examples showing how to configure request and response mappings for different API patterns. Each example includes the request body template, response ... - [Multi-Turn Conversations](https://docs.rhesis.ai/docs/endpoints/multi-turn-conversations.md): Test conversational AI systems that maintain context across multiple interactions. - [Request Mapping](https://docs.rhesis.ai/docs/endpoints/request-mapping.md): Request body templates define how Rhesis formats its [platform variables](/docs/endpoints#platform-managed-variables) into the JSON structure your API expects. ... - [Response Mapping](https://docs.rhesis.ai/docs/endpoints/response-mapping.md): Response mappings tell Rhesis how to extract [platform variables](/docs/endpoints#platform-managed-variables) from your API's response. Each field is evaluated ... - [SDK Endpoints](https://docs.rhesis.ai/docs/endpoints/sdk-endpoints.md): Register Python functions as testable endpoints using the Rhesis SDK. Instead of configuring endpoints manually in the dashboard, you decorate your functions an... - [Parameter Injection](https://docs.rhesis.ai/docs/experiments/connector-injection.md): Experiment parameters flow through for both REST and SDK connector endpoints. The same syntax works everywhere. - [Parameter Schema](https://docs.rhesis.ai/docs/experiments/parameter-schema.md): A *Parameter Schema* is the foundation for experiments. It declares the typed knobs your project supports — what can be configured, not what the values are. [Ex... - [SDK Usage](https://docs.rhesis.ai/docs/experiments/sdk-usage.md): How your application code reads parameter values, with the right caching and override semantics for production. - [Building and Evaluating](https://docs.rhesis.ai/docs/explorer/building-and-evaluating.md): This page covers the core Test Explorer workflow: create a session, choose the endpoint and metrics, build a topic tree, generate outputs, evaluate, and export. - [Scenarios](https://docs.rhesis.ai/docs/explorer/scenarios.md): Common ways to use Test Explorer and what to expect from each one. - [Test Generation](https://docs.rhesis.ai/docs/explorer/tests-generation.md): Generate test sets automatically using AI-powered templates or custom configurations. - [Workflow](https://docs.rhesis.ai/docs/explorer/workflow.md): Test Explorer combines manual iteration with a **suggestions pipeline**: you organize tests and topics, then optionally generate a batch of new prompts, embed a... - [Connect Your Application – Rhesis](https://docs.rhesis.ai/docs/getting-started/connecting-application.md): Once your Rhesis environment is set up, the next step is to connect your Generative AI application. Rhesis acts as an orchestrator, sending test prompts to your... - [Default Insurance Chatbot (Rosalind)](https://docs.rhesis.ai/docs/getting-started/default-chatbot.md): Get started instantly with Rhesis AI by exploring our pre-configured insurance chatbot, Rosalind. Available to all new users during onboarding, this demo endpoi... - [Projects](https://docs.rhesis.ai/docs/getting-started/projects.md): Organize and manage your testing projects with comprehensive project management capabilities. - [Run Evaluations](https://docs.rhesis.ai/docs/getting-started/run-evaluations.md): With your environment set up and your application connected, you are ready to run evaluations. In Rhesis, a group of test cases used to evaluate an application ... - [Setup Environment](https://docs.rhesis.ai/docs/getting-started/setup-environment.md): Before you can start testing your Generative AI applications, you need to set up your Rhesis environment. Rhesis offers three flexible deployment options to sui... - [Code Metrics](https://docs.rhesis.ai/docs/metrics/code-metrics.md): Code Metrics let you write evaluation logic directly in Python — on your own machine or infrastructure — and have the Rhesis backend invoke it automatically dur... - [DeepEval](https://docs.rhesis.ai/docs/metrics/deepeval.md): [DeepEval](https://github.com/confident-ai/deepeval) provides single-turn and conversational metrics for RAG, agents, toxicity, bias, and PII checks. Rhesis exp... - [Ragas](https://docs.rhesis.ai/docs/metrics/ragas.md): [Ragas](https://github.com/explodinggradients/ragas) evaluates RAG pipelines with referenceless and reference-based metrics (faithfulness, context relevance, an... - [Trace Metrics](https://docs.rhesis.ai/docs/metrics/trace-metrics.md): Trace Metrics let you run evaluation metrics against live production traces automatically. Every time your endpoint handles a request, Rhesis can evaluate the r... - [API Clients](https://docs.rhesis.ai/docs/organizations/api-clients.md): API Clients let external services exchange an organization-scoped OIDC access token for a Rhesis JWT using RFC 8693 token exchange. Use them for machine-to-mach... - [Single Sign-On](https://docs.rhesis.ai/docs/organizations/sso.md): Single Sign-On lets an Enterprise Edition organization authenticate users through its own OIDC provider. Use it when your organization needs centralized identit... - [Test Reviews](https://docs.rhesis.ai/docs/tasks/test-reviews.md): Override automated test evaluations with human judgement. When a test run produces results you disagree with, reviews let you correct them at the target that ma... - [Test Execution – Rhesis](https://docs.rhesis.ai/docs/test-runs/execution.md): Execute test sets against your AI endpoints to evaluate model performance, safety, and reliability. This guide explains the different configuration options avai... - [Import from File](https://docs.rhesis.ai/docs/test-sets/import-from-file.md): Import an existing collection of tests directly from a CSV, Excel, JSON, or JSONL file to create a new test set in seconds. - [Import from Garak](https://docs.rhesis.ai/docs/test-sets/import-from-garak.md): [Garak](https://github.com/NVIDIA/garak) is an open-source LLM vulnerability scanner developed by NVIDIA. Rhesis integrates Garak directly into the platform so ... - [Adversarial Testing](https://docs.rhesis.ai/docs/tests/adversarial-testing.md): Standard software testing verifies that systems work under expected conditions. Adversarial testing probes for failures when inputs are unexpected, malicious, o... - [Conversation Simulation](https://docs.rhesis.ai/docs/tests/conversation-simulation.md): **Intelligent Multi-Turn Testing for AI Applications** - [Multi-modal Testing](https://docs.rhesis.ai/docs/tests/multimodal-testing.md): Multi-modal testing lets you attach files to tests and have them sent to your endpoint alongside the prompt. Use this to test how your AI handles images, docume... - [Microsoft Agent Framework](https://docs.rhesis.ai/docs/tracing/agent-framework.md): Zero-config tracing for [Microsoft Agent Framework](https://github.com/microsoft/agent-framework) (MAF) agents and workflows. - [Auto-Instrumentation](https://docs.rhesis.ai/docs/tracing/auto-instrumentation.md): Zero-config tracing for LangChain and LangGraph applications. - [Conversation Tracing](https://docs.rhesis.ai/docs/tracing/conversation-tracing.md): See every conversation turn, every agent handoff, in one place. Rhesis groups traces into full conversation threads and visualizes the full multi-agent interact... - [Custom Spans](https://docs.rhesis.ai/docs/tracing/custom-spans.md): Create custom spans with specific names and attributes for advanced observability needs. - [Decorators](https://docs.rhesis.ai/docs/tracing/decorators.md): Two decorators are available: for tracing-only instrumentation, and for functions that Rhesis can **invoke from the platform** (remote testing) while still emit... - [Multi-Agent Tracing](https://docs.rhesis.ai/docs/tracing/multi-agent.md): Multi-agent tracing captures how agents in a system interact — which agent ran, what it received and produced, and when control passed from one agent to another... - [Semantic Conventions](https://docs.rhesis.ai/docs/tracing/semantic-conventions.md): Rhesis uses a semantic layer for consistent, framework-agnostic span naming across all AI operations. - [Setup](https://docs.rhesis.ai/docs/tracing/setup.md): Configure tracing in your application to start capturing traces. - [Polyphemus](https://docs.rhesis.ai/docs/tests/adversarial-testing/polyphemus.md): Polyphemus is a Rhesis-managed language model built for adversarial test generation. It is designed to produce the kinds of prompts that commercial models routi... - [Requesting Access](https://docs.rhesis.ai/docs/tests/adversarial-testing/requesting-access.md): Polyphemus can generate sensitive content for evaluation purposes. For this reason, access is not automatic—each request is reviewed by the Rhesis team before a... - [Using Polyphemus with the SDK](https://docs.rhesis.ai/docs/tests/adversarial-testing/sdk-usage.md): Once your access is approved, Polyphemus integrates with the Rhesis SDK the same way any other model provider does. You can use it with any synthesizer to gener... - [Configuration](https://docs.rhesis.ai/docs/tests/conversation-simulation/configuration.md): Configure Penelope's behavior through agent initialization, environment variables, or programmatic configuration. - [Examples & Use Cases](https://docs.rhesis.ai/docs/tests/conversation-simulation/examples.md): Complete working examples and practical patterns for testing conversational AI systems with Penelope. - [Execution Trace](https://docs.rhesis.ai/docs/tests/conversation-simulation/execution-trace.md): Penelope captures comprehensive execution traces that provide complete visibility into multi-turn test runs. These traces are structured, machine-readable, and ... - [Extending Penelope](https://docs.rhesis.ai/docs/tests/conversation-simulation/extending.md): Understand Penelope's architecture and learn how to extend it with custom tools for specialized testing needs. - [Getting Started](https://docs.rhesis.ai/docs/tests/conversation-simulation/getting-started.md): Get Penelope installed and run your first test in minutes. ## Guides - [Guides – Rhesis](https://docs.rhesis.ai/guides.md): Step-by-step guides to help you get the most out of Rhesis. Whether you're just getting started or building advanced custom solutions, these guides provide prac... - [How to Integrate Rhesis SDK into Your CI/CD Pipeline](https://docs.rhesis.ai/guides/ci-cd-integration.md): Automate LLM testing in your CI/CD pipeline to catch quality issues before they reach production. This guide shows you how to set up automated test execution, q... - [Building Custom Metrics with the Rhesis SDK](https://docs.rhesis.ai/guides/custom-metrics.md): While Rhesis provides integrations with DeepEval, Ragas, and other evaluation frameworks, you'll often need custom metrics tailored to your specific use case. T... - [How to Start Testing LLM and Agentic Apps in 10 Minutes with Rhesis AI](https://docs.rhesis.ai/guides/quick-start-guide.md): Testing LLM and agentic apps is challenging: outputs are non-deterministic, edge cases are unpredictable, and manual testing doesn't scale. This guide shows you... - [How to: Testing user journeys of AI agents](https://docs.rhesis.ai/guides/testing-user-journeys.md): You spent weeks crafting your agent: mapping **user journeys**, writing requirements, implementing features and tuning prompts and tools until the experience fe... ## SDK - [Agents](https://docs.rhesis.ai/sdk/agents.md): The SDK agents framework lets you build conversational workflows that can reason, call tools, and return structured results. Use it when a workflow needs multip... - [RhesisClient – Rhesis Python SDK](https://docs.rhesis.ai/sdk/client.md): The is the central entry point for the Rhesis SDK. It initializes telemetry, manages the connector for remote testing, and provides access to the Rhesis API. - [Connector](https://docs.rhesis.ai/sdk/connector.md): Register Python functions as testable endpoints in Rhesis using the SDK connector. This code-first approach automatically creates and manages endpoints, providi... - [Entities](https://docs.rhesis.ai/sdk/entities.md): Entities are the SDK's interface to the Rhesis platform. They provide typed Python objects that map directly to platform resources, enabling you to create, read... - [Test Execution](https://docs.rhesis.ai/sdk/execution.md): The SDK provides methods on for executing tests against endpoints, re-scoring existing outputs with different metrics, and managing which metrics are assigned t... - [SDK Installation & Setup – Rhesis](https://docs.rhesis.ai/sdk/installation.md): Get started with the Rhesis Python SDK - install the package, configure authentication, and learn the fundamental concepts including models, synthesizers, and m... - [Overview](https://docs.rhesis.ai/sdk/metrics.md): The Rhesis SDK provides a comprehensive metrics system for evaluating LLM-based systems. The metrics module supports multiple evaluation frameworks and allows y... - [Models](https://docs.rhesis.ai/sdk/models.md): Models are core components of the SDK that enable you to create and manage test sets for Gen AI applications. Models also play a crucial role in the evaluation ... - [Parameters & Experiments](https://docs.rhesis.ai/sdk/parameters.md): Manage typed configuration for your AI application. Define a schema once, create experiments with different values, and resolve the active configuration at runt... - [Statistics](https://docs.rhesis.ai/sdk/statistics.md): The SDK provides built-in methods for querying aggregated statistics across test runs and test results. Use these to build dashboards, generate reports, or anal... - [Synthesizers](https://docs.rhesis.ai/sdk/synthesizers.md): Generate test sets for evaluating LLM applications. Synthesizers use LLMs to create diverse, targeted test cases based on prompts, configurations, or source doc... - [Parameter Binding](https://docs.rhesis.ai/sdk/connector/binding.md): The parameter allows you to inject infrastructure dependencies (database connections, configuration, auth context) into your endpoint functions without exposing... - [Examples](https://docs.rhesis.ai/sdk/connector/examples.md): Complete working examples demonstrating various SDK connector patterns. - [File Attachments in SDK Connector Endpoints](https://docs.rhesis.ai/sdk/connector/files.md): Rhesis passes file attachments through SDK connector executions as metadata-rich references instead of embedding raw bytes in the WebSocket payload. Use when yo... - [Input/Output Mapping](https://docs.rhesis.ai/sdk/connector/mapping.md): Map your function's parameters and return values to Rhesis's standardized request/response format. The SDK supports both automatic detection and manual configur... - [Advanced Mapping](https://docs.rhesis.ai/sdk/connector/serializers.md): Learn how to map complex objects like Pydantic models, dataclasses, and custom types as function parameters and return values. The SDK automatically handles ser... - [Endpoints](https://docs.rhesis.ai/sdk/entities/endpoints.md): Endpoints represent AI services or APIs that tests execute against. They define how Rhesis connects to your application, sends test inputs, and receives respons... - [Experiments](https://docs.rhesis.ai/sdk/entities/experiments.md): An experiment is a named bundle of parameter values for a project. Each save creates an immutable sequential version such as , , or . - [Files](https://docs.rhesis.ai/sdk/entities/files.md): Files are binary attachments (images, PDFs, audio) that can be associated with Tests and TestResults. They are managed through a dedicated upload API rather tha... - [Model Entity](https://docs.rhesis.ai/sdk/entities/models.md): The Model entity represents an LLM configuration stored in the Rhesis platform. Models contain the provider, model name, and API key needed to connect to LLM se... - [Projects](https://docs.rhesis.ai/sdk/entities/projects.md): Projects represent the top-level organizational unit in Rhesis. They contain test sets, endpoints, and other resources for testing and evaluating AI application... - [Status](https://docs.rhesis.ai/sdk/entities/status.md): Statuses track the state of entities throughout the platform. Each status is scoped to an **entity type**, meaning different entities have different applicable ... - [Test Attributes](https://docs.rhesis.ai/sdk/entities/test-attributes.md): Test attribute entities provide structure and classification for tests. Categories, topics, and behaviors help organize test sets and enable filtered evaluation... - [Test Runs & Results](https://docs.rhesis.ai/sdk/entities/test-runs.md): Test runs track the execution of tests against endpoints. Each run produces test results containing the endpoint's response, evaluation metrics, and review stat... - [Test Sets & Tests](https://docs.rhesis.ai/sdk/entities/test-sets.md): Test sets are collections of individual test entities used to evaluate AI applications. Each test entity contains a prompt, expected behaviors, and metadata for... - [Conversational Metrics](https://docs.rhesis.ai/sdk/metrics/conversational.md): Conversational metrics (multi-turn metrics) evaluate the quality of interactions across multiple conversation turns. These metrics assess aspects like coherence... - [Single-Turn Metrics](https://docs.rhesis.ai/sdk/metrics/single-turn.md): Single-turn metrics evaluate individual exchanges between user input and system output. These metrics are ideal for assessing the quality of standalone response... ## Contribute - [Contribute](https://docs.rhesis.ai/contribute.md): We welcome contributions of all kinds: code, ideas, documentation, and bug reports. This section is technical documentation for developers working on and with R... - [Backend Documentation](https://docs.rhesis.ai/contribute/backend.md): This section is about the Rhesis backend: a **FastAPI** application that exposes REST APIs, handles authentication and organizations, and coordinates work with ... - [Coding Standards](https://docs.rhesis.ai/contribute/coding-standards.md): Coding standards for the Rhesis project: quality, consistency, and easier reviews. - [Connector](https://docs.rhesis.ai/contribute/connector.md): The connector enables bidirectional communication between your application (via the Rhesis SDK), the Rhesis backend, and worker processes. It allows remote func... - [Development setup](https://docs.rhesis.ai/contribute/development-setup.md): Two approaches: **local mixed stack** () for day-to-day work, or **all-in-one Docker** () for a full environment without local service wiring. - [Environment Variables](https://docs.rhesis.ai/contribute/environment-variables.md): Environment variables for the backend, frontend, and Celery workers. Local development is intentionally small: run once from the repository root, then to start ... - [Frontend Documentation](https://docs.rhesis.ai/contribute/frontend.md): Welcome to the Rhesis Frontend documentation. This section covers the React-based frontend application that provides the user interface for the Rhesis platform. - [Managing Documentation](https://docs.rhesis.ai/contribute/managing-docs.md): How to run, extend, and maintain the Rhesis docs site (Nextra + Next.js). Source lives under ; MDX content is in . - [Polyphemus (Development)](https://docs.rhesis.ai/contribute/polyphemus.md): Polyphemus is the model-serving service used for adversarial generation workloads. It proxies generation requests to Vertex AI and exposes authenticated REST en... - [Telemetry System](https://docs.rhesis.ai/contribute/telemetry.md): This page provides technical details about Rhesis's telemetry architecture for developers and contributors. - [Tracing System](https://docs.rhesis.ai/contribute/tracing-system.md): Technical documentation for the Rhesis tracing system architecture and implementation. - [API Clients (Token Exchange)](https://docs.rhesis.ai/contribute/backend/api-clients.md): The **API Clients** feature lets a tenant integrate machine-to-machine workloads with Rhesis without minting long-lived Rhesis API tokens by hand. The integrati... - [API Structure](https://docs.rhesis.ai/contribute/backend/api-structure.md): FastAPI routers under , one resource family per module where possible. Paths use normal REST patterns unless noted below. - [Architect Chat System](https://docs.rhesis.ai/contribute/backend/architect-chat-system.md): The backend Architect system combines REST session APIs, WebSocket event streaming, Celery orchestration, and in-process MCP tool dispatch to support conversati... - [Backend Architecture](https://docs.rhesis.ai/contribute/backend/architecture.md): FastAPI app with routers, services, SQLAlchemy models, and OpenAPI docs out of the box. - [Authentication](https://docs.rhesis.ai/contribute/backend/authentication.md): The Rhesis backend implements a native authentication system with a pluggable provider architecture. The system supports email/password authentication, OAuth pr... - [Authorization (RBAC)](https://docs.rhesis.ai/contribute/backend/authorization.md): This page documents the backend's role-based access control (RBAC) system: how every allow/deny decision is made, where it is enforced, and how to extend it. Fo... - [Background Tasks](https://docs.rhesis.ai/contribute/backend/background-tasks.md): The Rhesis backend uses Celery to handle asynchronous background tasks. This allows the API to offload time-consuming operations and improve responsiveness. The... - [Cascade Deletion and Restoration](https://docs.rhesis.ai/contribute/backend/cascade-operations.md): Configured parent/child relationships drive bulk soft-delete and restore so / stay thin: you declare edges once in instead of repeating queries in every CRUD he... - [Database Field Encryption for Sensitive Tokens](https://docs.rhesis.ai/contribute/backend/database-field-encryption.md): The Rhesis backend currently stores sensitive credentials as plaintext in the PostgreSQL database, including: - [Database Models](https://docs.rhesis.ai/contribute/backend/database-models.md): The Rhesis backend uses SQLAlchemy ORM to interact with the PostgreSQL database. All models are defined in the directory and inherit from a common class. - [Deployment](https://docs.rhesis.ai/contribute/backend/deployment.md): This document outlines the process for deploying the Rhesis backend to various environments. The application is containerized using Docker, making it deployable... - [Development Workflow](https://docs.rhesis.ai/contribute/backend/development-workflow.md): Recommended day-to-day workflow for the backend. - [Email Notification System](https://docs.rhesis.ai/contribute/backend/email-notifications.md): The worker can email users when selected Celery tasks finish. SMTP is pluggable (SendGrid and others). Sending is non-blocking and respects tenant context. - [Encryption Troubleshooting Guide](https://docs.rhesis.ai/contribute/backend/encryption-troubleshooting.md): Check Encryption Status - [Environment Configuration](https://docs.rhesis.ai/contribute/backend/environment-config.md): Configuration is environment variables (see [12-factor app](https://12factor.net/)); different files and hosts supply values per environment. - [Getting Started with Rhesis Backend](https://docs.rhesis.ai/contribute/backend/getting-started.md): Set up and run the backend locally for development. - [Multi-tenancy](https://docs.rhesis.ai/contribute/backend/multi-tenancy.md): Rhesis isolates data by organization and active project. Backend code should rely on the request-scoped database session instead of threading tenant identifiers... - [OData Query Guide](https://docs.rhesis.ai/contribute/backend/odata-guide.md): How to use OData-style query parameters such as , , , and with the Rhesis backend API. - [Rhesis Backend Documentation](https://docs.rhesis.ai/contribute/backend/README.md): This documentation provides comprehensive information about the Rhesis backend application, which is built with FastAPI and PostgreSQL. It covers architecture, ... - [Security Features](https://docs.rhesis.ai/contribute/backend/security.md): The Rhesis backend implements multiple layers of security to protect sensitive data and ensure secure access to the API. This document outlines the key security... - [Security Improvements: Organization Filtering](https://docs.rhesis.ai/contribute/backend/security-improvements.md): How we tightened org-scoped queries and added regression coverage so one tenant cannot read or mutate another’s rows through ordinary code paths. - [Soft Deletion Architecture](https://docs.rhesis.ai/contribute/backend/soft-deletion.md): Soft-deleted rows stay in the database ( set) but are hidden from normal queries. That preserves history and FK integrity while still supporting restore and adm... - [Test Result Statistics API](https://docs.rhesis.ai/contribute/backend/test-result-stats.md): aggregates for dashboards: multiple values, rich filters, and compact payloads. - [Test Result Status](https://docs.rhesis.ai/contribute/backend/test-result-status.md): This guide explains how individual test result statuses are determined and what they represent in the Rhesis backend. - [Human Reviews for Test Results](https://docs.rhesis.ai/contribute/backend/test-reviews.md): Human **reviews** on each test result sit beside automated : multiple reviewers, metric- or whole-test scope, and CRUD via the API. Nested holds rollups for qui... - [Test Run Status](https://docs.rhesis.ai/contribute/backend/test-run-status.md): This guide explains how test run statuses are determined and what they represent in the Rhesis backend. - [User Settings](https://docs.rhesis.ai/contribute/backend/user-settings.md): The User model stores preferences in a JSONB column, including LLM defaults, UI preferences, notifications, and localization settings. - [API Integration](https://docs.rhesis.ai/contribute/frontend/api-integration.md): This document explains how the Rhesis frontend application integrates with the Rhesis backend API. - [Architect Chat UI](https://docs.rhesis.ai/contribute/frontend/architect-chat-ui.md): This page documents the frontend architecture for the Architect chat experience in . - [Architecture Overview](https://docs.rhesis.ai/contribute/frontend/architecture.md): This document provides a high-level overview of the Rhesis frontend architecture. - [Authentication](https://docs.rhesis.ai/contribute/frontend/authentication.md): This document explains the authentication system used in the Rhesis frontend application. - [Component Library](https://docs.rhesis.ai/contribute/frontend/components.md): This document provides an overview of the reusable components available in the Rhesis frontend application. - [Getting Started](https://docs.rhesis.ai/contribute/frontend/getting-started.md): This guide will help you set up and run the Rhesis frontend application locally for development. - [Rhesis Frontend Documentation](https://docs.rhesis.ai/contribute/frontend/README.md): Welcome to the Rhesis Frontend documentation. This documentation provides comprehensive guides and references for developers working on the Rhesis frontend appl... - [Routing](https://docs.rhesis.ai/contribute/frontend/routing.md): This document explains the routing system used in the Rhesis frontend application, which is built with Next.js App Router. - [State Management](https://docs.rhesis.ai/contribute/frontend/state-management.md): This document explains the state management approach used in the Rhesis frontend application. - [Frontend Testing](https://docs.rhesis.ai/contribute/frontend/testing.md): This guide covers testing strategies and implementation for the Rhesis frontend application. - [Architect Agent (SDK)](https://docs.rhesis.ai/contribute/sdk/architect-agent.md): The class lives in . It drives the conversational test-suite design workflow — managing mode transitions, plan state, tool confirmation, and event emission acro... - [Framework telemetry integrations (SDK)](https://docs.rhesis.ai/contribute/sdk/integrations.md): The Rhesis Python SDK wires **automatic** tracing for some AI frameworks through small integration classes under . Each integration subclasses and is registered... - [Architecture](https://docs.rhesis.ai/contribute/tracing-system/architecture.md): Detailed architecture of the Rhesis tracing system. - [Data Structures](https://docs.rhesis.ai/contribute/tracing-system/data-structures.md): Schemas, database design, and data formats for the tracing system. - [Trace Lifecycle](https://docs.rhesis.ai/contribute/tracing-system/lifecycle.md): This page covers the complete trace lifecycle, including the race condition problem and the hybrid linking solution. - [Architect Background Tasks](https://docs.rhesis.ai/contribute/worker/architect-background-tasks.md): This page documents the worker-side runtime for Architect chat execution and async resume behavior. - [Worker Architecture and Dependencies](https://docs.rhesis.ai/contribute/worker/architecture.md): This document explains the relationships and dependencies between the Rhesis worker system, the backend API, and the SDK components. - [Worker Tasks and Background Processing](https://docs.rhesis.ai/contribute/worker/background-tasks.md): The Rhesis backend uses Celery to handle asynchronous background tasks. This allows the API to offload time-consuming operations and improve responsiveness. The... - [Execution Modes](https://docs.rhesis.ai/contribute/worker/execution-modes.md): When executing a test configuration (a set of tests against an endpoint), Rhesis supports two execution modes: **Sequential** and **Parallel**. The mode determi... - [GKE Worker Troubleshooting Guide](https://docs.rhesis.ai/contribute/worker/gke-troubleshooting.md): This guide covers troubleshooting Celery workers running in Google Kubernetes Engine (GKE), including using the built-in debugging tools. - [Worker Logging Guide](https://docs.rhesis.ai/contribute/worker/logging.md): This guide covers everything about logging in the Rhesis worker system, from configuration to analysis and troubleshooting. - [Multi-Worker RPC Coordination](https://docs.rhesis.ai/contribute/worker/multi-worker-rpc.md): When running multiple backend workers (e.g., ), RPC requests need to be routed to the specific worker that holds the WebSocket connection to the SDK. This is ac... - [Rhesis Worker Documentation](https://docs.rhesis.ai/contribute/worker/README.md): This directory contains documentation related to the Rhesis worker system, which handles background processing, task queues, and asynchronous operations using *... - [Test Execution System](https://docs.rhesis.ai/contribute/worker/test-execution.md): The Rhesis test execution system provides a flexible and extensible architecture for running tests against endpoints. It supports multiple test types and execut... - [Test Types](https://docs.rhesis.ai/contribute/worker/test-types.md): Rhesis supports multiple test types, each designed for different testing scenarios. The test type determines how the test is executed and evaluated. - [Trace Ingestion Pipeline](https://docs.rhesis.ai/contribute/worker/trace-ingestion-pipeline.md): This document explains how traces are processed after arriving via , covering the full pipeline from storage through enrichment and metric evaluation. - [Worker Troubleshooting Guide](https://docs.rhesis.ai/contribute/worker/troubleshooting.md): This document covers common issues you may encounter with the Rhesis worker system and how to resolve them. ## Optional ### Glossary - [Glossary](https://docs.rhesis.ai/glossary.md): Loading glossary...}> - [A/B Testing](https://docs.rhesis.ai/glossary/a-b-testing.md): Comparing two versions of a system by running identical tests against both to determine which performs better. - [Agent](https://docs.rhesis.ai/glossary/agent.md): title: 'Agent - Glossary', description: 'An autonomous AI system that reasons, plans, and takes sequences of actions—such as calling tools or delegating to sub-... - [Ambiguity Handling](https://docs.rhesis.ai/glossary/ambiguity-handling.md): How an AI system manages unclear, vague, or multi-interpretable user inputs that could have multiple valid meanings. - [API Token](https://docs.rhesis.ai/glossary/api-token.md): title: 'API Token - Glossary', description: 'Authentication credentials used to integrate Rhesis with your systems programmatically via the SDK or API.', } - [Auto-Configure](https://docs.rhesis.ai/glossary/auto-configure.md): title: 'Auto-Configure - Glossary', description: 'An AI-powered feature that generates endpoint request and response mappings by analyzing curl commands, API do... - [Baseline](https://docs.rhesis.ai/glossary/baseline.md): A reference point established from initial test results that serves as a benchmark for comparing future performance and detecting regressions. - [Behavior](https://docs.rhesis.ai/glossary/behavior.md): title: 'Behavior - Glossary', description: 'A formalized expectation that describes how your AI system should perform, such as response quality, safety, or accu... - [Categorical Scoring](https://docs.rhesis.ai/glossary/categorical-scoring.md): title: 'Categorical Scoring - Glossary', description: 'A metric scoring type that classifies responses into predefined categories such as excellent, good, fair,... - [Category](https://docs.rhesis.ai/glossary/category.md): title: 'Category - Glossary', description: 'A high-level classification for tests, such as Harmful or Harmless, used to organize and filter test cases.', } - [Chain-of-Thought](https://docs.rhesis.ai/glossary/chain-of-thought.md): An LLM prompting technique where the model shows its reasoning process step-by-step before providing a final answer. - [Confidence Score](https://docs.rhesis.ai/glossary/confidence-score.md): A numeric indicator of the AI's certainty or reliability in its response, used in evaluation metrics and threshold-based decisions. - [Confusion Matrix](https://docs.rhesis.ai/glossary/confusion-matrix.md): A table showing the performance of a classification system, displaying true positives, false positives, true negatives, and false negatives. - [Connector](https://docs.rhesis.ai/glossary/connector.md): title: 'Connector - Glossary', description: 'The SDK integration layer that connects your LLM application to Rhesis via the @endpoint decorator, enabling automa... - [Containment Rate](https://docs.rhesis.ai/glossary/containment-rate.md): The percentage of user interactions successfully handled by the AI system without requiring escalation, human intervention, or failure. - [Context Switching](https://docs.rhesis.ai/glossary/context-switching.md): Changing topics or focus within an ongoing conversation, testing the AI's ability to handle abrupt subject transitions. - [Context Window](https://docs.rhesis.ai/glossary/context-window.md): The maximum amount of text (measured in tokens) that an LLM can process at once, including both input and output. - [Edge Case](https://docs.rhesis.ai/glossary/edge-case.md): Unusual, boundary, or extreme scenarios that test the limits of an AI system's capabilities and robustness. - [Endpoint](https://docs.rhesis.ai/glossary/endpoint.md): title: 'Endpoint - Glossary', description: 'A complete configuration for calling an external API that represents the AI services or APIs you want to test.', } - [Evaluation Prompt](https://docs.rhesis.ai/glossary/evaluation-prompt.md): title: 'Evaluation Prompt - Glossary', description: 'Instructions provided to the judge model specifying what to evaluate and the criteria to use when assessing... - [Evaluation Steps](https://docs.rhesis.ai/glossary/evaluation-steps.md): title: 'Evaluation Steps - Glossary', description: 'A breakdown of the evaluation process into clear steps that guide the LLM judge when producing a score and r... - [Explorer](https://docs.rhesis.ai/glossary/explorer.md): Topic-based sessions that organize tests by topic and let suites evolve through outputs, evaluation, and AI suggestions. - [F1 Score](https://docs.rhesis.ai/glossary/f1-score.md): The harmonic mean of precision and recall, providing a single balanced metric that considers both false positives and false negatives. - [Fallback Behavior](https://docs.rhesis.ai/glossary/fallback-behavior.md): How an AI system responds when it cannot understand, answer, or fulfill a user's request, providing a default or alternative response. - [False Positive](https://docs.rhesis.ai/glossary/false-positive.md): When a test incorrectly passes or a metric incorrectly identifies something as acceptable when it should fail. - [Garak](https://docs.rhesis.ai/glossary/garak.md): title: 'Garak - Glossary', description: 'An open-source LLM vulnerability scanner integrated into Rhesis that tests AI applications for prompt injection, jailbr... - [Goal Achievement](https://docs.rhesis.ai/glossary/goal-achievement.md): title: 'Goal Achievement - Glossary', description: 'A metric used by Penelope to evaluate whether a multi-turn conversation accomplished its stated goal, produc... - [Graceful Degradation](https://docs.rhesis.ai/glossary/graceful-degradation.md): The ability of an AI system to maintain partial functionality and provide useful responses when facing errors, limitations, or unexpected conditions. - [Ground Truth](https://docs.rhesis.ai/glossary/ground-truth.md): Known correct answers, expected outputs, or validated reference data used to evaluate AI system accuracy and performance. - [Hallucination](https://docs.rhesis.ai/glossary/hallucination.md): When an LLM generates false, fabricated, or nonsensical information presented as fact, often with high confidence. - [Intent Understanding](https://docs.rhesis.ai/glossary/intent-understanding.md): The AI's ability to comprehend what a user wants to accomplish from their input, beyond just the literal words used. - [Jailbreak](https://docs.rhesis.ai/glossary/jailbreak.md): title: 'Jailbreak - Glossary', description: 'An adversarial technique that attempts to bypass an LLM\'s safety constraints to produce outputs the model was trai... - [Knowledge](https://docs.rhesis.ai/glossary/knowledge.md): title: 'Knowledge - Glossary', description: 'Domain context and source materials used to generate context-aware test scenarios for your AI application.', } - [Latency](https://docs.rhesis.ai/glossary/latency.md): The time delay between receiving user input and producing a response, a critical performance metric for conversational AI systems. - [LLM as a Judge](https://docs.rhesis.ai/glossary/llm-as-a-judge.md): An approach where an LLM evaluates AI responses against defined criteria, providing automated quality assessment. - [MCP](https://docs.rhesis.ai/glossary/mcp.md): title: 'MCP - Glossary', description: 'Model Context Protocol - a standard for connecting to external knowledge sources and importing domain context.', } - [Metric](https://docs.rhesis.ai/glossary/metric.md): title: 'Metric - Glossary', description: 'A quantifiable measurement that evaluates AI behavior using an LLM as a judge, returning pass/fail results with option... - [Metric Scope](https://docs.rhesis.ai/glossary/metric-scope.md): title: 'Metric Scope - Glossary', description: 'The test types (single-turn or multi-turn) that a metric can evaluate, defined during metric configuration.', } - [Model](https://docs.rhesis.ai/glossary/model.md): title: 'Model - Glossary', description: 'An AI model configuration used for test generation, evaluation, or as a judge in metric assessments.', } - [Multi-Turn Test](https://docs.rhesis.ai/glossary/multi-turn-test.md): title: 'Multi-Turn Test - Glossary', description: 'Goal-based conversation tests that evaluate your AI system across multiple turns, powered by Penelope.', } - [Numeric Scoring](https://docs.rhesis.ai/glossary/numeric-scoring.md): title: 'Numeric Scoring - Glossary', description: 'A metric scoring type that uses a numeric scale (e.g., 0-10) with a defined pass/fail threshold.', } - [OpenTelemetry](https://docs.rhesis.ai/glossary/opentelemetry.md): title: 'OpenTelemetry - Glossary', description: 'An open observability standard for collecting and exporting traces, metrics, and logs, used by the Rhesis SDK t... - [Organization](https://docs.rhesis.ai/glossary/organization.md): title: 'Organization - Glossary', description: 'The top-level organizational unit that provides data isolation and manages team access through invitations.', } - [Out-of-Scope Query](https://docs.rhesis.ai/glossary/out-of-scope-query.md): User requests that fall outside the AI system's designed capabilities, knowledge domain, or intended purpose. - [Overfitting](https://docs.rhesis.ai/glossary/overfitting.md): When a system performs well on test data but poorly on new, unseen inputs due to overly specific tuning or memorization. - [Pass/Fail Threshold](https://docs.rhesis.ai/glossary/pass-fail-threshold.md): title: 'Pass/Fail Threshold - Glossary', description: 'The minimum score required for a test to be considered passing, defined in the metric configuration.', } - [Penelope](https://docs.rhesis.ai/glossary/penelope.md): title: 'Penelope - Glossary', description: 'An autonomous testing agent that powers multi-turn tests, adapting its strategy based on AI responses to evaluate co... - [Playground](https://docs.rhesis.ai/glossary/playground.md): title: 'Playground - Glossary', description: 'An interactive, real-time interface for testing conversational endpoints and converting live conversations directl... - [Polyphemus](https://docs.rhesis.ai/glossary/polyphemus.md): title: 'Polyphemus - Glossary', description: 'The Rhesis-hosted LLM service that provides access to open-source models with built-in access control, rate limiti... - [Precision and Recall](https://docs.rhesis.ai/glossary/precision-and-recall.md): Complementary metrics measuring accuracy: precision shows how many positive predictions were correct, while recall shows how many actual positives were found. - [Project](https://docs.rhesis.ai/glossary/project.md): title: 'Project - Glossary', description: 'The top-level organizational unit that groups related endpoints, tests, test sets, and results together for a specifi... - [Prompt Injection](https://docs.rhesis.ai/glossary/prompt-injection.md): Malicious inputs that attempt to override system instructions, extract sensitive information, or manipulate AI behavior in unintended ways. - [Reasoning Instructions](https://docs.rhesis.ai/glossary/reasoning-instructions.md): title: 'Reasoning Instructions - Glossary', description: 'Guidance provided to the judge model explaining how to reason about the evaluation and weight differen... - [Regression Testing](https://docs.rhesis.ai/glossary/regression-testing.md): Testing to ensure that new changes, updates, or deployments don't break existing functionality or degrade AI performance. - [Rescoring](https://docs.rhesis.ai/glossary/rescoring.md): title: 'Rescoring - Glossary', description: 'Re-evaluating stored test outputs against metrics without re-invoking the endpoint, enabling cost-efficient experim... - [Retrieval-Augmented Generation (RAG)](https://docs.rhesis.ai/glossary/retrieval-augmented-generation.md): An LLM approach that retrieves relevant information from a knowledge base before generating responses, grounding outputs in specific source documents. - [Score Configuration](https://docs.rhesis.ai/glossary/score-configuration.md): title: 'Score Configuration - Glossary', description: 'Settings that define how metrics score responses, including numeric scales or categorical classifications... - [SDK](https://docs.rhesis.ai/glossary/sdk.md): title: 'SDK - Glossary', description: 'Software Development Kit - A Python library that provides programmatic access to Rhesis platform features for integration... - [Single-Turn Test](https://docs.rhesis.ai/glossary/single-turn-test.md): title: 'Single-Turn Test - Glossary', description: 'A test type that checks how the AI responds to a single prompt with no follow-up conversation.', } - [Smoke Testing](https://docs.rhesis.ai/glossary/smoke-testing.md): Quick, high-level validation tests that check basic functionality and critical features to determine if a system is stable enough for more detailed testing. - [Source](https://docs.rhesis.ai/glossary/source.md): title: 'Source - Glossary', description: 'A document or knowledge artifact uploaded to a Rhesis project that provides context for test generation or grounding A... - [Synthesizer](https://docs.rhesis.ai/glossary/synthesizer.md): title: 'Synthesizer - Glossary', description: 'An SDK component that automatically generates test cases from prompts, documents, or existing tests using LLM-pow... - [Task](https://docs.rhesis.ai/glossary/task.md): title: 'Task - Glossary', description: 'A work item used to track testing activities, issues, or improvements within the platform.', } - [Temperature](https://docs.rhesis.ai/glossary/temperature.md): A model parameter controlling randomness and creativity in LLM outputs, with lower values producing more deterministic responses and higher values producing mor... - [Test](https://docs.rhesis.ai/glossary/test.md): title: 'Test - Glossary', description: 'An individual test case that represents a prompt or input sent to your AI application, including metadata about behavior... - [Test Generation](https://docs.rhesis.ai/glossary/test-generation.md): title: 'Test Generation - Glossary', description: 'The process of automatically creating test cases using AI, based on prompts, configurations, and source mater... - [Test Result](https://docs.rhesis.ai/glossary/test-result.md): title: 'Test Result - Glossary', description: 'Aggregate analytics from multiple test runs that reveal trends and patterns in AI system quality over time.', } - [Test Run](https://docs.rhesis.ai/glossary/test-run.md): title: 'Test Run - Glossary', description: 'A snapshot capturing the complete result of executing a test set against an endpoint, including individual test resu... - [Test Set](https://docs.rhesis.ai/glossary/test-set.md): title: 'Test Set - Glossary', description: 'A collection of tests that can be executed together against an endpoint, similar to test suites in traditional softw... - [Test Set Type](https://docs.rhesis.ai/glossary/test-set-type.md): title: 'Test Set Type - Glossary', description: 'A required classification for a test set that determines which tests it can contain and how execution is handle... - [Test Type](https://docs.rhesis.ai/glossary/test-type.md): title: 'Test Type - Glossary', description: 'The classification of an individual test as either single-turn or multi-turn, determining how it is executed and wh... - [Topic](https://docs.rhesis.ai/glossary/topic.md): title: 'Topic - Glossary', description: 'A specific subject matter classification for tests, such as healthcare or financial advice, used for organization and a... - [Trace](https://docs.rhesis.ai/glossary/trace.md): title: 'Trace - Glossary', description: 'A record of an AI application\'s execution path that captures inputs, outputs, intermediate steps, and timing data for ... - [Turn-Taking](https://docs.rhesis.ai/glossary/turn-taking.md): The exchange pattern in conversations where the user and AI alternate speaking, fundamental to multi-turn dialogue evaluation. - [Utterance](https://docs.rhesis.ai/glossary/utterance.md): A single unit of communication in a conversation, either a user's input or the AI's response in a dialogue exchange. ### Changelog - [Rhesis Changelog](https://docs.rhesis.ai/changelog.md): All notable changes to the Rhesis project are documented in this file. ## Resources - Full content (all pages concatenated): https://docs.rhesis.ai/llms-full.txt - SDK API reference: https://rhesis-sdk.readthedocs.io/en/latest/ - GitHub: https://github.com/rhesis-ai/rhesis - Discord: https://discord.rhesis.ai - Website: https://www.rhesis.ai