Skip to Content
DocsGetting StartedOverview

Getting Started

This guide takes you from a fresh environment to your first automated evaluation of your LLM application.

Quick install (Python SDK)

Prefer code first? Install the SDK and list your test sets in a few lines:

Terminal
pip install rhesis-sdk
Python
import os
from rhesis.sdk.entities import TestSets

os.environ["RHESIS_API_KEY"] = "rh-your-api-key"  # from app.rhesis.ai
for ts in TestSets.all():
    print(ts.name)

For full SDK docs, see SDK installation.

What are you building?

Pick a path that matches your goal — each card links to deeper docs.

Testing AI agents: Exercise agents with goals, metrics, and traces.
🤖

Testing AI agents

Exercise agents with goals, metrics, and traces.

User journeys guide →
Testing RAG pipelines: Structure expectations, metrics, and generated tests.
📚

Testing RAG pipelines

Structure expectations, metrics, and generated tests.

Metrics →
Adversarial and safety: Red-team style checks and Polyphemus integration.
🛡️

Adversarial and safety

Red-team style checks and Polyphemus integration.

Adversarial testing →
CI/CD integration: Run evaluations from your pipeline with the SDK.
⚙️

CI/CD integration

Run evaluations from your pipeline with the SDK.

CI/CD guide →
Conversation testing: Multi-turn simulation with Penelope.
💬

Conversation testing

Multi-turn simulation with Penelope.

Conversation simulation →
Test from your IDE: Run the full test workflow from Cursor or Claude Code.
🖥️

Test from your IDE

Run the full test workflow from Cursor or Claude Code.

Agent Skill →

Prefer programmatic access? Use the Python SDK for test sets, runs, and the connector.

Getting started consists of three core steps:

Setup Environment

Choose how you want to run Rhesis: Cloud (fastest), Local, or Self-hosted. Setup Environment Guide →

Connect Application

Connect your LLM application to Rhesis via REST Endpoints or our Python SDK. Connect Application Guide →

Run Evaluations

Generate test sets interactively and run automated evaluations against your application. Run Evaluations Guide →

Next Steps

Core Concepts: Understand the fundamental concepts of testing LLM applications with Rhesis.
📚

Core Concepts

Understand the fundamental concepts of testing LLM applications with Rhesis.

Learn Core Concepts →
Tour: See the screens your team will use, from the Playground to reviews.
🧭

Tour

See the screens your team will use, from the Playground to reviews.

Take the tour →
Deployment: Learn how to run Rhesis locally or self-host it on your infrastructure.
💻

Deployment

Learn how to run Rhesis locally or self-host it on your infrastructure.

View Deployment Options →
Guides: Step-by-step tutorials for common use cases.
📖

Guides

Step-by-step tutorials for common use cases.

View Guides →

Need Help?

  • In-app support drawer: Click Support in the sidebar footer to open links to the documentation site, support email, GitHub, and Discord.
  • GitHub Issues: Report bugs or request features 
WelcomeSetup Environment