How to Start Testing LLM and Agentic Apps in 10 Minutes with Rhesis AI

Testing LLM and agentic apps is challenging: outputs are non-deterministic, edge cases are unpredictable, and manual testing doesn’t scale. This guide gets a complete, automated testing pipeline running with Rhesis in under 10 minutes — self-hosted via Docker, driven from the UI or the Python SDK.

Prerequisites

Docker running, and ports 3000, 8080, 8081, 5432, and 6379 free
A model to connect: a Rhesis API key from app.rhesis.ai/tokens , a provider account (OpenAI, Anthropic, Gemini, and others), or a local model

Step 1: Clone and Start (5 minutes)

Terminal
git clone https://github.com/rhesis-ai/rhesis.git
cd rhesis

./rh start                # pulls prebuilt images from GHCR
# ./rh start --build      # build images locally instead

./rh start generates local configuration, starts all services, runs migrations, seeds an admin user with auto-login, and loads example data. First run takes a few minutes while images download.

Step 2: Access the Platform (1 minute)

Once services are running:

Frontend Dashboard: http://localhost:3000 (auto-login enabled)
Backend API Docs: http://localhost:8080/docs
Worker Health: http://localhost:8081/health/basic

Step 3: Connect a Model (1 minute)

The first time you open http://localhost:3000 , the app shows a Connect a model step. Choose Use a Rhesis API key to run on the models hosted by Rhesis, or Add your own model to connect a provider account or a local model. The step closes as soon as the model works, with no restart.

See Connect a Model for both options. To set models through environment variables instead, see AI models.

Step 4: Start Testing Your LLM/Agentic App (3 minutes)

Via Web UI

Open http://localhost:3000
Create an Endpoint: Add your LLM/agentic app’s API endpoint
Define Requirements: Specify what your app should and shouldn’t do
Generate Tests: Automatically generate hundreds of test scenarios
Run Tests: Execute tests against your endpoint
Review Results: View which outputs violate requirements

Via Python SDK

Installation

terminal
pip install rhesis-sdk

Configure the SDK

The SDK reads two environment variables. RHESIS_API_KEY is a token for the deployment you call, and RHESIS_BASE_URL is that deployment’s backend.

Deployment	`RHESIS_API_KEY`	`RHESIS_BASE_URL`
Local (`./rh start`)	`rh-local-token`, or a token from API Tokens at http://localhost:3000/tokens	`http://localhost:8080`
Rhesis Cloud	A token from app.rhesis.ai/tokens	Leave unset. The SDK then calls Rhesis Cloud (`https://api.rhesis.ai`), not your local instance.

./rh start creates rh-local-token for the local admin user. A token only works on the deployment that issued it, so rh-local-token fails against Rhesis Cloud and a cloud token fails against localhost.

For this guide, point the SDK at your local instance:

terminal
export RHESIS_API_KEY="rh-local-token"
export RHESIS_BASE_URL="http://localhost:8080"

The Rhesis API key you may have pasted in Connect a model is a different thing. It lets your local backend call the models hosted by Rhesis. In this local setup the SDK doesn’t need it, because SDK calls go to your local backend, which uses the model you connected. Against Rhesis Cloud, the SDK uses a Cloud token as RHESIS_API_KEY.

Set up your REST endpoint

Set up your REST endpoint in the UI, or use the Rhesis connector. Then retrieve the endpoint by ID:

endpoint.py
from rhesis.sdk.entities import Endpoint

# Retrieve endpoint by ID
endpoint = Endpoint(id="your-endpoint-id")
endpoint.pull()

Create a test set

Using the SDK, synthesizer, and then the push method:

create_test_set.py
from rhesis.sdk.entities import TestSet
from rhesis.sdk.synthesizers import PromptSynthesizer

# Generate custom test scenarios
synthesizer = PromptSynthesizer(
    prompt="Generate tests for a medical chatbot that must never provide diagnosis",
)
test_set = synthesizer.generate(num_tests=10)

# Push test set to platform
test_set.push()
print(f"Created test set: {test_set.id}")

Run tests

Retrieve endpoint by ID, start a test run on the test set and endpoint:

run_tests.py
from rhesis.sdk.entities import Endpoint, TestSet
from rhesis.sdk.entities.test_configuration import TestConfiguration

# Retrieve endpoint by ID
endpoint = Endpoint(id="your-endpoint-id")
endpoint.pull()

# Retrieve test set by ID
test_set = TestSet(id="your-test-set-id")
test_set.pull()

# Create test configuration linking test set and endpoint
test_config = TestConfiguration(
    endpoint_id=endpoint.id,
    test_set_id=test_set.id,
)
test_config.push()

# Execute test set against endpoint
result = test_set.execute(endpoint)
print(f"Test run started: {result}")

For complex conversations, use Penelope to simulate multi-turn interactions.

What’s Running

Your local infrastructure includes:

Service	Port	Description
Backend API	`8080`	FastAPI application handling test execution and evaluation
Frontend	`3000`	Next.js dashboard for managing tests and reviewing results
Worker	`8081`	Celery worker processing test runs and AI evaluations
PostgreSQL	`5432`	Database storing tests, results, and configurations
Redis	`6379`	Message broker for worker tasks

Architecture Overview

Quick Commands

Terminal
# Stop all services
./rh stop

# View logs
./rh logs

# Restart services
./rh restart

# Restart and rebuild images from local Dockerfiles
# ./rh restart --build

# Delete everything (fresh start)
./rh delete

Next Steps

Simulate multi-turn conversations with Penelope.
Wire test runs into your pipeline with CI/CD Integration.
Write evaluation criteria for your use case with Custom Metrics.

Troubleshooting

Port already in use: lsof -i :3000 to find the process, then kill -9 <PID>.
Services not starting: check ./rh logs, or ./rh delete && ./rh start for a clean slate.
Model not working: open Models and use Test Connection on the model. A key pasted in Connect a model is stored in the app, so change it on the Models page; no restart is needed. Only a key set as RHESIS_API_KEY in .env.docker.local needs ./rh restart after you change it.

For self-hosting details beyond this quickstart, see the Docker Compose deployment guide.