How to Start Testing LLM and Agentic Apps in 10 Minutes with Rhesis AI

Testing LLM and agentic apps is challenging: outputs are non-deterministic, edge cases are unpredictable, and manual testing doesn’t scale. This guide shows you how to get a complete, automated testing pipeline up and running with Rhesis in under 10 minutes.

What You’ll Get

• Test generation: Generate hundreds of test scenarios from plain-language requirements
• Single-turn and multi-turn testing: Test both simple Q&A responses and complex conversations (via Penelope)
• LLM-based evaluation: Automated scoring of whether outputs meet your requirements
• Full testing platform: UI, API, and SDK for running and managing tests

Prerequisites

• Docker Desktop installed and running
• Git (to clone the repository)
• Ports 3000, 8080, 8081, 5432, and 6379 available on your system
• An AI provider API key (Rhesis API, OpenAI, Azure OpenAI, or Google Gemini)

Step 1: Clone and Start (5 minutes)

# Clone the repository
git clone https://github.com/rhesis-ai/rhesis.git
cd rhesis

# Start all services with one command
./rh start

The ./rh start command automatically:

• Checks if Docker is running
• Generates a secure database encryption key
• Creates .env.docker.local with all required configuration
• Enables local authentication bypass (auto-login)
• Starts all services (backend, frontend, database, worker)
• Creates the database and runs migrations
• Creates the default admin user (Local Admin)
• Loads example test data

Wait approximately 5-7 minutes for all services to start. You’ll see containers starting up in your terminal.

Step 2: Access the Platform (1 minute)

Once services are running:

• Frontend Dashboard: http://localhost:3000  (auto-login enabled)
• Backend API Docs: http://localhost:8080/docs 
• Worker Health: http://localhost:8081/health/basic 

Step 3: Configure AI Provider (1 minute)

Configure an AI provider to enable test generation:

Option 1: Use Rhesis API (Recommended)

1. Get your API key from https://app.rhesis.ai/ 
2. Edit .env.docker.local and add:

RHESIS_API_KEY=your-actual-rhesis-api-key-here

Option 2: Use Your Own AI Provider

Add your provider credentials to .env.docker.local:

# Google Gemini
GEMINI_API_KEY=your-gemini-api-key
GEMINI_MODEL_NAME=gemini-2.0-flash-001
GOOGLE_API_KEY=your-google-api-key

# Or Azure OpenAI
AZURE_OPENAI_ENDPOINT=your-endpoint
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o
AZURE_OPENAI_API_VERSION=your-version

# Or OpenAI
OPENAI_API_KEY=your-openai-key
OPENAI_MODEL_NAME=gpt-4o

After updating, restart services:

./rh restart

Step 4: Start Testing Your LLM/Agentic App (3 minutes)

Via Web UI

1. Open http://localhost:3000 
2. Create an Endpoint: Add your LLM/agentic app’s API endpoint
3. Define Requirements: Specify what your app should and shouldn’t do
4. Generate Tests: Automatically generate hundreds of test scenarios
5. Run Tests: Execute tests against your endpoint
6. Review Results: View which outputs violate requirements

Via Python SDK

pip install rhesis-sdk

export RHESIS_API_KEY="your-api-key"
export RHESIS_BASE_URL="http://localhost:8080"

import rhesis

rhesis.api_key = "your-api-key"
rhesis.base_url = "http://localhost:8080"

import os
from rhesis.sdk.entities import Endpoint

os.environ["RHESIS_API_KEY"] = "rh-your-api-key"  # Get from http://localhost:3000 
os.environ["RHESIS_BASE_URL"] = "http://localhost:8080"

# Retrieve endpoint by ID
endpoint = Endpoint(id="your-endpoint-id")
endpoint.pull()

import os
from rhesis.sdk.entities import TestSet
from rhesis.sdk.synthesizers import PromptSynthesizer

os.environ["RHESIS_API_KEY"] = "rh-your-api-key"  # Get from http://localhost:3000
os.environ["RHESIS_BASE_URL"] = "http://localhost:8080"

# Generate custom test scenarios
synthesizer = PromptSynthesizer(
    prompt="Generate tests for a medical chatbot that must never provide diagnosis",
)
test_set = synthesizer.generate(num_tests=10)

# Push test set to platform
test_set.push()
print(f"Created test set: {test_set.id}")

import os
from rhesis.sdk.entities import Endpoint, TestSet
from rhesis.sdk.entities.test_configuration import TestConfiguration

os.environ["RHESIS_API_KEY"] = "rh-your-api-key"  # Get from http://localhost:3000
os.environ["RHESIS_BASE_URL"] = "http://localhost:8080"

# Retrieve endpoint by ID
endpoint = Endpoint(id="your-endpoint-id")
endpoint.pull()

# Retrieve test set by ID
test_set = TestSet(id="your-test-set-id")
test_set.pull()

# Create test configuration linking test set and endpoint
test_config = TestConfiguration(
    endpoint_id=endpoint.id,
    test_set_id=test_set.id,
)
test_config.push()

# Execute test set against endpoint
result = test_set.execute(endpoint)
print(f"Test run started: {result}")

For complex conversations, use Penelope  to simulate multi-turn interactions .

What’s Running

Your local infrastructure includes:

Architecture Overview

Quick Commands

# Stop all services
./rh stop

# View logs
./rh logs

# Restart services
./rh restart

# Delete everything (fresh start)
./rh delete

Next Steps

• Read the docs: https://docs.rhesis.ai 
• Try examples: Check the examples/ directory in the repository
• Join Discord: Get help and share feedback
• Explore Penelope: Learn about multi-turn testing at https://docs.rhesis.ai/penelope 

Troubleshooting

Docker not running?

./rh start

Port already in use?

lsof -i :3000  # Check what's using the port
kill -9 <PID>  # Kill the process

Services not starting?

./rh logs  # Check logs for errors
./rh delete && ./rh start  # Fresh start

AI provider not working?

Verify your API key in .env.docker.local and restart with ./rh restart.

Need help? Join our Discord: https://discord.rhesis.ai