Tour

Rhesis is where the people building an AI agent and the people who know what it should answer work on the same test cases. This page shows the screens they meet along the way, and links to the page that covers each one in depth.

For what each object is called and how they relate, see Core Concepts. To run this yourself, start with Getting Started.

Connect the agent you are building

Everything else needs a live agent to talk to. An endpoint is that connection. Register a public REST URL, or use the SDK connector so the agent keeps running on your laptop or inside your VPC and opens an outbound WebSocket instead.

See Connect your application for both routes.

Let the people who know the domain talk to it

Once an endpoint is connected, anyone you invite can chat with the agent in the Playground. No local setup, no code, no waiting for an engineer to reproduce something. They can attach files, open the trace behind any reply to see which tools and retrieval steps ran, and put two conversations side by side to compare configurations.

This is where domain knowledge first enters Rhesis: as a real conversation with the real agent, rather than a description of one in a Slack thread.

Invite them from Organizations & Team.

Turn those conversations into tests

Any Playground conversation can be saved as a test from the conversation window. The test keeps the full message context, including attached files, so it replays exactly later. Tests carry a behavior, topic, and category, and group into test sets for batch runs.

Conversations are one source. You can also generate tests from your requirements, or import an existing set.

Write down what good looks like

A saved conversation says what happened. Behaviors and metrics say what should happen: a behavior is the expectation, and the metrics attached to it are the judges that score responses against it. Results roll up by behavior, so you can see which expectation is slipping rather than only which test failed.

Disagree with the machine

Automated scores are a signal, not the last word. A response can be technically correct and wrong for your brand, and a refusal that reads as a failure is sometimes the right call. A review is a human verdict that overrides the automated one, at the target that matters: the whole result, one metric, or a single conversation turn. The original score stays on record.

Every verdict written anywhere lands in the Annotations hub, filtered by open or resolved, so disagreements stay visible until the prompt, the model, or the metric is fixed. Anything needing follow-up becomes a task.

Run the same set on every change

Executing a test set against an endpoint creates a test run. Each run keeps its results, metric scores, conversation history, and review activity, so “is it getting better” becomes a comparison between runs instead of an opinion.

Three ways in

The same workspace, reached in whichever way suits the person using it.

Reading the feedback back from code

Reviews are attached to the test results they judge, so a script can pick up what the team decided:

from rhesis.sdk.entities import TestRuns

run = TestRuns.all()[0]

for result in run.get_test_results():
    for review in (result.get("test_reviews") or {}).get("reviews", []):
        target = review["target"].get("reference") or review["target"]["type"]
        print(target, review["status"]["name"], review["user"]["name"])
        print(review["comments"])

Next steps

• Connect the agent you are building, then open it in the Playground
• Invite the people who know the domain and share the endpoint with them
• Generate a first test set from your requirements and run it