Models

Models are the LLM providers Rhesis uses for three purposes:

• Test generation: create test cases from your requirements (see Tests)
• Evaluation (LLM-as-judge): score responses with LLM-powered Metrics
• Execution (multi-turn): drive conversation simulations with Penelope against your endpoints

Configure a default model per purpose in the model settings, or override it per run (see Test Execution).

Default Model

Rhesis Default is a Rhesis-hosted model, pre-configured for generation, evaluation, and execution. On Rhesis Cloud it works with no setup.

A self-hosted deployment needs a Rhesis API key to use it. Until the deployment has that key or another working model, the app opens on a Connect a model step, where you choose Use a Rhesis API key or Add your own model. See Connect a Model.

Members who cannot add models see “Ask an organization admin to connect a model, then check again.” Once an admin connects one, the member clicks Check again.

Supported Providers

Rhesis connects to major cloud providers and to local or self-hosted runtimes (Ollama, vLLM, LiteLLM proxy). A selection:

Connecting a Model Provider

Click Add Model to open the model connection drawer. The drawer has two steps:

Select a provider

Choose whether you are adding a language model or an embedder, then select the provider from the provider list. Providers display icons and badges so you can quickly distinguish hosted, cloud, and restricted-access options.

Connect to the provider

Fill in the connection details and defaults:

Field	Purpose
Connection Name	Unique display name for this connection
Model Name	Provider-specific model or deployment name, such as gemini-1.5-pro or gpt-4-turbo
API Endpoint	Endpoint URL, for providers that need one (see the table below)
API Key	Provider API key, when required
Default for Test Generation	Use this model when generating tests
Default for Evaluation (LLM as Judge)	Use this model for LLM-as-judge metrics
Default for Execution (Multi-Turn)	Use this model for conversation simulation with Penelope

Click Test Connection before saving.

Provider-Specific Connection Fields

Some providers require endpoint configuration in addition to model name:

Decision Models

A decision model answers typed questions about its input instead of generating text. Jev by TypeSafe AI is one: given the evaluated input and output and a metric’s criteria, it picks one of the metric’s categories and returns a probability for each.

• It can only judge categorical metrics. Numeric, binary and multi-turn metrics refuse it, and a test run shows those metrics as errored.
• It can be the default for evaluation, but not for test generation or execution.
• It returns no written reasoning. A result’s reason gives the chosen category and its probability.

Polyphemus Access Control

Polyphemus uses an explicit request-and-review workflow. If access is not granted yet, the model card shows Access Required and a Request Access action.

Request Workflow

1. Open Models and click Request Access on the Polyphemus card.
2. Submit request details in the access modal.
3. The request is recorded and review notification is sent.
4. After approval, your account is marked as verified and the model becomes available.

Request Fields

Next steps

• Build Metrics that use your connected models as the judge.
• Generate Tests with a model set as the generation default.