How to Integrate Rhesis SDK into Your CI/CD Pipeline

Get Your API Credentials

1. Log in to app.rhesis.ai 
2. Navigate to API Tokens
3. Generate a new API Token
4. Copy the Token (you’ll need it in the next step)

Configure CI/CD Secrets

Store your Rhesis credentials as environment variables in your CI/CD platform:

For GitHub Actions:

1. Go to your repository Settings → Secrets and variables → Actions
2. Click New repository secret
3. Add these secrets:

Optional: Set the base URL if using self-hosted Rhesis:

RHESIS_BASE_URL=https://your-rhesis-instance.com

Create the Workflow File

Create .github/workflows/rhesis-tests.yml in your repository. The workflow includes these key steps:

1. Define triggers - Run tests on pushes and pull requests:

on:
push:
    branches: [main, develop]
pull_request:
    branches: [main, develop]

2. Set up environment - Install Python and Rhesis SDK:

- name: Set up Python
uses: actions/setup-python@v6
with:
    python-version: '3.11'

- name: Install Rhesis SDK
run: pip install rhesis-sdk

3. Run tests - Execute the test runner with environment variables from secrets:

- name: Run Rhesis Tests
env:
    RHESIS_API_KEY: # Reference secrets.RHESIS_API_KEY
    RHESIS_ENDPOINT_NAME: # Reference secrets.RHESIS_ENDPOINT_NAME
    RHESIS_TEST_SET_NAME: # Reference secrets.RHESIS_TEST_SET_NAME
run: python .github/scripts/run_rhesis_tests.py

4. Upload results - Save test results as artifacts:

- name: Upload Test Results
if: always()
uses: actions/upload-artifact@v4
with:
    name: rhesis-test-results
    path: test-results.json

Copy the complete workflow file from examples/ci-cd/rhesis-tests.yml to your .github/workflows/ directory.

Create the Test Runner Script

Create .github/scripts/run_rhesis_tests.py. The script follows these key steps:

Step 1: Get endpoint - Retrieve the endpoint configuration:

endpoint = Endpoints.pull(name=endpoint_name)

Step 2: Get test set - Fetch the test set to execute:

test_set = TestSets.pull(name=test_set_name)

Step 3: Execute test set - Run the test set against the endpoint:

execution_response = test_set.execute(endpoint=endpoint)
test_configuration_id = execution_response.get('test_configuration_id')

Step 4: Poll for test run - Wait for the test run to appear:

test_run_id = poll_for_test_run(test_configuration_id)

Step 5: Wait for completion - Monitor the test run until it finishes:

test_run = TestRun(id=test_run_id)
wait_for_completion(test_run)

Step 6: Get results - Retrieve and analyze test results:

summary = get_test_results(test_run_id)
print_summary(summary)
save_results(summary)

Step 7: Check for failures - Fail the pipeline if any tests failed:

if summary["failed"] > 0:
    print("❌ CI/CD PIPELINE FAILED")
    sys.exit(1)
else:
    print("✅ CI/CD PIPELINE PASSED")
    sys.exit(0)

The script includes helper functions for polling, waiting, and result analysis. Copy the complete implementation from examples/ci-cd/run_rhesis_tests.py to your .github/scripts/ directory.

Test Your Integration

1. Copy the files to your repository:

   • Copy examples/ci-cd/rhesis-tests.yml → .github/workflows/rhesis-tests.yml
   • Copy examples/ci-cd/run_rhesis_tests.py → .github/scripts/run_rhesis_tests.py

2. Commit and push your changes:

git add .github/
git commit -m "Add Rhesis LLM testing to CI/CD"
git push origin your-branch

3. Monitor the workflow in the Actions tab of your GitHub repository.