Execution Modes

Overview

When executing a test configuration (a set of tests against an endpoint), Rhesis supports two execution modes: Sequential and Parallel. The mode determines how multiple tests are scheduled and executed.

Parallel Execution (Default)

Description

A single Celery task runs the tests concurrently through an async batch engine. This is the default mode and the fastest.

How It Works

Test Configuration
  ↓
Single Celery Task (execute_test_configuration)
  ↓
Async Batch Engine (semaphore-gated)
  ├─→ coroutine: Test A
  ├─→ coroutine: Test B
  └─→ coroutine: Test C
  ↓
Collect Results

• Tests are executed asynchronously within a single worker task
• Concurrency is managed internally using an asyncio batch runner and semaphore limits
• Results are collected after the entire batch completes

Use Cases

Best for: independent tests against endpoints that handle concurrent requests without rate limits, and large test suites where speed matters.

Considerations: may overwhelm an endpoint or hit rate limits, and concurrent failures are harder to debug.

Configuration

Parallel is the default mode. No configuration needed:

# Implicitly uses parallel mode
test_config = TestConfiguration(
    endpoint_id=endpoint.id,
    test_set_id=test_set.id,
    # attributes is empty or doesn't specify execution_mode
)

Or explicitly set:

test_config = TestConfiguration(
    endpoint_id=endpoint.id,
    test_set_id=test_set.id,
    attributes={
        "execution_mode": "Parallel"
    }
)

Sequential Execution

Description

Tests are executed one after another in sequence. Each test must complete before the next one starts.

How It Works

Test Configuration
  ↓
Test 1
  ↓
Test 2
  ↓
Test 3
  ↓
Collect Results

• Tests execute in order
• One test at a time
• No concurrent load on endpoint
• Predictable execution pattern

Use Cases

Best for: rate-limited or stateful endpoints, tests with dependencies on each other, and debugging (one test at a time is easier to trace).

Considerations: slower overall, with longer waits for results.

Configuration

Set in test configuration attributes:

test_config = TestConfiguration(
    endpoint_id=endpoint.id,
    test_set_id=test_set.id,
    attributes={
        "execution_mode": "Sequential"
    }
)

Or programmatically:

from rhesis.backend.tasks.execution.modes import set_execution_mode
from rhesis.backend.tasks.enums import ExecutionMode

success = set_execution_mode(
    db=db,
    test_config_id=test_config.id,
    execution_mode=ExecutionMode.SEQUENTIAL,
    organization_id=org_id
)

Implementation Details

Execution Flow

Both modes produce identical result structures:

{
    "status": "completed",
    "total_tests": 10,
    "tests_passed": 8,
    "tests_failed": 2,
    "execution_errors": 0,
    "execution_time": "2m 15s",
    "completed_at": "2024-01-15 10:30:00"
}

Result Processing

Both modes use the same collect_results task for consistency:

• Status tracking: status, final_status, task_state
• Progress metrics: completed_tests, failed_tests, total_tests
• Timing information: started_at, completed_at, execution_time
• Email notifications triggered for both modes

Task Orchestration

Parallel Mode (batch/__init__.py)

Parallel mode now uses a single task with an internal async fan-out:

# 1) Build execution context (prefetch test + metric data)
ctx = prefetch_execution_context(session, test_config, test_run, tests)

# 2) Run all tests concurrently with a semaphore
results = _run_async(run_batch(ctx, test_ids))

# 3) Trigger standard result collection
trigger_results_collection(test_config, str(test_run.id), results)

Inside run_batch, each test is executed as an asyncio task and guarded by:

• batch_concurrency (default 10)
• per_test_timeout (default 1800 seconds)
• cooperative cancellation checks against Celery revoke state
• one recovery pass for transient failures

Sequential Mode (sequential.py)

Sequential mode runs each test one-by-one in process, then triggers the same results collection task used by parallel mode:

results = []
for test in tests:
    result = asyncio.run(
        execute_test(
            db=session,
            test_config_id=str(test_config.id),
            test_run_id=str(test_run.id),
            test_id=str(test.id),
            endpoint_id=str(test_config.endpoint_id),
        )
    )
    results.append(result)

trigger_results_collection(test_config, str(test_run.id), results)

Choosing an Execution Mode

Decision Matrix

Performance Comparison

Example with 20 tests, 5s average test time:

Monitoring and Debugging

Parallel Mode

# Check worker status
celery -A rhesis.backend.worker inspect active

# Monitor task progress
celery -A rhesis.backend.worker events

Sequential Mode

# Follow sequential execution
tail -f celery.log | grep "execute_test"

Common Issues

Parallel Mode:

• Endpoint returns 429 (rate limit) → Switch to Sequential
• Inconsistent test results → Check for race conditions
• Worker overload → Reduce concurrency or use Sequential
• Run stuck in Progress after worker issues → check task failure/revoked signals and run status transitions

Sequential Mode:

• Tests taking too long → Consider Parallel if endpoint can handle it
• Bottleneck in single test → Optimize that test first

Related Documentation

• Test Execution System - Overall architecture
• Test Types - Single-turn vs Multi-turn
• Background Tasks - Celery configuration
• Architecture - Worker system details