Test Result Status

A test result’s status records whether the test passed its metric evaluations. It is computed once at execution time and stored on the TestResult.status relationship, which is then the source of truth for stats, email, UI, and API.

Status values

TestResultStatus (app/constants.py) has three values:

Status	Meaning
`Pass`	At least one metric exists and every metric has `is_successful: true`.
`Fail`	At least one metric has `is_successful: false`.
`Error`	No metrics were evaluated (missing/empty `test_metrics`) or the endpoint returned an HTTP error. Counted as an execution error, not a pass or fail.

A single failed metric fails the whole test.

Test metrics structure

The status is derived from the test_metrics JSON stored on the result:

test-metrics-structure.json
{
  "execution_time": 1.23,
  "metrics": {
    "Answer Relevancy": {
      "is_successful": true,
      "score": 0.85,
      "threshold": 0.7,
      "reason": "Answer is relevant to the question"
    },
    "Contextual Recall": {
      "is_successful": false,
      "score": 0.65,
      "threshold": 0.7,
      "reason": "Failed to recall sufficient context"
    }
  }
}

Two metrics with is_successful: true and one false yields status Fail. All true yields Pass. Missing or empty metrics yields Error.

Determination logic

determine_status_from_metrics (tasks/execution/executors/metrics.py) and create_test_result_record (tasks/execution/executors/results.py) apply the same rule:

status-determination.py
if has_http_error_in_result(processed_result):
    status_value = TestResultStatus.ERROR.value
elif not metrics_results:
    status_value = TestResultStatus.ERROR.value
else:
    all_metrics_passed = all(
        m.get("is_successful", False)
        for m in metrics_results.values()
        if isinstance(m, dict)
    )
    status_value = (
        TestResultStatus.PASS.value if all_metrics_passed else TestResultStatus.FAIL.value
    )

Status is set in three places:

Automated execution by the worker (executors/results.py).
POST /test_results and PUT /test_results/\{id\} (app/routers/test_result.py) — auto-set from test_metrics when status_id is not provided. Pass an explicit status_id to override.

Source of truth

After execution the stored status is authoritative. Statistics and reporting read it through categorize_test_result_status() (app/constants.py), which collapses the status name into a pass/fail/error bucket — they do not re-derive pass/fail from test_metrics. Deriving once at execution time keeps every surface consistent and avoids re-parsing JSONB.

Read test_metrics directly only for per-metric analytics, drill-down, and debugging — never to recompute overall pass/fail. Note that human reviews can override the stored status or individual metric outcomes; statistics account for those overrides.