Test Reviews

Human reviews let a reviewer record a verdict on a test result alongside the automated metric outcome — to confirm it, disagree with it, or comment on a specific metric. Reviews are stored in the test_reviews JSONB column on test_result; a metric- or test-level review can override the stored status, which is why statistics read the effective (post-review) result.

Structure

test_reviews holds a metadata summary (kept current on every operation, for fast list views) and a reviews array:

test-reviews.json
{
  "metadata": {
    "last_updated_at": "2025-10-10T14:15:00Z",
    "last_updated_by": {
      "user_id": "a1e2...",
      "name": "Alice"
    },
    "total_reviews": 1,
    "latest_status": {
      "status_id": "b6f1...",
      "name": "Pass"
    },
    "summary": "Last updated by Alice"
  },
  "reviews": [
    {
      "review_id": "e9b5a2c7-13f2-4a91-94d0-4df8c1c5f0a1",
      "status": {
        "status_id": "b6f1...",
        "name": "Pass"
      },
      "user": {
        "user_id": "a1e2...",
        "name": "Alice"
      },
      "comments": "LLM refusal was appropriate. Marking as passed.",
      "created_at": "2025-10-10T14:10:00Z",
      "updated_at": "2025-10-10T14:15:00Z",
      "target": {
        "type": "metric",
        "reference": "Refusal Detection"
      }
    }
  ]
}

Fields

Field	Type	Description
`review_id`	UUID	Unique ID, auto-generated on create.
`status`	object	`status_id` (UUID) and `name`, resolved from the Status model.
`user`	object	`user_id` (UUID) and `name` of the reviewer.
`comments`	string	Free-text reviewer note.
`created_at` / `updated_at`	ISO 8601	Set on create; `updated_at` refreshed on edit.
`target`	object	What the review applies to (see below).

metadata mirrors the latest state: last_updated_at, last_updated_by, total_reviews, latest_status, and a summary.

Target

target.type is a ReviewTarget value. For test results it is test_result (the whole result) or metric (a single metric, named in reference). The legacy value test is accepted and normalized to test_result.

target-values.json
{ "type": "test_result", "reference": null }
{ "type": "metric", "reference": "Refusal Detection" }

API

All four endpoints are on app/routers/test_result.py and update metadata automatically.

Method & path	Purpose
`POST /test_results/\{test_result_id\}/reviews`	Create a review (201). Auto-fills `review_id`, timestamps, and reviewer from the authenticated user; embeds status details from the Status model.
`PUT /test_results/\{test_result_id\}/reviews/\{review_id\}`	Update a review. All body fields optional; preserves `created_at`, refreshes `updated_at`.
`DELETE /test_results/\{test_result_id\}/reviews/\{review_id\}`	Remove a review; returns the deleted entry. Clears `latest_status` when the last review is removed.
`GET /test_results/\{test_result_id\}`	Returns the result with `test_reviews` plus the derived `last_review` and `matches_review`.

Create request

create-review.json
{
  "status_id": "735acfa0-cca2-48a1-bb90-ba10b16f1cdb",
  "comments": "Looks good after manual inspection",
  "target": {
    "type": "test_result",
    "reference": null
  }
}

Schemas: ReviewCreate, ReviewUpdate, ReviewResponse, ReviewTargetCreate in app/schemas/test_result.py.

Derived properties

ReviewsMixin (app/models/mixins.py) exposes read-only properties on TestResult:

last_review — the most recent review by updated_at, or None.
matches_review — whether the result’s status_id matches the latest review’s status.
review_summary — the metadata summary string.

Implementation notes

The column stores the whole structure as JSONB, so no extra tables are needed. When mutating it, flag the change so SQLAlchemy persists it:

flag-modified.py
from sqlalchemy.orm.attributes import flag_modified

flag_modified(test_result, "test_reviews")
db.commit()

When the last review is deleted, reviews becomes empty, total_reviews is 0, latest_status is null, last_review returns None, and matches_review returns False.