Data Structures

Schemas, database design, and data formats for the tracing system.

Span Structure

Canonical Span Schema

Spans sent from SDK to backend follow the shared OTELSpan schema (packages/rhesis/src/rhesis/telemetry/schemas.py), serialized as JSON:

Span Payload
{
  "trace_id": "a1b2c3d4e5f6...",
  "span_id": "1234567890abcdef",
  "parent_span_id": null,
  "project_id": "my-project",
  "environment": "development",
  "conversation_id": null,
  "span_name": "ai.llm.invoke",
  "span_kind": "CLIENT",
  "start_time": "2024-01-01T00:00:00.000000Z",
  "end_time": "2024-01-01T00:00:01.500000Z",
  "status_code": "OK",
  "status_message": null,
  "attributes": {
    "ai.model.name": "gpt-4",
    "ai.model.provider": "openai",
    "ai.llm.tokens.input": 10,
    "ai.llm.tokens.output": 25,
    "rhesis.test.run_id": "uuid",
    "rhesis.test.id": "uuid"
  },
  "events": [
    {
      "name": "ai.prompt",
      "timestamp": "2024-01-01T00:00:00.100000Z",
      "attributes": {
        "ai.prompt.role": "user",
        "ai.prompt.content": "Hello, world!"
      }
    },
    {
      "name": "ai.completion",
      "timestamp": "2024-01-01T00:00:01.400000Z",
      "attributes": {
        "ai.completion.content": "Hi there! How can I help?"
      }
    }
  ],
  "links": [],
  "resource": {
    "service.name": "my-service",
    "service.namespace": "rhesis",
    "deployment.environment": "development"
  }
}

span_name must match ai.<domain>.<action> (domains chain, workflow, pipeline rejected) or function.<name>.

Test Execution Context

Context attributes added to spans during test execution:

Test Context Attributes
test_execution_context = {
    "rhesis.test.run_id": "uuid",              # Which test run
    "rhesis.test.id": "uuid",                  # Which test definition
    "rhesis.test.configuration_id": "uuid",    # Which configuration
    # rhesis.test.result_id is linked after creation
}

Database Schema

trace Table

SQLAlchemy Model
class Trace(Base, EmbeddableMixin, TagsMixin, CommentsMixin, TasksMixin, FilesMixin, ReviewsMixin):
    __tablename__ = "trace"

    # id, nano_id, created_at, updated_at, deleted_at come from Base

    trace_id = Column(String(32), nullable=False, index=True)      # OTEL trace ID
    span_id = Column(String(16), nullable=False, index=True)       # OTEL span ID
    parent_span_id = Column(String(16), nullable=True, index=True)

    project_id = Column(GUID(), ForeignKey("project.id", ondelete="CASCADE"), nullable=False, index=True)
    organization_id = Column(GUID(), nullable=False, index=True)    # not a FK
    environment = Column(String(50), nullable=False, index=True)
    conversation_id = Column(String(255), nullable=True)

    test_run_id = Column(GUID(), ForeignKey("test_run.id", ondelete="SET NULL"), nullable=True, index=True)
    test_result_id = Column(GUID(), ForeignKey("test_result.id", ondelete="SET NULL"), nullable=True, index=True)
    test_id = Column(GUID(), ForeignKey("test.id", ondelete="SET NULL"), nullable=True, index=True)

    span_name = Column(String(255), nullable=False, index=True)
    span_kind = Column(String(20), nullable=False)

    start_time = Column(DateTime(timezone=True), nullable=False, index=True)
    end_time = Column(DateTime(timezone=True), nullable=False)
    duration_ms = Column(Float, nullable=False)

    status_code = Column(String(20), nullable=False, index=True)
    status_message = Column(Text, nullable=True)

    attributes = Column(JSONB, nullable=False, default=dict)
    events = Column(JSONB, nullable=False, default=list)
    links = Column(JSONB, nullable=False, default=list)
    resource = Column(JSONB, nullable=False, default=dict)

    processed_at = Column(DateTime(timezone=True), nullable=True)  # set once enrichment runs
    enriched_data = Column(JSONB, default=dict)

    trace_metrics = Column(JSONB, nullable=True)                   # LLM-based metric evaluation
    trace_metrics_status_id = Column(GUID(), ForeignKey("status.id"), nullable=True, index=True)
    trace_metrics_processed_at = Column(DateTime(timezone=True), nullable=True)

    trace_reviews = Column(JSONB, nullable=True)                   # human review state

Column Details

Column	Type	Description
`id`	UUID	Primary key
`trace_id`	VARCHAR(32)	OpenTelemetry trace ID (groups spans)
`span_id`	VARCHAR(16)	OpenTelemetry span ID (unique per span)
`parent_span_id`	VARCHAR(16)	Parent span for hierarchy
`project_id`	UUID	Project isolation, FK with `ON DELETE CASCADE`
`organization_id`	UUID	Multi-tenancy isolation (no FK)
`environment`	VARCHAR(50)	e.g. `development`, `production`
`conversation_id`	VARCHAR(255)	Groups turns of a multi-turn conversation
`test_run_id` / `test_result_id` / `test_id`	UUID	Linked test execution entities
`span_name`	VARCHAR(255)	Operation name (`ai.llm.invoke`)
`span_kind`	VARCHAR(20)	OTEL span kind (`CLIENT`, `INTERNAL`, …)
`duration_ms`	FLOAT	Calculated duration
`status_code`	VARCHAR(20)	`OK`, `ERROR`, `UNSET`
`attributes` / `events` / `links` / `resource`	JSONB	Span data
`processed_at`	TIMESTAMP	Set when enrichment last ran; `NULL` triggers re-enrichment
`enriched_data`	JSONB	Cached enrichment results
`trace_metrics` / `trace_metrics_status_id` / `trace_metrics_processed_at`	JSONB / UUID / TIMESTAMP	LLM-based metric evaluation results and status
`trace_reviews`	JSONB	Human review annotations

Indexes

Critical Indexes
CREATE INDEX idx_trace_trace_id ON trace(trace_id, start_time);       -- get all spans for a trace
CREATE INDEX idx_trace_project_time ON trace(project_id, start_time DESC);
CREATE INDEX idx_trace_org_time ON trace(organization_id, start_time DESC);
CREATE INDEX idx_trace_span_name_time ON trace(span_name, start_time DESC);
CREATE INDEX idx_trace_environment_time ON trace(environment, start_time DESC);
CREATE INDEX idx_trace_status_time ON trace(status_code, start_time DESC);
CREATE INDEX idx_trace_test_run ON trace(test_run_id, start_time DESC);
CREATE INDEX idx_trace_test_result ON trace(test_result_id);
CREATE INDEX idx_trace_test ON trace(test_id);
CREATE INDEX idx_trace_conversation ON trace(conversation_id, start_time DESC);

-- JSONB attribute queries
CREATE INDEX idx_trace_attributes ON trace USING GIN(attributes jsonb_path_ops);

-- Partial indexes for background workers scanning unprocessed rows
CREATE INDEX idx_trace_unprocessed ON trace(created_at) WHERE processed_at IS NULL;
CREATE INDEX idx_trace_metrics_unprocessed ON trace(created_at) WHERE trace_metrics_processed_at IS NULL;

Enrichment Data

The enriched_data JSONB column caches computed values (EnrichedTraceData, schemas/enrichment.py):

Enrichment Structure
{
  "costs": {
    "total_cost_usd": 0.023,
    "total_cost_eur": 0.021,
    "breakdown": [
      {
        "span_id": "1234567890abcdef",
        "model_name": "gpt-4",
        "input_tokens": 150,
        "output_tokens": 80,
        "total_cost_usd": 0.023,
        "total_cost_eur": 0.021
      }
    ]
  },
  "anomalies": [
    {
      "type": "slow_span",
      "span_id": "1234567890abcdef",
      "span_name": "ai.llm.invoke",
      "duration_ms": 12340,
      "message": "Span took 12.3s (threshold: 10s)"
    }
  ],
  "metrics": {
    "total_duration_ms": 12500,
    "span_count": 5,
    "error_count": 0
  },
  "models_used": [
    "gpt-4"
  ],
  "tools_used": [
    "search"
  ],
  "operation_types": [
    "llm.invoke",
    "tool.invoke"
  ],
  "root_operation": "ai.llm.invoke"
}

Enrichment Fields

Field	Description
`costs.total_cost_usd` / `total_cost_eur`	Total cost across all LLM spans
`costs.breakdown`	Per-span cost breakdown
`anomalies`	Detected anomalies: `slow_span` (>10s), `high_token_usage` (>10,000 tokens), `error`
`metrics`	Trace-level duration, span count, error count
`models_used` / `tools_used` / `operation_types`	Unique values seen across the trace’s spans
`root_operation`	The root span’s `span_name`

Common Query Patterns

Get Trace by ID

Query
SELECT * FROM trace
WHERE trace_id = 'abc123...'
ORDER BY start_time ASC;
-- Uses: idx_trace_trace_id

Get Traces for Test Run

Query
SELECT DISTINCT trace_id, MIN(start_time) as trace_start
FROM trace
WHERE test_run_id = 'uuid'
GROUP BY trace_id
ORDER BY trace_start DESC;
-- Uses: idx_trace_test_run

Get LLM Calls with Specific Model

Query
SELECT * FROM trace
WHERE attributes @> '{"ai.model.name": "gpt-4"}'
AND project_id = 'uuid'
ORDER BY created_at DESC;
-- Uses: idx_trace_attributes (GIN index)

Get Error Traces

Query
SELECT DISTINCT trace_id, span_name, status_code
FROM trace
WHERE status_code = 'ERROR'
AND project_id = 'uuid'
ORDER BY created_at DESC
LIMIT 100;
-- Uses: idx_trace_status_time

Get High-Cost Traces

Query
SELECT trace_id,
       enriched_data->'costs'->>'total_cost_usd' as cost_usd,
       enriched_data->>'models_used' as models
FROM trace
WHERE enriched_data IS NOT NULL
AND (enriched_data->'costs'->>'total_cost_usd')::float > 0.10
AND project_id = 'uuid'
ORDER BY (enriched_data->'costs'->>'total_cost_usd')::float DESC
LIMIT 50;

HTTP Request Format

Ingestion Endpoint

Endpoint: POST /telemetry/traces

Headers:

Headers
Authorization: Bearer <api_key>
Content-Type: application/json

Payload:

Request Body
{
  "spans": [
      { ... span 1 ... },
      { ... span 2 ... }
  ]
}

Response Codes

Status	Meaning	Action
200	Success	Spans ingested (post-processing dispatched separately)
401	Unauthorized	Check API key
422	Validation error, or no `project_id` could be resolved	Fix span format, or pass a project-scoped token / `X-Project-Id` header
500	Server error	Retry with backoff

Validation Errors

Common 422 errors:

Validation Error
{
  "detail": [
    {
      "loc": [
        "spans",
        0,
        "span_name"
      ],
      "msg": "span_name cannot use framework concept 'chain'. Use primitive operations: llm, tool, retrieval, embedding",
      "type": "value_error"
    }
  ]
}

Why PostgreSQL + JSONB?

Aspect	Benefit
Single Database	Simplifies operations, existing expertise
JSONB Flexibility	Schema can evolve without migrations
GIN Indexes	Fast attribute queries
ACID Compliance	Reliable linking operations
Familiar SQL	Easy debugging and ad-hoc queries

Future Scaling

If trace volume exceeds PostgreSQL capacity:

Partition by time - Monthly partitions for retention
TimescaleDB - Hypertable for time-series optimization
ClickHouse - Columnar store for analytics
Archive strategy - Move old traces to cold storage