Data Structures
Schemas, database design, and data formats for the tracing system.
Span Structure
OTLP Span Format
Spans sent from SDK to backend follow the OTLP/JSON format:
Test Execution Context
Context attributes added to spans during test execution:
Database Schema
traces Table
Column Details
| Column | Type | Description |
|---|---|---|
id | UUID | Primary key (internal) |
trace_id | VARCHAR(32) | OpenTelemetry trace ID (groups spans) |
span_id | VARCHAR(16) | OpenTelemetry span ID (unique per span) |
parent_span_id | VARCHAR(16) | Parent span for hierarchy |
span_name | VARCHAR(255) | Operation name (ai.llm.invoke) |
start_time | TIMESTAMP | Span start time |
end_time | TIMESTAMP | Span end time |
duration_ms | FLOAT | Calculated duration |
status_code | VARCHAR(50) | OK, ERROR, UNSET |
organization_id | UUID | Multi-tenancy isolation |
project_id | UUID | Project isolation |
test_run_id | UUID | Linked test run |
test_result_id | UUID | Linked test result |
test_id | UUID | Linked test definition |
attributes | JSONB | Span attributes |
events | JSONB | Span events (prompts, completions) |
enriched_data | JSONB | Cached enrichment results |
Indexes
Index Usage
| Query | Index Used |
|---|---|
| Get spans by trace_id | idx_trace_trace_id |
| Get traces for test run | idx_trace_test_run |
| Query by attribute (model, provider) | idx_trace_attributes |
| Filter by organization | idx_trace_org_project |
| Find error traces | idx_trace_status |
Enrichment Data
The enriched_data JSONB column caches computed values:
Enrichment Fields
| Field | Description |
|---|---|
costs.total_cost_usd | Total cost in USD |
costs.total_cost_eur | Total cost in EUR |
costs.breakdown | Per-span cost breakdown |
anomalies | Detected anomalies |
metadata.models_used | Unique models in trace |
metadata.total_tokens | Sum of all tokens |
metadata.span_count | Number of spans |
enriched_at | Enrichment timestamp |
Common Query Patterns
Get Trace by ID
Get Traces for Test Run
Get LLM Calls with Specific Model
Get Error Traces
Get High-Cost Traces
HTTP Request Format
Ingestion Endpoint
Endpoint: POST /telemetry/traces
Headers:
Payload:
Response Codes
| Status | Meaning | Action |
|---|---|---|
| 200 | Success | Spans ingested |
| 401 | Unauthorized | Check API key |
| 422 | Validation Error | Fix span names/attributes |
| 500 | Server Error | Retry with backoff |
Validation Errors
Common 422 errors:
Why PostgreSQL + JSONB?
| Aspect | Benefit |
|---|---|
| Single Database | Simplifies operations, existing expertise |
| JSONB Flexibility | Schema can evolve without migrations |
| GIN Indexes | Fast attribute queries |
| ACID Compliance | Reliable linking operations |
| Familiar SQL | Easy debugging and ad-hoc queries |
Future Scaling
If trace volume exceeds PostgreSQL capacity:
- Partition by time - Monthly partitions for retention
- TimescaleDB - Hypertable for time-series optimization
- ClickHouse - Columnar store for analytics
- Archive strategy - Move old traces to cold storage