Tracing System
Technical documentation for the Rhesis tracing system architecture and implementation.
For SDK Users: See the Tracing documentation for usage guides. This section covers the internal architecture and design for developers and contributors.
Overview
The tracing system captures OpenTelemetry-compliant traces from SDK-instrumented applications. It supports two operating modes:
- Test Mode: Traces linked to test runs, test cases, and test results
- Production Mode: Traces from live application monitoring
High-Level Architecture
Key Technologies
| Component | Technology | Purpose |
|---|---|---|
| SDK Tracer | OpenTelemetry Python | Span creation with AI semantic conventions |
| Batch Processor | OTEL BatchSpanProcessor | Batches spans, exports every 5 seconds |
| Transport | OTLP/HTTP | Chunked (100 spans/request), retried with backoff, to /telemetry/traces |
| Storage | PostgreSQL + JSONB | Flexible span storage, GIN-indexed attributes |
| Post-processing | Celery | Test-result linking, cost/anomaly enrichment, metric evaluation |
| Linking | Service layer | Hybrid strategy for test-context linking |
Communication Channels
The SDK uses two independent channels:
| Channel | Protocol | Purpose | Used By |
|---|---|---|---|
| Tracing | HTTP POST | Export OpenTelemetry spans | @observe, @endpoint |
| Testing | WebSocket | Remote test invocation | @endpoint only |
Design Principles
- OpenTelemetry Standard - Industry-standard OTLP protocol for interoperability
- Hybrid Linking - Two strategic linking points to handle race conditions
- Idempotent Operations - Safe to call linking multiple times
- Cache Enrichment - Enrichment is skipped when every span in a trace is already processed
- Async Post-Processing - Ingestion never blocks on linking, enrichment, or metric evaluation
- Fire-and-Forget Dispatch - If the Celery broker is unreachable at dispatch time, ingestion still succeeds; post-processing for that batch is skipped and logged as a warning
Performance Characteristics
| Operation | Timing | Notes |
|---|---|---|
| Span creation (SDK) | ~0.1ms | Per span, negligible overhead |
| BatchProcessor delay | 5000ms | Fixed by OpenTelemetry design |
| Span export (OTLP) | ~10ms | Network call |
| Backend ingestion | ~10-20ms | Spans stored before async dispatch |
| Linking + enrichment | ~50-100ms | Background (Celery, post_ingest_link) |
| Trace query | ~10ms | Cached enrichment |
| End-to-end | ~5 seconds | Test start to queryable trace |
Key Files
SDK
| File | Purpose |
|---|---|
sdk/src/rhesis/sdk/telemetry/tracer.py | Tracer class — creates spans via the shared OTEL provider |
packages/rhesis/src/rhesis/telemetry/provider.py | TracerProvider + BatchSpanProcessor setup |
packages/rhesis/src/rhesis/telemetry/exporter.py | RhesisOTLPExporter — chunked, retrying OTLP HTTP export |
packages/rhesis/src/rhesis/telemetry/schemas.py | Canonical span schema, span-name validation |
sdk/src/rhesis/sdk/telemetry/attributes.py | AI semantic convention attribute constants |
sdk/src/rhesis/sdk/decorators/observe.py | @observe decorator |
sdk/src/rhesis/sdk/decorators/endpoint.py | @endpoint decorator |
Backend
| File | Purpose |
|---|---|
apps/backend/.../routers/telemetry.py | Ingestion (POST /telemetry/traces) and query endpoints |
apps/backend/.../tasks/telemetry/post_ingest.py | post_ingest_link — linking + enrichment dispatch |
apps/backend/.../services/telemetry/linking_service.py | TraceLinkingService |
apps/backend/.../services/telemetry/enrichment/ | Cost/anomaly enrichment (core.py, processor.py) |
apps/backend/.../models/trace.py | Trace SQLAlchemy model |
apps/backend/.../crud.py | create_trace_spans(), update_traces_with_test_result_id(), get_trace_by_id(), mark_trace_processed() |
Next Steps
- Architecture - Detailed component architecture
- Trace Lifecycle - Complete flow and race condition handling
- Data Structures - Schemas and database design