Distributed Tracing with OpenTelemetry: From Zero to Production

In a microservices system, a single user request touches 5–10 services. When something is slow or broken, how do you know which service is responsible? Distributed tracing gives you the complete picture.

Core Concepts

• Trace: A complete end-to-end journey of one request through your system
• Span: A single operation within a trace (one HTTP call, one DB query)
• Context propagation: Passing trace metadata (trace ID, span ID) across service boundaries via headers
• Baggage: Key-value pairs that travel with the trace (e.g., user ID, request ID)

OpenTelemetry Setup in Go

Initialize the Tracer Provider

Instrument HTTP Handlers

Create Custom Spans

Propagate Context Across HTTP Calls

Docker Compose for Jaeger

Open http://localhost:16686 to see traces.

Production Sampling Strategy

For critical paths (payment, auth), use AlwaysSample. For high-volume health checks, use NeverSample.

What Good Traces Tell You

1. Latency breakdown: Which service/query is slow?
2. Error propagation: Where did the error originate?
3. Dependency map: Which services call which?
4. Bottlenecks: Sequential calls that could be parallelized

A single trace showing DB query: 2.3s out of a total: 2.5s request tells you exactly where to optimize — no guesswork.