Setting Up OpenTelemetry Distributed Tracing Across Node.js Services
A request that touched four internal services and took five seconds instead of the usual two hundred milliseconds used to mean opening four separate log dashboards, eyeballing timestamps, and guessing which service actually caused the delay. Wiring up OpenTelemetry distributed tracing across those services replaced that guesswork with an actual trace showing exactly which hop ate the time, and getting the auto-instrumentation and context propagation right across service boundaries was less about writing tracing code and more about correctly wiring existing pieces together.
The Instrumentation File Has to Load Before Anything Else
OpenTelemetry's Node SDK works by patching modules as they're required, wrapping http, pg, express, and others with tracing logic. If application code imports and uses those modules before the SDK has registered its patches, the calls happen unpatched and produce no spans. This means the instrumentation setup has to run first, before any application import, which in practice means loading it via Node's --import flag rather than importing it as a regular module inside your entry file, where import ordering isn't guaranteed to run early enough.
// instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { resourceFromAttributes } from '@opentelemetry/resources';
const sdk = new NodeSDK({
resource: resourceFromAttributes({
'service.name': process.env.SERVICE_NAME ?? 'orders-api',
}),
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
}),
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();# package.json start script
node --import ./dist/instrumentation.js dist/server.jsAuto-Instrumentation Covers More Than I Expected
getNodeAutoInstrumentations() pulls in a bundle covering HTTP, Express, and common database drivers automatically, creating spans for incoming requests, outgoing calls, and database queries without touching route handler code at all. This alone, before writing a single manual span, gave visibility into which downstream call within a request was slow, which was most of what the original five-second-request mystery actually needed to solve.
Context Propagation Across a Service Boundary Isn't Automatic by Magic
Auto-instrumentation traces within one service; a trace spanning multiple services requires the trace context to travel with the request itself. HTTP instrumentation handles this by injecting a traceparent header on outgoing requests and reading it on incoming ones, following the W3C Trace Context standard, which is what stitches Service A's span and Service B's span into the same trace instead of two disconnected traces that happen to have similar timestamps. This works automatically for standard HTTP clients the auto-instrumentation covers, but a service communicating over something else, a message queue, a raw TCP socket, needs the trace context manually extracted and injected at that boundary instead.
Manual Spans for the Business Logic Auto-Instrumentation Can't See
Auto-instrumentation traces "an HTTP call happened" and "a database query ran," but has no idea a chunk of application code is doing meaningful, potentially slow work in between, validating a complex payload, running a pricing calculation. Wrapping that logic in a manually created span, using the tracer obtained from the OpenTelemetry API, made a slow pricing calculation visible as its own named span in the trace instead of just appearing as unexplained time between two auto-instrumented spans.
import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('orders-api');
async function calculatePricing(order) {
return tracer.startActiveSpan('calculate-pricing', async (span) => {
try {
const result = await runPricingEngine(order);
span.setAttribute('order.item_count', order.items.length);
return result;
} finally {
span.end();
}
});
}Sampling: Tracing Everything Gets Expensive Fast
Tracing 100% of requests on a high-traffic service generates a genuinely large volume of span data and a real bill from whichever backend is storing it. Configuring a sampler, we used a parent-based sampler that traces a fixed percentage of new traces but always continues tracing any trace that started sampled elsewhere in the chain, kept costs reasonable while ensuring a trace was never partially recorded, missing spans from services later in the chain because a downstream sampler independently decided not to record.
The Collector Sits Between Your Services and the Backend
Exporting traces directly from every service straight to the observability vendor's endpoint works but couples every service to that vendor's specific API and rate limits. Running an OpenTelemetry Collector as a separate lightweight service that receives traces from all our services and forwards them onward let us change observability backends once, in the collector's config, without touching a single line of instrumented service code, a decoupling that paid for itself the one time we did switch vendors.
Final Verdict
The auto-instrumentation packages do most of the actual work, HTTP and database spans and cross-service context propagation come essentially for free once the instrumentation file is loading correctly before application code. The real engineering is in the pieces auto-instrumentation can't see: manual spans around meaningful business logic, a sampling strategy that doesn't break trace continuity, and a collector layer that keeps services from being hard-wired to one specific backend.
