Stop Instrumenting Your Code: Why eBPF is the New Baseline for Observability

There's a moment in every high-throughput system where you realise your distributed traces are lying to you. You look at a Jaeger flame graph showing the API gateway calling a downstream payment service. The gateway reports a 300ms response time. The payment service reports it processed the request in 15ms. That leaves 285ms of empty, unaccounted space.

Your first instinct is to blame the network. Then you blame the load balancer. The truth is usually worse: the request was sitting in a Linux CPU runqueue, it suffered a silent TCP retransmit, or the container was aggressively CPU-throttled by the hypervisor. Your application-layer tracing SDKs cannot see any of this because they operate entirely in user space.

If you are running a fintech ledger or a high-frequency trading platform, relying on manual application instrumentation is a performance and maintenance liability. You are paying a serialization tax to capture incomplete data.

Here is why platform teams are shifting to eBPF for observability, and what you should actually deploy.

The Blind Spot in Your Tracing SDK

When you instrument a Go or Java microservice using the standard OpenTelemetry SDKs, you are wrapping business logic in user-space timers. You add @WithSpan in Java or tracer.Start(ctx) in Go.

This approach has three fatal flaws in production:

1. It misses the kernel. A user-space span starts when the application pulls the request off the socket. It misses the time the packet spent in the NIC queue, the iptables routing overhead, and the CPU scheduling delay before the thread actually woke up.
2. It creates garbage. In a system handling 10,000 TPS, allocating span objects, serializing them, and flushing them over gRPC to a collector introduces measurable GC pressure and CPU overhead.
3. It rots. Codebases evolve. Engineers forget to wrap new database calls in spans. You end up with broken trace contexts and orphaned spans.

You cannot optimise what you cannot see. If a pod is experiencing a noisy neighbour issue on the Kubernetes node, your SDK trace just shows a database query taking 200ms instead of 2ms. It doesn't tell you why.

What eBPF Actually Sees

eBPF allows you to run sandboxed programs directly in the Linux kernel without changing kernel source code or loading unstable modules. Instead of guessing why a network call was slow, you hook directly into the kernel's execution path.

By attaching to kprobes and tracepoints, an eBPF agent can measure the exact microsecond a packet hits tcp_retransmit_skb. It sees exactly how long a thread spent in the runqueue waiting for CPU time. It tracks DNS resolution wait times at the socket layer, completely bypassing whatever DNS caching your application runtime claims to be doing.

You get this visibility without modifying a single line of application code. You drop a DaemonSet onto your Kubernetes nodes, and the kernel starts emitting telemetry.

The OTel eBPF Profiler Changes the Math

For years, eBPF observability was firmly in the "experimental" bucket. You had to write custom C code, compile it against specific Linux headers using BCC, and pray it didn't panic your node.

The General Availability of the OpenTelemetry eBPF-based continuous profiler changes this. Built on Linux CO-RE (Compile Once, Run Everywhere), it requires Linux 5.8+ but completely removes the compilation headache. It is now a zero-code, production-ready standard.

You can enable it in the OpenTelemetry Collector via the ebpf receiver. Here is what that actually looks like in your otelcol.yaml as of recent releases:

receivers:
  ebpf:
    collection_interval: 10s
    protocols:
      http:
        endpoint: 0.0.0.0:8080
    metrics:
      tcp.retransmits:
        enabled: true
      process.cpu.runqueue.delay:
        enabled: true

processors:
  batch:
    send_batch_size: 10000
    timeout: 5s

exporters:
  otlp:
    endpoint: "otel-gateway:4317"
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [ebpf]
      processors: [batch]
      exporters: [otlp]

This configuration grabs network and scheduling metrics straight from the kernel and pushes them into your standard OTLP pipeline. No sidecars, no SDK updates, no redeploying your services.

The Contenders

If you are evaluating eBPF observability today, you are generally looking at four paths.

The TLS Blind Spot

eBPF is not magic. There is one specific failure mode you will hit the moment you roll this out: encrypted payloads.

Because eBPF network tracing typically hooks at the socket layer, it reads raw bytes. If your microservices communicate over TLS (which they should, especially in BFSI), the kernel only sees encrypted ciphertext. eBPF cannot parse the HTTP headers or the gRPC method names from a TLS stream.

To get around this, eBPF tools use uprobes to hook directly into the user-space crypto libraries (like OpenSSL or BoringSSL) before the payload is encrypted.

This is incredibly brittle. If your application uses a non-standard crypto library, or if you write in Go (where crypto/tls frequently changes its ABI between minor versions), the uprobe will fail to attach. Your eBPF agent will silently drop the payload, and your L7 metrics will vanish.

If you rely heavily on Go microservices and strict mTLS, be prepared to babysit your eBPF agent upgrades every time you bump your Go compiler version.

What I Actually Do

I do not rip out all application SDKs. eBPF cannot tell you that "User 123 failed a KYC check." Business logic still belongs in user-space spans.

But I completely ban manual instrumentation for infrastructure concerns. You should not be writing spans to measure database query latency, HTTP client request times, or DNS resolution.

If you are running Linux 5.8 or newer, deploy the OpenTelemetry Collector as a DaemonSet with the eBPF receiver enabled. Let the kernel measure the network and the CPU. Keep your application code focused on the business domain. The moment you see a 200ms gap in your trace, the eBPF runqueue metrics will tell you exactly which hypervisor stole your CPU cycles.