Debugging gRPC Connection Leakage and HTTP/2 Stream Leaks Using eBPF and bpftrace

It is 3:00 AM, and your service’s memory usage is climbing in a perfect linear slope. You check your file descriptor metrics, and they are stable. You query netstat or ss on the host, and the number of active TCP connections is perfectly constant. Yet, the container runtime is aggressively killing your pods due to out-of-memory (OOM) errors, and downstream services are intermittently reporting context deadlines exceeded. This is the insidious nature of gRPC connection leakage and HTTP/2 stream exhaustion. Because gRPC multiplexes many logical requests over a single TCP connection, traditional socket-level metrics will completely miss leaked HTTP/2 streams that continue to consume memory and block the connection stream pools. When standard APMs and OS tools leave you blind, the only way to inspect this state without modifying production code is to go lower in the stack. By leveraging eBPF (Extended Berkeley Packet Filter) and bpftrace, we can inspect kernel-level socket transitions, dissect encrypted HTTP/2 traffic at the syscall boundary, and hook into user-space runtime functions to trace these leaks directly back to the offending lines of code.

The Mechanics of HTTP/2 Multiplexing and Leaks

To understand how these leaks occur, we must first dissect how gRPC sits on top of HTTP/2. Unlike HTTP/1.1, which opens a new TCP connection per concurrent request (or queues them), HTTP/2 establishes a single TCP connection between a client and server and multiplexes multiple bidirectional streams over it.

Each RPC call corresponds to a unique HTTP/2 stream. These streams are identified by a monotonically increasing 31-bit stream ID (odd for client-initiated, even for server-initiated). When an RPC finishes, its stream is terminated with a RST_STREAM frame or a DATA/HEADERS frame containing the END_STREAM flag.

In production, leaks manifest in two distinct architectural failures:

1. TCP Connection Leaks: This occurs when the client application repeatedly instantiates the gRPC client transport (grpc.ClientConn in Go) instead of caching and reusing a singleton client. Every initialization spawns new background TCP connections, goroutines, and buffers. If the application neglects to call Close() on the connection, these sockets remain in the ESTABLISHED state, slowly exhausting kernel socket buffers, ephemeral ports, and file descriptors.
2. HTTP/2 Stream Leaks: This is far more common and difficult to diagnose. The application correctly reuses the grpc.ClientConn, but it leaks individual streams. This typically happens during streaming RPCs (client-streaming, server-streaming, or bidirectional streaming) where the client initiates a stream but fails to consume it to EOF or fails to cancel the underlying context. Because the parent TCP connection is healthy, the OS sees a single socket. However, the client and server runtimes maintain state, memory buffers, and tracking goroutines for every open stream. Eventually, the stream count hits the negotiated MaxConcurrentStreams limit (typically 100). Once this limit is reached, any subsequent RPC on that connection will block indefinitely or time out.

The following code illustrates a classic connection leak where a new connection is instantiated per request:

The next code block illustrates an HTTP/2 stream leak in Go. A client starts a server-streaming RPC but exits the loop early without invoking context cancellation or reading the stream to completion:

Why Traditional Tools Leave You Blind

When troubleshooting these issues, engineers instinctively reach for commands like lsof, netstat, or ss. Let’s look at why they fail to diagnose stream leaks:

• Layer 4 vs. Layer 7 Blindness: ss and lsof query the kernel’s socket table. If your client has leaked 500 streams but routing goes over a single TCP connection, ss will report a single ESTABLISHED socket with small queue sizes. It has zero visibility into the HTTP/2 frames, stream states, or pending stream allocations inside that socket.
• Metric Discrepancies: Prometheus metrics generated by interceptors (e.g., go-grpc-middleware) can show that grpc_client_started_total is much higher than grpc_client_handled_total. However, they do not tell you where the leak is happening or which connection is affected.
• Runtime Profiling Overhead: Using Go’s pprof to dump goroutines can help identify leaked runtime functions (e.g., google.golang.org/grpc/internal/transport.(*http2Client).reader), but doing this under high load can cause significant CPU spikes, and resolving the raw addresses to exact line numbers in production binaries requires symbols that might be stripped.

By running an eBPF program, we bypass these limitations. eBPF operates within the kernel, allowing us to trace socket state changes directly or parse user-space memory buffers at the syscall layer without interrupting the running application.

Instrumenting Socket Transitions with eBPF

To diagnose connection leaks (where connections are created but never closed), we can trace the state transitions of the kernel’s TCP sockets using bpftrace. The kernel tracepoint sock:inet_sock_set_state fires every time a TCP socket transitions (e.g., from SYN_SENT to ESTABLISHED, or from ESTABLISHED to CLOSE).

Here is a bpftrace script that tracks these socket state changes, measures connection lifetimes, and identifies connections that were opened but never terminated:

When you run this script and execute the buggy code from snippet-1, you will see a constant stream of new sockets transitioning to ESTABLISHED without corresponding transitions to CLOSE. When you terminate the script with Ctrl-C, the END block will dump every single socket that was leaked during the run, indicating the target server and the client process name.

Peering into the Stream: eBPF Frame Parsing

When dealing with stream leaks, the socket remains open, so the state transition tracer will not report any anomaly. To detect stream leaks, we must look inside the socket.

HTTP/2 frames start with a 9-byte header:

• Length: 24 bits (3 bytes)
• Type: 8 bits (1 byte)
• Flags: 8 bits (1 byte)
• Stream Identifier: 31 bits (4 bytes)

In non-encrypted development environments (or behind a sidecar proxy where TLS is terminated), we can hook into sys_enter_write and inspect the buffer contents directly.

Here is a bpftrace script that parses write buffers on the fly to detect HTTP/2 frame types and their stream IDs:

If a client is leaking streams, you will see a large number of HEADERS frames (initiating streams) with odd IDs, but you will not see corresponding RST_STREAM frames or DATA frames containing the END_STREAM flag (flag 0x01) for those stream IDs. This asymmetry is the signature of a stream leak.

Uprobes: Tracing gRPC Internals in Go Binaries

In production, gRPC traffic is encrypted via TLS, rendering kernel-level packet filters or plain syscall buffer tracers blind to the payload. To solve this, we can attach user-space probes (uprobes) directly to the application binary. Go binaries, by default, contain symbols that allow us to attach uprobes to specific functions.

In the Go gRPC client runtime, every new HTTP/2 stream is instantiated using google.golang.org/grpc/internal/transport.(*http2Client).NewStream. By attaching a uprobe to this symbol, we can log stream allocations and extract the user-space stack trace (ustack()) to locate the exact file and line number creating the stream.

Since Go 1.17, arguments are passed in registers. For x86_64:

• AX: Receives the first argument (often the receiver struct *http2Client)
• BX, CX: Interface context values
• DX: Receives the third argument (in this case, the *transport.Stream pointer)

When you execute this script against the leaking client binary, every stream allocation will print a detailed stack trace showing the exact invocation path, leading directly to the buggy client wrapper in snippet-2.

Architectural Mitigation: Hardening gRPC Configurations

While eBPF provides the diagnostic capabilities to find these leaks, you must harden your gRPC layer to prevent them from taking down your infrastructure when they inevitably occur.

1. Hardening the gRPC Client

Clients must define explicit timeout policies, enforce keepalive probes to clean up orphaned connections, and use proper context patterns.

2. Hardening the gRPC Server

The server must proactively manage idle connections, cycle old connections to facilitate load balancing, and set hard limits on concurrent streams.

Summary Checklist for Production Debugging

If you suspect a leak in your cluster, follow this runbook:

1. Verify Socket Count: Run ss -antp | grep <port> to check if the total number of TCP connections matches your expectations. If it is steadily rising, use the socket state transition script (Snippet 3) to identify connection leakage.
2. Profile Goroutines: Use Go’s pprof to check if grpc.ClientConn or transport reader goroutines are accumulating.
3. Trace Stream Allocation: In development/staging, run the HTTP/2 frame parsing script (Snippet 4) to track unmatched HEADERS and RST_STREAM frames.
4. Locate Code Origin: Attach the uprobe script (Snippet 5) to the binary’s NewStream function to capture stack traces at the point of stream creation.
5. Mitigate: Apply the client and server configurations from Snippet 6 and Snippet 7 to enforce connection lifetimes and protect system memory.