Building a Zero-Copy TCP Loopback Server in Go using splice(2) and sendfile(2) System Calls

In high-throughput microservices architectures, sidecar proxies like Envoy or Linkerd, and local service meshes, the network loopback interface (lo) is a major source of silent CPU drain. At packet rates exceeding 1 million per second, the traditional kernel-to-user-space copy loop (read(2) into an application buffer, followed by a write(2) back to the destination socket) saturates L1/L2 caches, triggers persistent TLB shootdowns, and drives up p99 tail latencies via Go garbage collection pauses. When you are moving gigabytes of payload data per second through a local proxy, copying that data across the user-space boundary is a design failure. By leveraging the Linux kernel’s splice(2) and sendfile(2) system calls directly in Go, we can route TCP payloads entirely within kernel space, achieving a 3x throughput improvement while slashing CPU usage by over 60%.

The Memory Bottleneck of Traditional Network I/O

In typical network programming, transferring data from a source socket to a destination socket requires copying data back and forth between kernel space and user space. This operation appears simple when using abstraction layers like Go’s io.Copy, but under the hood, the Linux kernel executes a sequence of operations that degrade performance under load:

1. Context Switch 1 (User to Kernel): The application invokes Read(), triggering a trap into the kernel. The calling thread is blocked or placed in a waiting state while the OS accesses the socket’s receive buffer (RX buffer).
2. CPU Memory Copy 1 (Kernel to User): The CPU copy engine reads the network payload from the kernel RX buffer in physical memory and writes it into a user-space buffer allocated by the Go runtime.
3. Context Switch 2 (Kernel to User): The system call completes, returning execution to the application.
4. Context Switch 3 (User to Kernel): The application invokes Write(), passing the filled user-space buffer, transitioning execution back to the kernel.
5. CPU Memory Copy 2 (Kernel to User): The CPU copies the payload from the user-space buffer to the socket’s transmit buffer (TX buffer) in kernel space.
6. Context Switch 4 (Kernel to User): The system call returns, and the application resumes.

During this process, the data bytes are copied twice by the CPU. At low traffic volumes, this is negligible. However, at a throughput of 10 Gbps, the CPU spends significant cycles executing memory copies. The user-space buffers generate considerable garbage collector (GC) overhead in Go, forcing the allocator to cycle through memory segments rapidly.

Furthermore, data copying pollutes the CPU L1/L2 caches. The data read from the socket must be cached, displacing active hot data used for application logic. This cache thrashing causes unpredictable execution times, manifesting as tail latency spikes (p99/p99.9) in request processing.

Splicing and Sizing the Kernel Pipe

Linux provides splice(2) to bypass this overhead. splice(2) transfers data between two file descriptors without copying data between kernel space and user space. It shifts page table references (the underlying pages containing the data) instead of copying the actual bytes.

The API signature of the system call is:

ssize_t splice(int fd_in, off_t *off_in, int fd_out, off_t *off_out, size_t len, unsigned int flags);

Crucially, Linux enforces a design constraint on splice(2): at least one of the two file descriptors must refer to a pipe. For a TCP loopback server, this means we cannot splice a socket directly to another socket. Instead, we must route the data through an intermediate kernel pipe:

Socket RX -> Splice -> Kernel Pipe -> Splice -> Socket TX

While this introduces an intermediate step, the data remains inside kernel space. The kernel pipe is a circular buffer of memory pages managed via the kernel’s struct pipe_buffer. Writing to the pipe inserts references to the socket’s pages, and reading from the pipe extracts them.

To prevent throughput bottlenecks, we must manage the capacity of the kernel pipe. By default, a Linux pipe has a capacity of 65,536 bytes (64KB). If the sending socket pushes data faster than the receiving socket can consume it, the pipe fills up immediately, causing splice(2) to return EAGAIN or block. This results in execution stalls.

To maximize throughput, we must resize the kernel pipe using fcntl(2) with the F_SETPIPE_SZ command. Resizing the pipe to 1MB or 2MB allows it to buffer larger bursts of data, aligning it with modern TCP window sizes.

Interfacing with Go’s Netpoller via RawConn

Implementing raw system calls in Go presents a challenge: Go’s runtime manages network file descriptors using an internal asynchronous network poller (netpoller) backed by epoll(7) on Linux. If you extract a file descriptor using conn.File() and call blocking system calls on it, the descriptor is placed back into blocking mode, which spawns extra OS threads and disrupts Go’s cooperative scheduler.

To execute low-level operations safely, we must use syscall.RawConn. This interface allows us to register callbacks with Go’s network poller, enabling us to run raw system calls when the file descriptor is ready for read or write events, without blocking Go’s execution threads.

Let’s begin by extracting the raw connection control from a standard TCP connection.

Now that we can obtain the raw connection, we need to create the intermediate kernel pipe and resize its internal buffer using fcntl.

Implementing the Loopback Splice Loop

With the kernel pipe created and optimized, we can implement the zero-copy splicing loop. Splicing involves two phases:

1. Splicing data from the source socket file descriptor into the write end of the pipe (pipeWFd).
2. Splicing data from the read end of the pipe (pipeRFd) into the destination socket file descriptor.

Because we are working in non-blocking mode, either system call can fail with EAGAIN or EWOULDBLOCK if there is no data to read, or if the write buffer is full. We handle this by registering our functions with Go’s RawConn.Read and RawConn.Write interfaces. Returning false from these callbacks signals the Go netpoller to park the goroutine and wake it when the file descriptor becomes ready.

Siphoning with sendfile(2)

While splice(2) is suitable for bidirectional socket-to-socket piping, sendfile(2) is optimized for unidirectionally streaming data from a file on disk directly to a network socket.

Its API signature is:

ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

Historically, Linux allowed socket-to-socket transfers using sendfile. In modern kernels, however, in_fd must support mmap-like operations (such as a physical disk file, or a memory-backed file descriptor like memfd_create). The destination out_fd must be a socket.

When implementing a local caching server, proxying large static payloads from disk over TCP using sendfile bypasses the user-space page cache entirely. Let’s look at how to implement this using Go’s RawConn.Write.

Building a Production-Grade Zero-Copy TCP Proxy

To use these patterns in production, we need a server that listens on a local port, accepts connections, connects to a backend target, and runs the zero-copy loop bidirectionally.

The proxy must handle resource cleanup properly. If we fail to close the pipe file descriptors during connection teardown, the system will eventually leak file descriptors, leading to EMFILE errors (“Too many open files”).

Production Edge Cases, Failure Modes, and Mitigations

While zero-copy network operations improve performance, they also introduce specific failure modes that do not occur in traditional user-space network code:

1. Pipe Budget Exhaustion

If you resize pipes to 1MB (F_SETPIPE_SZ = 1,048,576) and run 20,000 concurrent proxy connections, the kernel will attempt to allocate up to 40,000 pipes (since each bidirectional proxy connection requires 2 pipes). 40,000 pipes * 1MB = 40GB of kernel memory allocated for buffer storage.

If kernel memory is exhausted, the call to fcntl will fail, or the kernel may invoke the Out-Of-Memory (OOM) killer. Additionally, Linux enforces a global pipe capacity limit in /proc/sys/fs/pipe-max-size (typically 1MB). If a non-root process attempts to exceed this value via fcntl, the call will return EPERM.

Mitigation: Implement a system fallback mechanism. If resizing the pipe fails with EPERM or ENOMEM, fall back to a smaller size (such as 64KB) or fallback to standard copying.

2. TLS Connections (Encrypted Streams)

You cannot use splice or sendfile directly on standard TLS/SSL connections. Because TLS encrypts payloads, the user-space application must decrypt the incoming stream before routing it. Splicing raw TLS packets directly to a backend server results in routing encrypted payloads that the proxy cannot modify, inspect, or manage.

Mitigation: For TLS connections, you must use standard user-space copying loops (io.Copy) unless you are utilizing Linux Kernel TLS (kTLS). kTLS offloads TLS encryption and decryption to the kernel, allowing splice and sendfile to operate on cleartext data while the kernel handles packet encryption on the wire.

3. File Descriptor Leaks

In traditional Go, GC finalizers eventually close leaked sockets or files. However, raw file descriptors created via syscall.Pipe2 are not tracked by the Go runtime allocator. If your code panics or exits early without calling syscall.Close on both ends of the pipe, the file descriptors remain open, consuming system resources until the process runs out of descriptors.

Mitigation: Always wrap pipe allocation in defensive defer statements as shown in the handleConnection method of ZeroCopyProxy. Ensure both read and write ends of the pipe are closed in all execution branches.

4. Non-Supported Filesystems

Executing sendfile or splice on unsupported filesystem layers (such as virtual network filesystems like NFS, or virtual endpoints like /proc) can result in EINVAL.

Mitigation: Ensure your application validates the incoming socket type and filesystem type. If a zero-copy syscall returns EINVAL or ENOSYS, immediately fall back to a standard copy loop.

Performance Benchmarks

To quantify performance differences, we compared three implementations on a Linux kernel 6.5 server, routing traffic through the local loopback interface:

• Approach A: Standard Go proxy utilizing io.Copy (user-space buffer copy, 32KB allocation size).
• Approach B: Custom splice proxy with default kernel pipe size (64KB).
• Approach C: Optimized splice proxy with pipe capacity resized to 1MB.

Testing was conducted using iperf3 over 30-second sustained transfers.

At peak capacity, the optimized splice implementation achieved a 3x throughput improvement over standard user-space copies while using a fraction of the CPU cycles. Because the zero-copy proxy avoids allocating application-side buffers, memory allocations in the hot path drop to zero, eliminating GC pause overhead.

To monitor these improvements in production, you can measure context switching and CPU page-fault activity using perf and strace:

# Verify system call execution and watch for context switches
strace -c -p $(pgrep my-go-proxy)

Look at the count of sys_splice vs read and write syscalls. A successful zero-copy pipeline will show high counts of splice alongside low user-space context-switching metrics. Using tools like bpftrace to track kernel cache hits and page movements can help confirm that physical memory copy operations are avoided.