Linux io_uring Deep Dive: Submission Queues, Completion Rings, and Zero-Copy Networking

At 100k+ connections, the Linux epoll model starts showing cracks. Not because poll itself is slow — it isn’t — but because everything surrounding it is: the read() and write() syscalls that follow each event, the user/kernel boundary crossings per I/O operation, the buffer copies on every receive. A busy proxy server doing 500k req/s can spend north of 30% of CPU time just on syscall overhead and buffer management. io_uring, introduced in Linux 5.1 and matured through 5.10–6.x, is the first serious attempt to fix this at the kernel interface level rather than papering over it in userspace.

The Core Abstraction: Two Lock-Free Ring Buffers

io_uring exposes two shared memory rings between user space and the kernel. The Submission Queue (SQ) is where your application drops io_uring_sqe structs describing work to be done. The Completion Queue (CQ) is where the kernel drops io_uring_cqe structs describing results. Both rings are mapped into user space via mmap — there is no copy, no message passing, no socket involved in communicating between the two sides. The rings are just memory.

Each SQE is 64 bytes and carries an opcode (IORING_OP_READ, IORING_OP_SEND, IORING_OP_ACCEPT, etc.), a file descriptor, a buffer pointer, a length, an offset, flags, and a user_data field you use to correlate completions to submissions. CQEs are 16 bytes: a result code, flags, and the echoed user_data. The index relationship between SQ entries and the SQE array is intentionally indirect — the SQ ring holds indices into a separately mmap’d SQE array, which lets you manipulate SQEs in-place without disturbing the ring structure.

// snippet-1
// Setting up an io_uring instance with SQPOLL and fixed buffers
#include <liburing.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>

#define QUEUE_DEPTH     4096
#define FIXED_BUF_COUNT 256
#define FIXED_BUF_SIZE  65536  // 64KB per buffer

struct io_uring ring;
struct iovec fixed_bufs[FIXED_BUF_COUNT];
void *buf_pool;

int setup_uring(void) {
    struct io_uring_params params = {0};

    // SQPOLL: kernel thread polls submission queue — zero syscall on hot path
    params.flags = IORING_SETUP_SQPOLL;
    params.sq_thread_idle = 2000;  // park kernel thread after 2s idle (ms)

    if (io_uring_queue_init_params(QUEUE_DEPTH, &ring, &params) < 0)
        return -1;

    // Pre-allocate and register fixed buffers: kernel pins these pages once
    buf_pool = mmap(NULL, FIXED_BUF_COUNT * FIXED_BUF_SIZE,
                    PROT_READ | PROT_WRITE,
                    MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0);
    if (buf_pool == MAP_FAILED)
        return -1;

    for (int i = 0; i < FIXED_BUF_COUNT; i++) {
        fixed_bufs[i].iov_base = (char *)buf_pool + i * FIXED_BUF_SIZE;
        fixed_bufs[i].iov_len  = FIXED_BUF_SIZE;
    }

    // io_uring_register pins pages into kernel memory once — subsequent
    // reads/writes using buf_index skip the get_user_pages() call entirely
    return io_uring_register_buffers(&ring, fixed_bufs, FIXED_BUF_COUNT);
}

★ Insight ───────────────────────────────────── The IORING_SETUP_SQPOLL flag turns submission into a pure memory write — your process writes to the SQ ring and a kernel thread sees it without any io_uring_enter() syscall. The kernel thread parks after sq_thread_idle milliseconds of inactivity (configurable), so you pay one syscall to wake it up on cold start but zero on steady-state hot paths. At 500k rps, eliminating syscalls entirely can recover 15–20% CPU. ─────────────────────────────────────────────────

The Submission Path in Detail

With SQPOLL disabled (the default), submission works like this: get an SQE pointer from the ring, fill it, advance the SQ tail, call io_uring_enter() to notify the kernel. With SQPOLL enabled, io_uring_enter() is only needed if the kernel thread has parked (the ring exposes a IORING_SQ_NEED_WAKEUP flag for exactly this check). In both modes, batch submissions are first-class: you can push 64 SQEs into the ring and issue a single io_uring_enter(64, 0, 0), amortizing the syscall cost across all of them.

SQE chaining (IOSQE_IO_LINK) lets you express dependencies: a chained sequence executes serially within the batch, but multiple independent chains execute in parallel. This is the primitive you’d use to implement request pipelining or staged I/O without callbacks.

// snippet-2
// Batched accept + recv chain using fixed buffers
void submit_accept_and_recv(struct io_uring *ring, int server_fd, int buf_idx) {
    struct io_uring_sqe *accept_sqe = io_uring_get_sqe(ring);
    struct io_uring_sqe *recv_sqe   = io_uring_get_sqe(ring);

    struct sockaddr_storage client_addr;
    socklen_t client_len = sizeof(client_addr);

    // Accept with IOSQE_IO_LINK: recv only fires after accept completes
    io_uring_prep_accept(accept_sqe, server_fd,
                         (struct sockaddr *)&client_addr, &client_len, 0);
    accept_sqe->flags    = IOSQE_IO_LINK;
    accept_sqe->user_data = 0xACCEPT_TAG;

    // IORING_OP_READ_FIXED uses pre-registered buffer — no get_user_pages()
    io_uring_prep_read_fixed(recv_sqe,
        /* fd will be filled from accept result via IOSQE_IO_HARDLINK in 5.17+,
           here we use a fd placeholder and fixup on CQE */
        -1,
        fixed_bufs[buf_idx].iov_base,
        FIXED_BUF_SIZE,
        0,          // offset
        buf_idx);   // buf_index into registered buffer table
    recv_sqe->user_data = (uint64_t)buf_idx | (0xRECV_TAG << 32);

    // One syscall submits both; kernel handles sequencing via the link
    io_uring_submit(ring);
}

Completion Harvesting and the CQE Loop

The completion side is simpler: the kernel writes CQEs to the ring and advances the CQ tail. Your application reads from the CQ head and advances it when done. The kernel never needs to notify you — you just poll the ring. For low-latency servers, a tight CQE polling loop with io_uring_peek_cqe() (non-blocking) beats io_uring_wait_cqe() (blocking) at the cost of a hot CPU core.

// snippet-3
// Production CQE drain loop with latency histogram
void process_completions(struct io_uring *ring, struct latency_histo *histo) {
    struct io_uring_cqe *cqe;
    unsigned head;
    unsigned count = 0;

    // io_uring_for_each_cqe iterates without advancing the head
    io_uring_for_each_cqe(ring, head, cqe) {
        uint64_t tag      = cqe->user_data >> 32;
        uint32_t buf_idx  = (uint32_t)cqe->user_data;
        int      res      = cqe->res;

        if (res < 0) {
            // res is negated errno: -ECONNRESET, -EPIPE, etc.
            handle_error(tag, buf_idx, -res);
        } else {
            switch (tag) {
                case ACCEPT_TAG:
                    on_accept(res);          // res = new fd
                    break;
                case RECV_TAG:
                    on_recv(buf_idx, res);   // res = bytes read
                    break;
                case SEND_TAG:
                    release_buf(buf_idx);    // recycle fixed buffer
                    break;
            }
        }
        count++;
    }

    // Advance head by all consumed entries in one shot
    io_uring_cq_advance(ring, count);

    // Optional: track batch size for back-pressure tuning
    latency_histo_record(histo, count);
}

★ Insight ───────────────────────────────────── io_uring_cq_advance(ring, count) batches the head advance instead of calling io_uring_cqe_seen() per entry. This matters: the CQ head is a shared atomic that the kernel reads to determine available ring space. Batching the advance reduces false sharing on the cache line that holds the head pointer — measurable at >200k CQEs/sec. The io_uring_for_each_cqe macro generates a plain memory-order loop; no locks, no atomics on the read path. ─────────────────────────────────────────────────

Zero-Copy Networking with Fixed Buffers and IORING_OP_SEND_ZC

Standard send() copies data from user space into a kernel socket buffer. At 10Gbps line rate with 1400-byte packets, that’s ~900k copies/sec per core. io_uring offers two mechanisms to eliminate this.

Fixed buffers (demonstrated above) eliminate the get_user_pages() overhead inside kernel I/O paths by pre-pinning pages at registration time. The pages stay pinned for the lifetime of the ring — no TLB shootdowns per operation. This is not zero-copy in the network sense (data still copies into the socket buffer), but it eliminates the page table walk overhead and can cut latency by 5–10μs per operation on NUMA systems.

IORING_OP_SEND_ZC (Linux 6.0+, io_uring 5.19) is the real zero-copy send. It passes the buffer’s physical pages directly to the NIC DMA engine, bypassing the socket buffer entirely. The catch: your buffer is in use by the NIC until a notification CQE arrives (separate from the submit CQE). You must not touch the buffer until you receive the IORING_CQE_F_NOTIF flag on a subsequent CQE.

// snippet-4
// Zero-copy send with IORING_OP_SEND_ZC (Linux 6.0+)
// Two CQEs per send: immediate (IORING_CQE_F_MORE) + completion notif
void submit_send_zc(struct io_uring *ring, int fd, int buf_idx, size_t len) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(ring);

    // IORING_OP_SEND_ZC: kernel pins buffer pages, DMA from user memory
    io_uring_prep_send_zc_fixed(sqe, fd,
                                fixed_bufs[buf_idx].iov_base,
                                len,
                                0,        // flags
                                0,        // zc_flags
                                buf_idx); // registered buffer index
    sqe->user_data = (uint64_t)buf_idx | (0xSEND_ZC_TAG << 32);

    io_uring_submit(ring);
    // You will receive TWO CQEs:
    //   1. res >= 0, flags & IORING_CQE_F_MORE  -> send accepted, buffer still in use
    //   2. res == 0, flags & IORING_CQE_F_NOTIF -> NIC done, buffer safe to reuse
}

// In the CQE handler:
void handle_send_zc_cqe(struct io_uring_cqe *cqe, buf_pool_t *pool) {
    uint32_t buf_idx = (uint32_t)cqe->user_data;

    if (cqe->flags & IORING_CQE_F_MORE) {
        // First CQE: send submitted, DO NOT recycle buf yet
        return;
    }
    if (cqe->flags & IORING_CQE_F_NOTIF) {
        // Second CQE: NIC finished DMA, safe to recycle
        buf_pool_release(pool, buf_idx);
    }
}

SQPOLL, CPU Affinity, and the NUMA Trap

SQPOLL’s kernel thread is pinned to a CPU by default but can be bound with IORING_SETUP_SQ_AFF and params.sq_thread_cpu. On multi-socket systems, getting this wrong is catastrophic: a SQPOLL thread on socket 0 accessing buffers allocated on socket 1 pays 80–120ns of remote NUMA latency per operation instead of 40–60ns local. At 1M I/O ops/sec, that’s 80ms/sec of pure latency tax.

Practical rule: allocate your fixed buffers and your io_uring instance on the same NUMA node, and bind the SQPOLL thread to a CPU on that node. Use numactl --membind for buffer allocation and IORING_SETUP_SQ_AFF for thread binding.

# snippet-5
# Verify io_uring SQPOLL thread NUMA locality
# Check which CPU the sq_thread is pinned to
SERVER_PID=$(pgrep -f my-uring-server)

# io_uring spawns io_uring-sq kernel threads per ring
for tid in $(ls /proc/${SERVER_PID}/task/); do
    comm=$(cat /proc/${SERVER_PID}/task/${tid}/comm 2>/dev/null)
    if echo "$comm" | grep -q "iou-sqp"; then
        cpu=$(cat /proc/${SERVER_PID}/task/${tid}/status | grep Cpus_allowed_list)
        numa=$(cat /sys/bus/cpu/devices/cpu$(taskset -cp $tid 2>/dev/null | \
               awk -F: '{print $2}' | tr -d ' ')/topology/physical_package_id 2>/dev/null)
        echo "SQPOLL thread $tid: $cpu (NUMA node $numa)"
    fi
done

# Check buffer allocation locality with numastat
numastat -p ${SERVER_PID}

Real-World Performance Numbers

Benchmarks run on a bare-metal server (Intel Xeon Gold 6338, 2x32 cores, 100GbE Mellanox ConnectX-6), kernel 6.6, comparing an echo server implementation across I/O models:

The jump from epoll to basic io_uring comes mostly from batching. The jump from basic io_uring to SQPOLL is the syscall elimination. Fixed buffers and SEND_ZC stack on top with diminishing returns — you’re eventually bottlenecked by PCIe DMA bandwidth and NIC queue depth.

Ecosystem Maturity: What’s Actually Production-Ready

liburing (maintained by Jens Axboe, the io_uring author) is the only userspace library you should use. It handles ring setup, memory barriers, and all the platform-specific atomics. Version 2.5+ is stable for production.

Tokio (Rust async runtime) has io_uring support via tokio-uring crate, though the main runtime still defaults to epoll for compatibility reasons. For new Rust network services, monoio (ByteDance) is built entirely on io_uring from the ground up and shows 40% better throughput than Tokio on their internal benchmarks.

nginx added preliminary io_uring support in 1.25.x for static file serving. It uses IORING_OP_SENDFILE to replace sendfile(2) and shows 15–20% improvement on file-heavy workloads.

Failure modes to know: the IORING_SETUP_SQPOLL thread is a privileged kernel thread — on containers without CAP_SYS_NICE, it requires explicit seccomp/apparmor policy whitelisting. Several Kubernetes distributions block io_uring by default due to CVE-2022-29582 and related privilege escalation bugs in the 5.10–5.15 window. Verify your kernel version is ≥6.1 and your container runtime policy explicitly permits io_uring_setup.

// snippet-6
// Runtime capability check before using SQPOLL
#include <sys/capability.h>
#include <liburing.h>
#include <errno.h>

int can_use_sqpoll(void) {
    // SQPOLL requires CAP_SYS_NICE (or running as root) on kernel < 5.12
    // On 5.12+ it's allowed for unprivileged users with RLIMIT_NICE >= 0
    struct io_uring_params probe = {0};
    probe.flags = IORING_SETUP_SQPOLL;

    struct io_uring ring;
    int ret = io_uring_queue_init_params(1, &ring, &probe);
    if (ret == -EPERM) {
        // Fall back to standard submission (still faster than epoll via batching)
        return 0;
    }
    if (ret == 0) {
        io_uring_queue_exit(&ring);
        return 1;
    }
    return 0;  // other error, don't use SQPOLL
}

// Check io_uring feature availability via IORING_REGISTER_PROBE
int probe_features(struct io_uring *ring) {
    struct io_uring_probe *probe = io_uring_get_probe_ring(ring);
    if (!probe) return -1;

    int has_send_zc  = io_uring_opcode_supported(probe, IORING_OP_SEND_ZC);
    int has_fixed_fd = io_uring_opcode_supported(probe, IORING_OP_FIXED_FD_INSTALL);

    printf("SEND_ZC: %s, FIXED_FD: %s\n",
           has_send_zc  ? "yes" : "no (need kernel 6.0+)",
           has_fixed_fd ? "yes" : "no (need kernel 6.2+)");

    free(probe);
    return 0;
}

★ Insight ───────────────────────────────────── io_uring_get_probe_ring() issues IORING_REGISTER_PROBE to enumerate which opcodes the running kernel actually supports — critical for code that needs to run across kernel versions without crashing on unrecognized opcodes. This is much more reliable than parsing /proc/version or uname -r, because distribution kernels backport selectively. Always probe at runtime rather than compile-time version gating. ─────────────────────────────────────────────────

Putting It Together: A Minimal Production Pattern

The practical io_uring server loop follows a prepare → submit → drain pattern that looks nothing like the epoll state machine most backend engineers learned:

// snippet-7
// Minimal production event loop skeleton
void run_server(int server_fd) {
    setup_uring();  // see snippet-1
    arm_accept(server_fd);

    while (1) {
        // 1. Drain completions (non-blocking peek first)
        unsigned ready = io_uring_cq_ready(&ring);
        if (ready > 0) {
            process_completions(&ring, &histo);
        }

        // 2. Prepare next batch of work
        //    (re-arm accepts, queue pending sends, etc.)
        prepare_next_batch(&ring);

        // 3. Submit — with SQPOLL this may be a no-op if kernel thread is awake
        unsigned submitted = io_uring_submit(&ring);

        // 4. If nothing ready and nothing submitted: wait on CQ
        //    io_uring_wait_cqe_timeout with 1ms avoids spinning at idle
        if (ready == 0 && submitted == 0) {
            struct io_uring_cqe *cqe;
            struct __kernel_timespec ts = { .tv_sec = 0, .tv_nsec = 1000000 };
            io_uring_wait_cqe_timeout(&ring, &cqe, &ts);
        }
    }
}

The key insight is that io_uring is not a replacement for your event loop — it is your event loop. There’s no epoll fd to poll, no separate readiness notification mechanism. The CQ ring is the notification primitive. You batch submissions, drain completions, and repeat. The syscall boundary exists at the edges of the busy period, not in the middle of every I/O.

When io_uring Is the Wrong Tool

io_uring is not universally better. For services that do one I/O per request (simple proxy, single-query database shim), the batching advantage disappears. For services that are CPU-bound rather than I/O-bound, you’re adding complexity with no headroom to recover. The SQPOLL thread permanently consumes a CPU core even at idle — unacceptable if you’re running 50 microservices on a 4-core box. And in environments where kernel version is fixed (legacy RHEL 8.x ships 4.18, which predates io_uring entirely), none of this applies.

The target is clear: io_uring pays off when you have high-throughput, latency-sensitive I/O, control over your kernel version (≥6.1 recommended), and can dedicate resources to the SQPOLL thread. Network proxies, message brokers, database storage engines, and high-frequency trading feed handlers are the natural habitat. Use it there. Keep epoll everywhere else.