Linux Kernel Tuning for AI Servers: What Actually Moves the Needle

The default Ubuntu kernel on a 6.8 or 6.11 HWE stack is fine for about 80% of AI workloads. This article is for the other 20% — multi-socket EPYC pushing eight GPUs, a vLLM serving box hitting a vm.max_map_count wall under load, or an inference server whose tail latency is being eaten by CPU C-state transitions.

Most public "Linux tuning for AI" advice is recycled database guidance from 2015 and a non-trivial fraction of it quietly makes inference latency worse. Below is the smaller, honest set of changes that actually help on a Kentino-class K-AI box (4–8 GPUs on Xeon or EPYC, Ubuntu 22.04 / 24.04, kernel 6.x). Driver and CUDA stack assumed installed per L01 and L02.

The honest hierarchy of wins

Before any sysctl changes, anchor on the size of the prize:

If you take nothing else from this article: NUMA awareness and the CPU governor are the two big wins. Everything else is rounding error unless you have measured.

CPU governor: the cheapest 5–15% on the table

Ubuntu defaults to powersave (intel_pstate) or ondemand (acpi_cpufreq) depending on driver. Both ramp clocks reactively, which costs 5–15% on inference TTFT and on the CPU-side prep around a vLLM forward pass — tokenisation, scheduling, sampling logits.

For a serving box, set performance and forget:

sudo apt install -y cpufrequtils
echo 'GOVERNOR="performance"' | sudo tee /etc/default/cpufrequtils
sudo systemctl restart cpufrequtils

# Verify
for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do cat $c; done | sort -u

On EPYC Genoa / Turin with kernel 6.5+, the amd-pstate driver replaces acpi-cpufreq. Same idea, set scaling_governor to performance. The newer amd-pstate-epp mode keeps performance while allowing per-core boosting; leave EPP at performance, not balance_performance.

Caveat: on a training box at sustained ~100% GPU utilisation, the CPUs are loaded anyway and the governor matters less. The big win is on inference servers that idle between requests and need to ramp instantly.

NUMA: the difference between an EPYC box being fast and mediocre

Single-socket Xeon or EPYC servers have nothing to think about — one NUMA node, every memory access is local. Dual-socket boxes are where the 10–30% lives.

The default scheduler will happily place a vLLM worker on socket 0 and have it page-fault into memory allocated on socket 1 — every load is a cross-socket UPI / Infinity Fabric hop. For a model with hundreds of GB of weights walked once per token, this is real.

Inspect:

numactl --hardware                            # node count, sizes, distance matrix
nvidia-smi topo -m                            # GPU↔CPU NUMA affinity
cat /sys/class/net/<iface>/device/numa_node   # NIC NUMA affinity

The pattern you want — and that NCCL detects automatically — is GPU N pinned to the NUMA node hosting its PCIe root complex, not the other socket. For a manually launched inference process, pin both CPU and memory to the local node:

# vLLM on a 2-socket EPYC, GPUs 0–3 on NUMA node 0
numactl --cpunodebind=0 --membind=0 \
  python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 4

Two related knobs:

• kernel.numa_balancing migrates pages towards the CPU touching them. Helpful for general workloads, occasionally counterproductive for AI: a prefill walks weights once, the kernel migrates pages, the next request pulls them back. Leave on by default (1); if jitter correlates with numa_pte_updates in /proc/vmstat, try sysctl kernel.numa_balancing=0.
• NCCL on dual-socket — set NCCL_SOCKET_IFNAME to the management NIC and NCCL_IB_HCA to the NIC on the same NUMA node as the GPUs. Wrong NIC silently halves inter-node throughput.

Single-socket EPYC 9004/9005 server: ignore this whole section.

Transparent Huge Pages: madvise, not always

THP folds 4 KB pages into 2 MB pages opportunistically, reducing TLB pressure. PyTorch and the CUDA driver benefit modestly. The trap is always mode under memory churn — the kernel stalls a thread for 50–100 ms compacting memory, torpedoing any latency SLO.

The right setting is madvise: applications that know what they are doing call madvise(MADV_HUGEPAGE) on long-lived allocations and get THP; nothing else does. Set via /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="... transparent_hugepage=madvise"

update-grub && reboot. Defrag should match — defer+madvise lets compaction happen in the background rather than in the allocating thread's path:

echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

For vLLM / SGLang / TensorRT-LLM in production, madvise is the right default. PyTorch with CUDA Unified Memory (research code) sometimes benefits from always — measure before flipping.

Explicit 1 GB hugepages: only if you have measured

Explicit hugepages reserved at boot via hugetlbfs give a small additional win for very large LLM weights by eliminating TLB misses on the weight scan. The catch: memory is reserved up front and unavailable to the rest of the system, the application has to be built to use them, and the gain is 1–3% for most serving workloads.

GRUB_CMDLINE_LINUX_DEFAULT="... default_hugepagesz=1G hugepagesz=1G hugepages=64"

Worth it for a TensorRT-LLM box where the engine fits in pinned memory and you are chasing the last percent of TTFT. Not worth it for a general vLLM server that swaps models or any training box. Skip on a first build; revisit only if perf stat -e dTLB-load-misses shows TLB pressure and you have RAM headroom.

vm.max_map_count and nofile: the walls vLLM hits at scale

Default vm.max_map_count on Ubuntu is 65530 — a count of distinct memory mappings a process can hold. vLLM serving a large model with high concurrency, or any framework using lots of mmap'd safetensors shards, blows past this and dies with Cannot allocate memory. The Elasticsearch-derived value of 262144 is the bare minimum; for vLLM serving 70B+ with hundreds of concurrent sequences, push to 1048576. Costs nothing — soft limit, not a reservation.

Default ulimit -n on Ubuntu is 1024. Comically low: vLLM, Triton, PyTorch DataLoader workers, NCCL, and the gRPC layer between them open hundreds of FDs each. Hit the limit and you get EMFILE: too many open files and a process that wedges silently.

# /etc/sysctl.d/99-ai-server.conf
vm.max_map_count = 1048576

# /etc/security/limits.d/99-ai-server.conf
*    soft    nofile  1048576
*    hard    nofile  1048576

# /etc/systemd/system.conf.d/99-limits.conf
[Manager]
DefaultLimitNOFILE=1048576

sudo sysctl --system and systemctl daemon-reexec. Verify with ulimit -n and cat /proc/<pid>/limits.

IRQ affinity: pinning NIC interrupts to local CPUs

On a 100 GbE ConnectX-6/7 NIC sustaining 10+ GB/s of dataset streaming or RDMA traffic, interrupts have to land on CPUs (a) on the same NUMA node as the NIC, and (b) not the same cores running DataLoader workers. The default irqbalance does a passable job; under heavy load, it does not.

The cleanest fix is Mellanox's set_irq_affinity.sh from mlnx-tools:

sudo systemctl disable --now irqbalance
sudo /usr/sbin/set_irq_affinity.sh enp1s0f0                  # all NIC-local cores
sudo /usr/sbin/set_irq_affinity_cpulist.sh 4-11 enp1s0f0     # pin to specific cores

The wrong move is leaving irqbalance running on a box where you have also manually pinned — they fight. Pick one. Serving box with one or two RDMA NICs: pin manually, disable irqbalance. General-purpose box with many interfaces: leave irqbalance on with --banirq to exclude the critical NIC.

Network kernel buffers: only for fast storage / streaming paths

For a single-node inference server talking to clients over plain TCP at modest rates, the default net.core.rmem_max / wmem_max of 208 KB is fine. For a node pulling training data from 100 GbE NFS or object store, or a vLLM frontend behind a high-RPS load balancer, the defaults are the ceiling on your bandwidth-delay product. A starting set for a 100 GbE-attached box:

# /etc/sysctl.d/99-ai-network.conf
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.ipv4.tcp_rmem = 4096 87380 268435456
net.ipv4.tcp_wmem = 4096 65536 268435456
net.core.netdev_max_backlog = 250000
net.core.somaxconn = 65535

256 MB max is overkill for in-rack 100 GbE (BDP is ~1.25 MB) but harmless — TCP auto-tuning grows buffers only as needed up to the cap. On RDMA (RoCE / IB) for the data path these knobs do not matter — RDMA bypasses the kernel TCP stack. They still matter for the management plane, NFS, and model downloads.

C-states: latency vs idle power

For a latency-sensitive serving box, the cheapest tail-latency improvement after the CPU governor is disabling deep C-states. C6 entry/exit is in the tens of microseconds; for a request that should reply in 30 ms, a CPU coming out of C6 to handle post-token sampling adds noticeable jitter.

GRUB_CMDLINE_LINUX_DEFAULT="... intel_idle.max_cstate=1 processor.max_cstate=1"

update-grub && reboot. Verify with cpupower idle-info — only C0 and C1 should be available. Cost: 30–50 W of idle power per socket because cores never enter deep sleep. On a dual-socket 8-GPU server, 60–100 W of permanent overhead — trivial against 3.5–4.5 kW of sustained GPU draw under load.

Apply to latency-sensitive inference servers and real-time robotics paths. Skip on training boxes (latency does not matter, idle power adds up over months) and batch jobs.

Filesystem-related kernel tuning (preview to L04)

A handful of fs.* sysctls matter:

fs.aio-max-nr = 1048576              # default 65536 too low for vLLM weight loaders
fs.inotify.max_user_watches = 524288 # tooling that watches checkpoint dirs

fs.aio-max-nr is the one that bites — frameworks doing async I/O against many shards (DALI, vLLM weight loaders) burn through the default 65536 on big models. Filesystem choice itself (XFS vs ZFS vs ext4), mount options, and O_DIRECT semantics live in L04.

Disabling unneeded kernel modules

A headless serving box has no business loading modules for bluetooth, sound, webcam, wireless, joystick, or the print server. Each is more code in the attack surface and a small boot delay. Individually trivial; collectively, a cleaner box is easier to reason about.

# /etc/modprobe.d/blacklist-ai-server.conf
blacklist bluetooth
blacklist btusb
blacklist snd_hda_intel
blacklist uvcvideo
blacklist joydev

# Plus
sudo systemctl disable --now bluetooth.service cups.service avahi-daemon.service \
                              ModemManager.service whoopsie.service apport.service

Boot drops from ~30 s to ~10 s on a typical EPYC build and lsmod becomes readable. Zero inference performance impact, modest attack-surface reduction.

Two profiles: inference vs training

The tuning shape is genuinely different. Drop-in sketches:

Inference (/etc/sysctl.d/99-ai-inference.conf)

vm.max_map_count = 1048576
vm.swappiness = 1
vm.overcommit_memory = 1
kernel.numa_balancing = 1
fs.aio-max-nr = 1048576
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.ipv4.tcp_rmem = 4096 87380 268435456
net.ipv4.tcp_wmem = 4096 65536 268435456
net.core.netdev_max_backlog = 250000
net.core.somaxconn = 65535

Plus on the kernel command line: transparent_hugepage=madvise intel_idle.max_cstate=1 processor.max_cstate=1. Governor: performance. NIC IRQs pinned to NUMA-local cores, irqbalance off.

Training (/etc/sysctl.d/99-ai-training.conf)

vm.max_map_count = 1048576
vm.swappiness = 1
vm.overcommit_memory = 1
kernel.numa_balancing = 0       # page migration mid-epoch is just churn
fs.aio-max-nr = 1048576
fs.inotify.max_user_watches = 524288
net.core.rmem_max = 536870912
net.core.wmem_max = 536870912
net.ipv4.tcp_rmem = 4096 131072 536870912
net.ipv4.tcp_wmem = 4096 131072 536870912

Kernel command line: transparent_hugepage=madvise (skip C-state pinning — training is throughput-bound, idle power adds up over weeks). Governor: performance if you have the power budget. NCCL pinned to local NUMA via NCCL_IB_HCA.

The "I tuned everything and it got slower" trap

The single most common failure mode is the cargo-cult dump: copy a hundred sysctls from a blog post, reboot, observe worse performance, no idea which knob to back out. Method that works:

1. Establish a baseline — nccl-tests all_reduce_perf, vLLM benchmark_throughput.py, training step time. Write numbers down.
2. Change one thing.
3. Re-run the same benchmark, three times for noise.
4. Keep if the median is meaningfully better. Revert if not.
5. Document the change and why, in version control next to the sysctl file.

We have seen production boxes with 40 lines of sysctl tuning that were collectively 2% slower than Ubuntu defaults — every individual change neutral or worse, but the operator was certain "the tuning helps."

Red Hat's tuned 2.27 ships explicit ai-inference and ai-training profiles and is a credible shortcut on RHEL / Rocky. Ubuntu does not ship it; apt install tuned works but is less polished. On Ubuntu, hand-rolled drop-ins in /etc/sysctl.d/ and a clean GRUB command line are easier to audit and you know exactly what is set.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.