Ubuntu Pinning and NVIDIA Driver Management for AI Servers

If you are responsible for an AI server that other people depend on, the most likely cause of your next outage is not a hardware failure, a model bug, or a power event. It is an apt-get upgrade that you or your distribution ran in the background, which pulled in a new kernel, which the NVIDIA DKMS module did not rebuild against cleanly, and which left nvidia-smi returning Failed to initialize NVML: Driver/library version mismatch on the next reboot. That is the single most common Kentino support call after a server has been running for two or three months.

This article is about how to make that not happen. It covers which Ubuntu LTS to pick in mid-2026, how to install the NVIDIA driver in a way you can actually reason about, how to pin the driver and the kernel so that an unattended upgrade cannot quietly destroy your stack, and how to recover when somebody has done it anyway.

The audience is whoever is going to type sudo on a 4-GPU or 8-GPU box that is supposed to keep working for years.

Why Ubuntu LTS at all

There are technically capable Linux distributions that are not Ubuntu — Debian stable, RHEL, Rocky, AlmaLinux, openSUSE Leap. None of them are wrong, and a few of them are arguably better engineered. We still recommend Ubuntu LTS for AI servers, for four reasons that have nothing to do with elegance.

NVIDIA tests against Ubuntu LTS first. Their apt repositories ship cuda-keyring packages explicitly for ubuntu2204, ubuntu2404, and (since spring 2026) ubuntu2604. The driver installation guide names Ubuntu LTS by version number; it names other distributions in the back of the document.

The container ecosystem assumes Ubuntu. Every NGC image (nvcr.io/nvidia/pytorch, nvcr.io/nvidia/tritonserver, TensorRT-LLM) is built on an Ubuntu LTS base. The host distribution does not have to match the container, but when it does, kernel and userspace assumptions line up cleanly.

Most third-party AI tooling assumes Ubuntu in its README. vLLM, llama.cpp, SGLang, FlashAttention build instructions, the Hugging Face Accelerate examples — all written and tested against Ubuntu first. RHEL works; you will be the one filing the issues when it does not.

The pool of people who have hit your problem before is largest on Ubuntu. When nvidia-smi returns nonsense at 02:00, the first three Stack Overflow hits will be Ubuntu-specific. That matters more than it should.

Debian and RHEL are not bad choices for an AI server. They are smaller test surfaces, slower NVIDIA validation cycles, and a smaller community when something goes sideways. If you have a strong organisational reason for one of them — air-gapped FedRAMP, an existing RHEL site licence, a Debian-shop sysadmin team — use it. If you do not, default to Ubuntu LTS.

22.04 vs 24.04 vs 26.04 in May 2026

Three LTS releases are in active support today. The honest matrix:

A few things worth pulling out of that table.

22.04 is still the largest install base globally, and it is what NVIDIA's validation pipeline has the most history on. If you have a working 22.04 fleet, do not panic-upgrade. It is fully supported until April 2027 with standard updates, and Ubuntu Pro extends security maintenance to 2032 if you need it. But if you are racking a new server in mid-2026, you are picking your OS for the next four to five years. 22.04 has only one year of standard support left. Do not start there.

24.04 is the sweet spot for production AI work in 2026. It has had two years of bug-fix and HWE kernel revisions, every NVIDIA driver branch from R535 forward has had production-quality packages in NVIDIA's ubuntu2404 repository, every NGC image lists 24.04 as a supported base, and you have until April 2029 before standard support runs out. This is what we deploy on new Kentino builds today.

26.04 is fresh out the door. It ships R595 as the default driver in the official Ubuntu repository, includes CUDA in the main repos for the first time, and has good Wayland / NVIDIA story improvements that matter for desktops and not at all for headless inference servers. The risk for an AI server is timing: in the first three to four months after a .0 LTS release, ubuntu-drivers recommendations and DKMS module behaviour move around, and NVIDIA's own repository takes a few cycles to fully validate every driver branch against the new kernel. The pattern from past LTS releases (20.04, 22.04, 24.04) is that the first point release — 26.04.1 in August 2026 — is when production deployments start to make sense. If you can wait, wait. If you cannot, run a soak test for two weeks before putting anything important on it.

The driver install paths

There are three ways to put the NVIDIA driver on an Ubuntu box. They are not equally good.

ubuntu-drivers install is the easy path. Canonical maintains a tool that introspects your hardware, picks a driver branch it considers recommended, and installs the matching package from the Ubuntu archive. It handles Secure Boot signing for you (using Canonical's key), it knows about HWE kernels, and on a desktop with a single consumer GPU it works fine. On an AI server it is the brittle path: the recommended branch can change between releases of the ubuntu-drivers-common package, the version you get on apt update depends on what Ubuntu has packaged this week, and you have less control over precisely which driver minor version is on disk. Use it for the laptop. Do not use it for the server.

apt from NVIDIA's CUDA repository is the manual but controllable path. You add NVIDIA's cuda-keyring package, which sets up the right apt source for your Ubuntu version, and then install exactly the packages you want — cuda-drivers-580, nvidia-driver-580, nvidia-dkms-580, no surprises. NVIDIA controls the packaging, you control the version. This is the Kentino default.

The .run installer from developer.nvidia.com writes files to /usr/local, manages its own kernel module rebuilds, and does not integrate with apt at all. It is the right answer in two narrow cases: brand-new hardware where the driver branch you need has not made it into the apt repository yet, or a workstation where you are actively developing or debugging the driver. Otherwise it is the wrong answer. It will fight DKMS, it will not be picked up by apt-mark hold, and it will quietly break on the next kernel update because the rebuild path is now manual.

Concretely, on a 24.04 server with a fresh install, the Kentino-default sequence is:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update

# Install the driver branch you actually want, by number.
sudo apt install -y cuda-drivers-580 nvidia-dkms-580
sudo reboot

After reboot, nvidia-smi should report driver 580.x and the maximum CUDA version the driver supports (CUDA 13 for R580). That is the whole installation. Notice what is not in there: no cuda meta-package, no cuda-toolkit-13-0 (only needed if you compile CUDA code on the host — see L02), no ubuntu-drivers. You picked the version, you installed the version, you know what is on disk.

The driver branches that matter in 2026

NVIDIA ships its Linux drivers in branches with specific support windows. As of mid-2026:

R580 is the right default in 2026. It supports every GPU in the Kentino lineup (5090, 4090, RTX Pro 6000 Blackwell both editions, L40, L4) including the Blackwell architecture's sm_120 compute capability, and it ships native CUDA 13 support — which is what every recent vLLM, PyTorch 2.10/2.11, and TensorRT-LLM release targets. R570 is the conservative LTSB choice if you have a fleet of Ada (4090 / L40 / L4) that does not need the newest features and you want to skip a driver-branch transition for another year. R595 is what Ubuntu 26.04 ships by default and what you want for the very newest hardware; on a server that runs only what is in the Kentino lineup today, R580 is more conservative and equally featured.

The driver / CUDA / PyTorch tri-version dance

Three independent versions have to line up. They sit on top of each other like this:

PyTorch wheel (e.g. torch==2.11.0+cu130)
        │  ships its own CUDA runtime, cuDNN, NCCL inside the wheel
        ▼
CUDA runtime version (here: 13.0)
        │  the driver must support this CUDA version or newer
        ▼
NVIDIA driver branch (here: R580 → supports CUDA 13.x)

The rule is that the driver supports a maximum CUDA runtime version. Anything older than that maximum, on the same major version, will run. So R580 (max CUDA 13.x) runs PyTorch wheels built against 13.0, 13.1, 13.2; it does not run wheels built against the (hypothetical) CUDA 14.0 until you upgrade the driver branch. Older drivers cannot run newer CUDA — there is forward compatibility for some datacentre cards, but it is fragile and not how you want to operate.

Practical map of what Kentino servers run today:

Your job is to fix the driver and let everything above it follow. You do not need to install CUDA on the host at all if you serve everything from containers (and you should — see L02). The host needs the driver and the container toolkit. That is it.

Pinning — the single most valuable command in this article

Once the driver is in and nvidia-smi is happy, pin it. Pinning means telling apt not to upgrade, downgrade, or remove specific packages, even if a dependency or a security update would otherwise touch them. The command is apt-mark hold.

# Pin the driver branch.
sudo apt-mark hold \
  cuda-drivers \
  cuda-drivers-580 \
  nvidia-driver-580 \
  nvidia-driver-580-server \
  nvidia-dkms-580 \
  libnvidia-compute-580 \
  libnvidia-compute-580-server

# Pin the running kernel and its headers — DKMS needs both to rebuild,
# and a kernel jump that DKMS does not handle is exactly how you lose the GPU.
RUNNING=$(uname -r)
sudo apt-mark hold "linux-image-${RUNNING}" "linux-headers-${RUNNING}"

# If you are on the Ubuntu HWE kernel meta-package, hold that too:
sudo apt-mark hold linux-generic-hwe-24.04 linux-image-generic-hwe-24.04 linux-headers-generic-hwe-24.04

# Verify.
apt-mark showhold

That is roughly half of every "my AI server stopped working" call solved. When apt update && apt upgrade runs — manually, from a configuration management tool, or under unattended-upgrades — those held packages will not move. You will see them listed under "kept back" in the upgrade output, and you can deliberately review and unhold them at a maintenance window when you are ready.

A few traps. People hold nvidia-driver-580 and forget cuda-drivers-580, which is the actual meta-package on the CUDA repo. If cuda-drivers-580 upgrades because a transitive dependency wants a newer point release, it will pull a new nvidia-dkms-580 with it, and you have lost the pin. List both. People hold the kernel image but not the headers, the kernel updates anyway because of a meta-package, and DKMS fails to find headers when it tries to rebuild — nvidia-smi works until reboot, then does not. Always hold image and headers together.

The kernel and DKMS gotchas

DKMS (Dynamic Kernel Module Support) is the system that rebuilds out-of-tree kernel modules whenever the kernel changes. The NVIDIA nvidia-dkms-XXX package registers the driver as a DKMS module so that, in theory, an apt-managed kernel upgrade triggers a clean module rebuild and you reboot into a working system.

In practice DKMS rebuilds fail in three ways that recur:

The kernel headers are not installed for the new kernel — common when the kernel package was installed but the headers package was filtered out by an apt preferences pin or a custom selection. The new kernel boots, the NVIDIA module is missing, nvidia-smi returns "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver."

The new kernel is too new for the driver branch. R570 does not build cleanly against the latest 6.14 kernel; R580 does. If you let an HWE bump push you onto a kernel the driver branch was never tested against, DKMS produces a build error in /var/lib/dkms/nvidia/.../make.log and you reboot into a working but GPU-less system.

Secure Boot is enabled and the rebuilt module is unsigned. Canonical's ubuntu-drivers-installed packages come with Canonical's signing handled. NVIDIA's CUDA-repo packages do not; they require you to enroll a Machine Owner Key (MOK) and sign the rebuilt modules at every kernel change. DKMS can do the signing automatically with the right per-module config, but it has to be set up before the first rebuild, not after the system fails to boot. If you do not need Secure Boot — and on a headless inference server in a locked rack you arguably do not — disable it in firmware before installing the driver and skip this entire class of problem.

Pinning the kernel side-steps all three. The kernel does not change without your involvement, the headers stay matched, the rebuilt module continues to load. When you do want to take a new kernel — for a CVE or a real feature — you do it deliberately, with a snapshot, ready to roll back.

Bare-metal CUDA — when (rarely) it is still right

L02 makes the case that bare-metal CUDA is mostly the wrong choice now: containers hide the CUDA-version dance, decouple the host from the workload, and impose negligible performance cost. That is correct for ~95% of installs.

The exceptions are narrow. Kernel-level GPU profiling with Nsight Systems / Nsight Compute, where the container layer hides the syscalls you care about. Driver work — running pre-release drivers from NVIDIA, debugging crashes that involve the kernel module itself. Tightly air-gapped environments where pulling and trusting OCI images is harder than maintaining a frozen apt mirror. A handful of HPC sites that have built everything around module load and bare-metal MPI and are not interested in changing.

For everyone else: install the driver bare-metal, leave the host clean of CUDA, and run CUDA inside containers. The nvidia-container-toolkit (covered in L02) bridges the two cleanly.

Unattended upgrades — what to allow and what to block

Ubuntu's unattended-upgrades package is good and you should leave it on. Skipping security updates entirely on a network-connected server is worse than the risk of an automated upgrade going wrong. The right pattern is: security updates yes, kernel and NVIDIA packages no.

Edit /etc/apt/apt.conf.d/50unattended-upgrades and add to the blacklist block:

Unattended-Upgrade::Package-Blacklist {
    "linux-image-";
    "linux-headers-";
    "linux-generic";
    "linux-modules-";
    "nvidia-";
    "libnvidia-";
    "cuda";
    "cuda-";
    "libcudnn";
};

apt-mark hold already prevents these packages from upgrading. The blacklist is belt-and-braces — it makes the intent explicit, it stops unattended-upgrades from even logging an attempt, and it survives a careless apt-mark unhold from a junior admin. Run both.

Confirm by tailing /var/log/unattended-upgrades/unattended-upgrades.log after the next run. You should see security packages going through and the held / blacklisted ones being skipped explicitly.

The "I dist-upgraded and now nothing works" recovery playbook

Somebody ran apt full-upgrade (or your CM tool did) and on the next reboot the GPUs are gone. Order of operations:

1. Boot. If the system does not boot, hold shift / Esc at the GRUB prompt and pick the previous kernel from the boot menu. Confirm nvidia-smi works under the old kernel. If it does, the new kernel is the problem; pin the old one (apt-mark hold linux-image-<old>) and continue.
2. If the system booted but nvidia-smi says NVML mismatch, you are running new userspace against an old loaded module. Reboot. Almost always fixes it.
3. If nvidia-smi says the driver is not loaded, check /var/lib/dkms/nvidia/<version>/build/make.log and dmesg | grep -i nvidia. The most common failures: missing kernel headers (apt install linux-headers-$(uname -r) then dkms autoinstall); a kernel too new for the driver (downgrade the kernel or upgrade the driver branch deliberately); Secure Boot rejecting the unsigned module (mokutil --sb-state, then either disable SB or enroll a MOK).
4. If the driver packages themselves were dist-upgraded into a state you did not want — for instance, the meta-package pulled R585 over your pinned R580 because somebody removed the hold — purge and reinstall: sudo apt purge '*nvidia*' '*cuda*' && sudo apt autoremove, then redo the install from the CUDA repo at the version you actually want, then re-apply the holds.
5. If you have a snapshot — and you should, see below — restore it. This is twenty minutes back to known-good versus four hours of debugging.

Snapshot before you touch the driver or the kernel

Whatever your storage layout, take a snapshot before any driver or kernel operation. ZFS users (/ on ZFS or a separate dataset) get this for free with zfs snapshot rpool@pre-driver-upgrade-2026-05-15. LVM users with thin pools can lvcreate --snapshot. ext4 users without LVM are stuck with full backups; this is one of the reasons we cover filesystem layout in L04.

The discipline that pays off: never touch the driver, the CUDA packages, or the kernel without taking a snapshot in the previous five minutes. The cost of a snapshot is seconds. The cost of not having one when an upgrade goes sideways is a half-day of downtime and somebody's evening.

LTS upgrade path

Ubuntu LTS-to-LTS upgrades happen every two years; for AI servers we recommend skipping every other one. The path is:

22.04  ──(skip 23.x interim, skip 24.04 if production-stable)──▶  26.04
24.04  ──(skip 25.x interim, skip 26.04 if production-stable)──▶  28.04

The reasoning: the cost of an LTS upgrade on an AI server is high (driver re-validation, container image re-test, often a re-run of integration benchmarks), the benefit of going one step rather than two is small, and Ubuntu's LTS-to-LTS upgrade tooling fully supports the skip. Plan the upgrade, schedule a maintenance window, snapshot, run do-release-upgrade, soak-test for a week, then move workloads back. Going 22.04 → 26.04 directly is supported and is what most Kentino customers will end up doing late in 2026 or in 2027.

What to avoid: in-place upgrades during a development sprint, upgrading without a snapshot, upgrading and then immediately dist-upgrading the NVIDIA stack at the same time. Change one thing at a time.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.