CUDA, cuDNN, and the NVIDIA Container Toolkit: A Sane Setup Path for an AI Server

You have a freshly racked server with four or eight GPUs, Ubuntu 22.04 or 24.04, the NVIDIA driver pinned (per L01), and nvidia-smi returning sensible numbers. That is the floor. Above it sits an awkward stack — CUDA, cuDNN, the container toolkit, the runtimes, the NGC images — that most guides hand-wave through. This article walks from the driver up to the point where docker run --gpus all lights up every card and a vLLM container serves tokens.

Opinionated about which path to take and honest about which one bites.

The driver / runtime / toolkit triad

Three things ship under the "CUDA" umbrella and people conflate them constantly:

The driver is what matters at runtime. A driver supports a maximum CUDA runtime version; the runtime must be that or older. The toolkit is only needed if you compile CUDA code yourself — for 95% of installs you do not need nvcc on the host.

This is why containers work: the image ships its own CUDA runtime, the host only needs the driver. The host driver is the only thing that has to be right. Everything else is portable.

┌─────────────────────────────────────────────────┐
│ Container: PyTorch 2.9 + CUDA 13.0 runtime      │  ← shipped in image
│ Container: vLLM + CUDA 13.0 runtime             │  ← shipped in image
├─────────────────────────────────────────────────┤
│ Host: NVIDIA driver 570.x                       │  ← only thing host needs
│ Host: nvidia-container-toolkit                  │  ← bridges driver → container
└─────────────────────────────────────────────────┘

This picture is the difference between an afternoon of work and three days of dependency hell.

CUDA 12.x or 13.x — which to pick

Both are in active use in May 2026. The honest matrix for the workloads Kentino servers run:

The Kentino default is CUDA 13.0 on new builds. Every GPU in the lineup (5090, 4090, RTX Pro 6000 Blackwell, L40, L4) is supported, vLLM now ships 13.0 wheels by default on PyPI and the vllm/vllm-openai image uses 13.0, PyTorch 2.11 (what vLLM ships against) is built for 13.0, and new features (FlashAttention 3 on Blackwell, FP8 training kernels) target 13.x first.

Stay on 12.8 only if: reproducing a pinned benchmark, a vendor SDK has not moved, or pre-Blackwell cards exclusively. Nothing older than 12.4 on new installs — bugs fixed upstream two years ago.

Installing the driver and CUDA cleanly

Ubuntu 24.04 LTS (22.04 is identical). The clean path is NVIDIA's CUDA apt repo:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update

# Driver + CUDA runtime libs. No nvcc — only what apps need to run.
sudo apt install -y cuda-drivers-570 cuda-runtime-13-0

# Optional full toolkit (nvcc, headers). Only if you compile on the host.
# sudo apt install -y cuda-toolkit-13-0

sudo reboot

After reboot, nvidia-smi should report driver 570.x and CUDA 13.0. Traps to avoid:

• Do not install the cuda meta-package unless you want everything. It pulls toolkit, samples, GDS, Nsight, and pins packages you did not want. Pick cuda-runtime-13-0 or cuda-toolkit-13-0 explicitly.
• Pin the driver so unattended-upgrades cannot roll the kernel module: sudo apt-mark hold cuda-drivers cuda-drivers-570 nvidia-driver-570 nvidia-dkms-570 libnvidia-compute-570.
• One driver branch at a time. Mixing 550 and 570 packages produces a system that boots but where nvidia-smi returns an obscure version-mismatch error. The fix is apt purge '*nvidia*' '*cuda*' and start over — better not to get there.

cuDNN — apt or tarball?

cuDNN is the deep-learning primitives library — convolution, attention, RNN. PyTorch / TensorFlow / JAX depend on it. As of 9.x there are two install paths.

apt (recommended):

sudo apt install -y libcudnn9-cuda-13 libcudnn9-dev-cuda-13

Installs to /usr/lib/x86_64-linux-gnu/, gets picked up by the dynamic linker, and integrates with the package manager so security updates flow through. Only one CUDA-version of cuDNN 9 can be installed at a time — no libcudnn9-cuda-12 and libcudnn9-cuda-13 side-by-side via apt.

Tarball:

# Fetch from the redist manifest at developer.download.nvidia.com/compute/cudnn/redist/
tar -xf cudnn-linux-x86_64-9.5.0.x_cuda13-archive.tar.xz
sudo cp -r cudnn-linux-x86_64-*/include/* /usr/local/cuda/include/
sudo cp -r cudnn-linux-x86_64-*/lib/* /usr/local/cuda/lib64/

Tarball makes sense for multiple cuDNN versions on one host, air-gapped bundles, or micro-version pinning. Everything else: apt. NVIDIA's docs recommend distribution packages where possible — the tarball is not an installer, it is a redistribution archive.

Honest answer: if you are using NGC containers, do not install cuDNN on the host at all. It ships in the image.

Containers vs bare-metal — when to do which

The single biggest decision in the stack, with no universal answer. The trade-off as seen on real Kentino builds:

Containerized vs bare-metal CUDA performance is negligible on modern kernels — single-digit percent at worst, usually noise. Anyone claiming otherwise is benchmarking storage or network, not GPU compute.

Bare-metal: chasing the last 2–3% for a paper, kernel debugging with cuda-gdb / compute-sanitizer / Nsight where the container layer is in the way, or one process owning the machine for years.

Containers (the Kentino default): multi-user boxes, PyTorch 2.5 and 2.11 side-by-side, swapping inference frameworks (vLLM, SGLang, TensorRT-LLM) without rebuilding the host, or a deployment you can docker pull onto a new server and have running in five minutes.

For a robotics lab or research group: containers. For a benchmark rig with one workload, bare-metal is fine.

nvidia-container-toolkit — what it is and how to set it up

The toolkit is the glue that gives a container access to the host's GPU. Without it, nvidia-smi inside a container fails. With it, the container sees the driver, the devices, and an injected runtime stub.

Install:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Smoke test
docker run --rm --gpus all nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi

That last command should show the host's nvidia-smi output from inside the container. If it does, the stack is wired.

Podman

Podman uses CDI:

sudo nvidia-ctk runtime configure --runtime=podman
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

podman run --rm --device=nvidia.com/gpu=all \
  nvcr.io/nvidia/pytorch:25.12-py3 nvidia-smi

Rootless Podman + CDI is the cleanest setup for multi-user boxes where you do not want users in the docker group (which is effectively root).

CDI vs legacy runtime

Two modes for exposing GPUs:

• Legacy nvidia runtime hook — what --gpus all has used for years. Docker invokes a hook that mounts driver libraries and devices into the container at runtime.
• CDI (Container Device Interface) — an open spec for declaring device access. nvidia-ctk cdi generate writes a YAML file enumerating GPUs as named devices (nvidia.com/gpu=0, nvidia.com/gpu=all); any CDI-aware runtime consumes them.

Status mid-2026:

For a single-server Docker install, legacy still works fine. For anything that touches Kubernetes, rootless, or multi-user, switch to CDI — it is where NVIDIA's investment is going. Migration is non-invasive: install the toolkit, generate the CDI spec, swap --gpus all for --device=nvidia.com/gpu=all. The two modes can coexist.

GPU passthrough — what --gpus all actually does

When you run docker run --gpus all my-image the runtime hook reads NVIDIA_VISIBLE_DEVICES / NVIDIA_DRIVER_CAPABILITIES, mounts /dev/nvidia* into the container with cgroup permissions, bind-mounts the host's driver .so files in, and sets LD_LIBRARY_PATH so the linker finds them.

What you control:

docker run --gpus all ...                                     # all GPUs
docker run --gpus '"device=0,1"' ...                          # by index
docker run --gpus '"device=GPU-abc123..."' ...                # by UUID
docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility ...
docker run --device=nvidia.com/gpu=0 --device=nvidia.com/gpu=1 ...  # CDI

For an 8-GPU server doing tensor-parallel vLLM, use --gpus all. For a multi-tenant box where users get 4-GPU slices, pin by UUID — index-based pinning breaks if you re-cable.

NGC containers — when to use them

NVIDIA's NGC catalog (nvcr.io) ships images for PyTorch, TensorRT, TensorRT-LLM, Triton, and others. Large (5–15 GB) but tested and shipped with cuDNN / NCCL / CUDA runtime combos NVIDIA has QA'd. May 2026:

Tags are <year>.<month>-py3, cut monthly. Not exact upstream — patched and rebuilt — but tracks closely.

Honest case for NGC: if you run TensorRT-LLM or Triton, use the NGC image. The work of matching cuDNN, NCCL, TensorRT, and CUDA is done. For upstream PyTorch or vLLM, the public images (pytorch/pytorch:..., vllm/vllm-openai:...) are smaller and faster to update — NGC is overkill.

MIG — and why it does not apply to Kentino's lineup

MIG partitions a single GPU into up to seven isolated slices, each with its own memory, compute, and fault domain. Useful for multi-tenant inference — five users get 1/7th of an H100 with hard isolation.

MIG is not available on consumer (RTX 5090, 4090) or workstation Blackwell (RTX Pro 6000) cards. It is restricted to datacenter SKUs — H100, H200, A100, B100, B200, GB200. Kentino does not source SXM-form-factor datacenter GPUs, so MIG is not in any Kentino build.

The substitute on consumer / workstation cards is MPS (Multi-Process Service) — multiple processes share the GPU compute scheduler. Not isolated like MIG (one bad process can OOM-kill the device) but for trusted multi-tenant inference it works and costs nothing. If you genuinely need MIG, you need the datacenter SKU line through a different integrator.

Troubleshooting — the errors that bite everyone

CUDA error: no kernel image is available for execution on the device (error 209). Container's CUDA runtime has no kernels for your GPU's compute capability — usually a CUDA 12.4 image on a Blackwell card (sm_120) that needs 12.8+. Fix: newer image, or build with the right TORCH_CUDA_ARCH_LIST.

CUDA error: forward compatibility was attempted on non supported HW (error 100). Host driver is older than the runtime in the container wants. Fix: upgrade the host driver — and install from NVIDIA's repo, not Ubuntu's stale packaged version.

Failed to initialize NVML: Driver/library version mismatch. Driver upgraded via apt without reboot; kernel module is old, userspace is new. Fix: reboot. (If you cannot, rmmod the nvidia modules and modprobe them back — but reboot is safer.)

nvidia-container-cli: initialization error: nvml error: driver not loaded. Toolkit installed but the driver kernel module is not loaded. Usually a fresh install where nvidia-smi on the host already fails. Fix the host, then the container.

OCI runtime exec failed: ... no such file or directory on --gpus all. Runtime hook is not registered with Docker. Re-run sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker. If not fixed, check /etc/docker/daemon.json for a "runtimes" block.

Building from source vs prebuilt

PyTorch, vLLM, llama.cpp, FlashAttention — all have prebuilt wheels and images. Build from source only when you need an unreleased feature, are on a compute capability upstream wheels do not target, or compile against custom CUDA / cuDNN. A from-source PyTorch is 30–90 minutes on EPYC with zero perf gain unless you pass custom CMake flags. Source builds are the answer to a specific problem, not the default.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.