Filesystem Choice for AI Servers: XFS, ZFS, ext4, and Why Btrfs Isn't on the List

If you have just spent €40k on GPUs, the filesystem you pour your data onto deserves more than five minutes of thought. The Ubuntu installer happily lays ext4 across everything; for many AI workloads that is the right answer, for some it is not. Pick the wrong filesystem for a 50 TB dataset volume and you lose throughput, lose data, or watch your DataLoader sit idle behind a metadata bottleneck.

This covers which filesystem to put where on a Kentino-class AI server (4–8 GPUs, NVMe boot, NVMe scratch, slower bulk tier). Ubuntu 22.04 / 24.04, 6.x kernel.

The short version

One-paragraph version: ext4 for the OS, XFS for hot scratch, ZFS where integrity matters more than the last 10% of throughput, no Btrfs in production.

ext4 — the boring default that is usually right

ext4 has been the default on Ubuntu, Debian, Fedora, and RHEL for over a decade. Well understood, recoverable with fsck, 16 TB per file, 1 EB per volume. Nothing exciting — the point.

Use ext4 for root (every Ubuntu live USB knows how to fix it; far fewer know how to import a zpool), for /home, and for VM image stores at small to medium scale.

Where ext4 stops being right: wide parallel I/O (its journal serialises metadata updates — twenty DataLoader workers on one ext4 volume bottleneck on the journal long before saturating NVMe); single huge files (handled, but extent allocation is less friendly than XFS for sequential >100 GB files — training shards, checkpoints, video corpora); snapshots (none natively; LVM thin snapshots work but are clunky).

mkfs.ext4 -L root /dev/nvme0n1p2

If you find yourself doing exotic ext4 tuning for an AI workload, you picked the wrong filesystem.

XFS — the right answer for hot scratch and training data

XFS was an SGI filesystem for high-throughput sequential I/O on big arrays. That heritage matches AI workloads almost perfectly: large files (training shards, parquet, webdataset tars, checkpoints), sequential reads, many parallel readers, tolerance for "the workload regenerates the data."

What XFS does that ext4 does not: allocation groups — the volume splits into 4–32 independent groups that read and write in parallel without a global journal lock, exactly what a multi-worker DataLoader wants; delayed and speculative preallocation so large sequential writes get contiguous extents; no practical file size limit (8 EB); and xfs_growfs for reliable one-liner extension after RAID growth.

What it does not: no native snapshots, no file-data checksums (metadata CRC-protected since 2014), no shrink — grow only.

A reasonable mkfs.xfs for an 8× NVMe RAID0 scratch volume, 64 KiB chunk:

# su = stripe unit = chunk size; sw = number of data disks
mkfs.xfs -f -d su=64k,sw=8 -l size=512m -L scratch /dev/md0

# largeio + swalloc: behave for >1 MB I/O
# allocsize=1g: preallocate aggressively for big sequential writes
# noatime: save a write per read under heavy DataLoader load
mount -o noatime,largeio,swalloc,allocsize=1g /dev/md0 /scratch

noatime and largeio matter most. XFS on small files is acceptable but not its strong suit — for hundreds of millions of <16 KB images, repack into webdataset / parquet / lmdb rather than swap filesystems (see below).

ZFS — the right answer when integrity, snapshots, or compression matter

ZFS is a filesystem, volume manager, and software RAID layer in one:

• End-to-end checksumming. Bit rot detected on read, corrected silently with redundancy. For a 50 TB dataset on spinning rust for two years, this matters; for ephemeral NVMe scratch over six weeks, it does not.
• Snapshots and clones. Atomic, COW, effectively free. Snapshot before a run, fork a clone, roll back — the workflow ZFS exists for.
• Built-in compression. lz4 default-on; zstd for better ratios. On text / JSON / parquet, compression usually increases effective throughput because CPU cost is less than I/O saved.
• ARC — its own RAM-based read cache, separate from the Linux page cache. The famous footgun.
• Native RAID-Z — no mdadm.

What it does not do well for AI: eats RAM (default ARC is ~50% of system memory — on a 512 GB server that is 256 GB the kernel suddenly does not have, and training jobs expecting page cache or huge pages get OOM-killed; cap the ARC); not the fastest for raw sequential reads (well-tuned XFS on NVMe RAID0 beats ZFS by 15–30% — checksumming and COW cost real cycles); ignores O_DIRECT (all I/O through ARC by design — fine for most workloads, a hard incompatibility with GPUDirect Storage).

ARC tuning — the one knob you must set

On a fresh Ubuntu + ZFS install on a 512 GB box, ZFS claims 256 GB for ARC. Set zfs_arc_max at install, not after the first OOM.

# Cap ARC at 64 GB. Bytes, not GB. 64 * 1024^3 = 68719476736
echo "options zfs zfs_arc_max=68719476736" | sudo tee /etc/modprobe.d/zfs.conf
echo 68719476736 | sudo tee /sys/module/zfs/parameters/zfs_arc_max   # apply now
sudo update-initramfs -u                                             # persist

Rule of thumb: cap ARC at 10–20% of system RAM on an AI server. The classic "2 GiB base + 1 GiB per TiB" sizing is for storage boxes; on an AI box most RAM is reserved for model weights, activations, and framework page cache.

A sane ZFS layout for an AI server data tier

# 8 × 16 TB SAS in two RAID-Z2 vdevs (6+2 each) — two vdevs give 2× IOPS
zpool create -o ashift=12 \
  -O compression=zstd -O atime=off -O xattr=sa -O recordsize=1M \
  data \
  raidz2 /dev/sd[a-h] \
  raidz2 /dev/sd[i-p]

# Datasets sized for their workload
zfs create -o recordsize=1M   data/datasets       # large training shards
zfs create -o recordsize=128k data/checkpoints    # mixed size
zfs create -o recordsize=16k  data/metadata       # small files (json, yaml)

zfs snapshot data/datasets@pre-run-2026-05-14
zfs rollback data/datasets@pre-run-2026-05-14

ashift=12 = 4 KiB blocks — get this wrong at pool creation and you cannot fix it without destroying the pool. recordsize=1M for large-file workloads (default 128 KiB is too small for multi-GB shards). xattr=sa stores xattrs inline, massively faster than the default.

Btrfs — generally avoid for production AI

Btrfs has the same conceptual feature set as ZFS — COW, snapshots, checksums, built-in RAID. On paper, right; in practice we do not recommend it on Kentino servers. RAID5/6 has known data-loss bugs the project itself flags as not production-ready (RAID1/10 are stable, parity modes are not); performance degrades badly under the fragmentation pattern AI workloads produce (gradient checkpoints, caches); snapshot performance falls off a cliff above a few hundred snapshots, which a per-experiment workflow hits within a quarter; and tooling is less mature than ZFS or XFS for replication, monitoring, and scrub scheduling. Fine on a laptop. For a multi-GPU server with tens of TB of training data, pick XFS or ZFS.

RAID layouts — what to pair the filesystem with

On 16 TB+ drives a single-disk rebuild can take 24–72 hours, and a second failure across same-batch drives in that window is not negligible. Single-parity is no longer the safe default. Use Z2/RAID6 or mirrors.

Small files versus large files

The question that traps people more than any filesystem choice: how does your filesystem behave with 50 million 4 KB images in a directory tree? Answer for every filesystem here: badly.

Per-file metadata (inode, dentry, timestamps, xattrs) runs 200–500 bytes regardless of file size — 50 million tiny files burn 10–25 GB of metadata before storing a byte of pixel data. The open/read/close pattern pegs the metadata side and stalls DataLoader workers.

The fix is not the filesystem. Do not store small files as files. Pack into webdataset / tar / parquet / lmdb / hdf5 / safetensors and stream sequentially — PyTorch's WebDataset and NVIDIA DALI both expect this. The filesystem then sees a small number of large files, which every filesystem here is good at. If you must keep individual files at scale, ZFS with recordsize=16k and xattr=sa is the least-bad option, but the real answer is to repack.

Direct I/O, page cache, and O_DIRECT

PyTorch leans on the Linux page cache for repeated dataset epochs — epoch 1 from disk, epochs 2..N from RAM. Some workloads use O_DIRECT to bypass the cache for large reads where data is touched once. NVIDIA's GPUDirect Storage goes further, DMAing NVMe → GPU directly.

ext4 and XFS honour O_DIRECT properly — aligned direct I/O, no page cache. ZFS ignores O_DIRECT (all I/O through ARC, by design): fine for most workloads, a hard incompatibility with GPUDirect Storage. Alignment matters — buffer and offset must align to device block size (typically 4 KB).

If you plan to use GPUDirect Storage to feed an 8-GPU box at full bandwidth, /data must be XFS on NVMe. If you do not know what GPUDirect Storage is and training is compute-bound, you do not need it yet.

NVMe multipath

Dual-port U.2 / E1.S NVMe present as two PCIe paths each. The Linux NVMe driver does native multipath (nvme_core.multipath=Y, default in modern kernels), giving failover and round-robin load balancing. Invisible at the filesystem layer, but it matters for failover (without it, a path/HBA failure makes the drive disappear and the filesystem panics) and bandwidth (a multipathed U.2 drive hits ~14 GB/s versus 7 GB/s single-path).

cat /sys/module/nvme_core/parameters/multipath   # expect: Y
nvme list-subsys

For consumer M.2 NVMe (5090 desktop-class builds) multipath is not relevant — one path. For 8-GPU EPYC builds with enterprise U.2 / E1.S in a dual-controller backplane, leave it on.

Checksumming overhead

On a 64-core EPYC host the cost is invisible. On a 16-core Xeon under sustained load, ZFS can steal 5–10% of effective CPU — whether that matters depends on whether training is CPU-bound on preprocessing (often yes for vision) or GPU-bound (often yes for large LLMs).

The honest framing: ZFS catches bit rot you would otherwise never notice until your model trains on subtly corrupted data. For a dataset you spent six months building, 5–10% is worth it. For ephemeral scratch you regenerate weekly, it is not.

NFS for shared datasets across cluster nodes

Once you have more than one server, where the dataset lives gets interesting. Three patterns:

1. Copy to every node. Simple, wastes capacity. Workable for <1 TB and ≤4 nodes.
2. NFS from a dedicated storage server. One canonical copy. Network is the bottleneck — 25 GbE minimum, 100 GbE for multi-node training.
3. Parallel filesystem (BeeGFS, Lustre, WekaFS). Bigger lift, scales past where NFS falls over.

For 1–4 nodes, NFS is right. Server: XFS on RAID10 NVMe, exported via NFSv4. Client: mount with nconnect=8 to parallelise across TCP connections.

# Server /etc/exports
/data/shared 10.0.10.0/24(rw,async,no_subtree_check,no_root_squash)

# Client
mount -t nfs -o vers=4.2,nconnect=8,proto=tcp,rsize=1048576,wsize=1048576 \
  storage01:/data/shared /mnt/shared

NFS over 100 GbE with nconnect=8 delivers 8–10 GB/s sustained — enough for 16 GPUs on typical vision or LLM training. Past that, BeeGFS / Lustre — a different article.

Complete example: 8-GPU EPYC, 24 NVMe + 8 SAS

2× 480 GB NVMe  (boot)     → md RAID1   → ext4 → /
8× 7.68 TB NVMe (scratch)  → md RAID0   → XFS  → /scratch
8× 7.68 TB NVMe (data)     → md RAID10  → XFS  → /data
8× 16 TB SAS    (archive)  → 2× RAID-Z2 → ZFS  → /archive

ZFS ARC: 64 GB of 512 GB RAM. NVMe multipath: on. NFS export: /data over 100 GbE.

mkfs.ext4 -L root /dev/md0
mkfs.xfs -f -d su=64k,sw=8 -l size=512m -L scratch /dev/md1
mkfs.xfs -f -d su=64k,sw=4 -l size=512m -L data    /dev/md2

zpool create -o ashift=12 \
  -O compression=zstd -O atime=off -O xattr=sa -O recordsize=1M \
  archive raidz2 /dev/sd[a-h]

echo "options zfs zfs_arc_max=68719476736" > /etc/modprobe.d/zfs.conf
update-initramfs -u

Raw NVMe throughput where you need it, redundancy where the data is irreplaceable, compressed checksummed bulk for everything else.

---

This is part of the Kentino Wiki, a reference series on AI compute, storage, and the systems that connect them. Comments and corrections welcome at info@kentino.com.