Storage Tiers in an AI Server: Model, Dataset, Scratch, Checkpoint

There is a recurring conversation we have with buyers about NVMe. They ask for "as fast as possible" because the spec sheets show 14 GB/s on a Gen5 drive and a 4090 next to a Samsung 9100 PRO sounds matched. Then we ask what the box is for, the answer is "inference, mostly", and the right build turns out to be one drive instead of eight. Storage in an AI server is the part of the spec where over-buying is easy, painless to the vendor, and almost completely invisible to the workload.

This article covers the four real storage tiers in a single AI server, what each one actually has to do, what the 2026 NVMe market looks like, when Gen5 is worth its premium, why hardware RAID stopped mattering for NVMe, and how to size each tier without burning money. Filesystem choice is a separate question handled in L04; cluster-shared storage (NFS, BeeGFS, Lustre, object stores) is K04. Here we stay inside the chassis.

The four storage tiers

A serious AI server has four distinct storage roles. They have wildly different size, throughput, endurance, and cost profiles. Treating them as one pool is the first mistake.

A 4-GPU inference server typically needs only the first tier and maybe a sliver of the second. An 8-GPU training box needs all four, and the four want different drives. The question "how many TB of NVMe?" is meaningless without picking a tier.

Model storage

The model tier is what holds your library of weights — every Llama, Qwen, Mistral, Stable Diffusion, Flux, Whisper variant you ever expect to load. It is also where the OS, container images, conda envs, and inference server binaries live. For most builds these collapse onto the boot drive.

Sizing is a function of how many models you keep resident. Some examples at common quantizations:

• Llama 3 70B Q4: ~40 GB
• Llama 3 70B FP8: ~70 GB
• Qwen2.5-VL 72B Q4: ~45 GB plus a few GB of vision tower
• Mistral Large 2 (123B) Q4: ~70 GB
• Llama 4 Maverick MoE FP8: ~250 GB
• A modest VLM library (8 models, mixed quantizations): 300–500 GB
• A research lab keeping 30 models hot: 2–4 TB

Read-mostly. The model loads once at process startup and lives in VRAM for the rest of the day. The drive sees a 40 GB sequential read at vLLM startup, then largely idles. A single 2–4 TB Gen4 NVMe — even a consumer drive like a Samsung 990 PRO — handles this without breathing hard. Pay for capacity, not throughput.

Dataset storage

Dataset is where training corpora and fine-tuning material sit. This tier is the one customers chronically under-spec, because the dataset they are working with today fits and they forget that the next one will not.

Realistic ranges:

• Single-domain LoRA fine-tune corpus: 10–100 GB
• Multimodal fine-tuning set (images + text): 1–10 TB
• Pretraining-scale text corpus (FineWeb subset, etc.): 5–50 TB
• Video pretraining or robotics demonstration set: 20–200 TB
• Genomic, scientific, or scraped-web archive: 100 TB – 1 PB+

The access pattern is sequential read, repeated across epochs, across many DataLoader workers in parallel. Throughput-sensitive (because the GPU is waiting), latency-tolerant. Write rate is essentially zero once the data is staged.

For datasets above ~10 TB the right answer usually leaves the chassis — see K04 for NFS, BeeGFS, and object-store patterns. Inside the chassis, a dataset tier of HDDs in RAIDZ2 (cheap, large, slow but adequate after page cache warms) or a mid-tier U.2 NVMe RAID is reasonable up to maybe 50 TB.

Scratch

Scratch is the unsung hero. It absorbs everything that does not fit cleanly elsewhere: decoded image shards, intermediate activations spilled by ZeRO-Offload, dataset caches built by webdataset/DALI, temporary dataset shuffles, container build artifacts, and the dump of a crashed run. Read-heavy and write-heavy at the same time.

Sizing: 2–8 TB is the sweet spot. Less than 2 TB and a single fine-tune fills it. More than 8 TB and you are usually building a small dataset tier and calling it scratch — fine, but admit what you are doing.

Endurance matters here in a way it does not for the other tiers. Scratch eats writes — 5–20 TB written per day is not unusual on a heavily-used training box. A 1 DWPD enterprise drive at 4 TB allows 4 TB written per day for five years; a serious training scratch wants 1–3 DWPD, mixed-use grade.

Checkpoint

The checkpoint tier is where training state lands. Bursty: silent for ten minutes, then 50–500 GB written in a few seconds, then silent again.

A 70B model checkpoint at FP16 is ~140 GB of weights. With FSDP, gradients and optimizer state push the per-checkpoint footprint to 400–700 GB. With DeepSpeed ZeRO-3 and full optimizer state (Adam: 8 bytes per parameter for the master FP32 + first/second moment), a 70B checkpoint can land near 1 TB.

Sustained write rate of a single Gen5 NVMe is ~10–13 GB/s. A 700 GB checkpoint writes in roughly a minute, optimistically — if every rank streams to its own drive. Synchronously, training blocks for that minute. The fix is async sharded checkpointing (covered below).

NVMe Gen4 vs Gen5 — when the premium is worth it

The 2026 NVMe market is split between mature Gen4 (5–7 GB/s) and now-mainstream Gen5 (12–14 GB/s). Top of the consumer pile sits the Samsung 9100 PRO at 14.8 GB/s sequential read, 13.4 GB/s sequential write. Enterprise Gen5 is in the same neighborhood — Solidigm D7-PS1010 at 14.5 GB/s read, Micron 9550 MAX at 14 GB/s read, Kioxia CD8, SanDisk DC SN861 — all delivering 13–14 GB/s read and 8–10 GB/s sustained write.

Cost premium for Gen5 over comparable-capacity Gen4 in mid-2026 is 30–50% at the consumer end and 20–35% at the enterprise end.

When Gen5 actually pays back:

• Model loading from cold. A 250 GB MoE model loads in 18 seconds on Gen5 vs 36 seconds on Gen4. Matters if you swap models often.
• Training I/O bound. Vision pipelines decoding raw JPEGs from disk, multi-modal datasets where preprocessing is the bottleneck.
• Checkpoint to local NVMe before async copy. The faster the local write, the sooner training resumes.
• Cluster storage server backing 8+ training nodes. Aggregated client demand exceeds Gen4 ceiling.

When Gen4 is fine and Gen5 is wasted money:

• Inference servers. Model loads once at startup; the rest of the day the drive is at single-digit MB/s.
• OS root and container store. Boot once a month, install packages occasionally.
• Dataset tier where shards are pre-decoded and large. PyTorch DataLoader with 8 workers reading WebDataset shards saturates at maybe 4–6 GB/s aggregate; Gen4 covers it.

The honest version: about 70% of customer builds we ship are pure inference, and for those Gen4 NVMe across the board is correct. Gen5 is a training-tier decision.

Form factors — M.2, U.2/U.3, E3.S

M.2 Gen5 will throttle. A Samsung 9100 PRO 2 TB sustained at 13 GB/s in a hot chassis hits 75 °C and clocks back. Heatsink-equipped models help. M.2 is fine for boot and modest workloads; it is not the right form factor for a sustained-write scratch tier.

U.2 is the workhorse. Every enterprise NVMe drive Kentino ships in a Bone64c or Supermicro chassis is U.2, almost universally interchangeable with U.3 backplanes. 25 W thermal envelope, dual-port for multipath, hot-swappable, 7.68 TB and 15.36 TB capacities sit at the price-density sweet spot.

E3.S is the future, slowly. Designed from scratch for Gen5/Gen6 thermals. For a single-server or small-cluster Kentino build, U.2/U.3 remains the practical choice through 2026; E3.S enters the picture for refresh cycles in 2027.

Avoid stacking M.2 carriers in a hot chassis. PCIe-card adapters that hold four M.2 drives behind a switch chip work, but in a 4U server with eight GPUs the airflow over those carriers is poor. We see thermal throttling within ten minutes of sustained load. If you need four NVMe in a server, use U.2.

Endurance — DWPD as a buying signal

For an AI server, the mapping is straightforward:

• Model tier: read-intensive, 0.3–1 DWPD.
• Dataset tier: read-intensive, 0.3–1 DWPD.
• Scratch tier: mixed-use, 1–3 DWPD. This is the tier that earns its endurance budget.
• Checkpoint tier: mixed-use, 1–3 DWPD. Bursty writes accumulate; 1 DWPD on a 4 TB drive only allows 4 TB/day, which a heavy training cadence exceeds.

Write-intensive (3+ DWPD) is overkill for almost every AI workload. Buying the wrong endurance tier is recoverable by replacement; buying the wrong throughput class often is not. Get throughput right at build time, treat endurance as something you can upgrade.

RAID for NVMe — hardware is dead, software is the answer

This deserves its own subsection because customers still ask. The short version: if your build has a hardware RAID controller in front of NVMe drives in 2026, it is misconfigured.

Hardware RAID controllers (LSI/Broadcom MegaRAID, Microchip SmartRAID, HPE Smart Array, Dell PERC) were designed for SAS/SATA. The controller's onboard ASIC handles parity, caching, and command queuing, and at SAS bandwidth (12 Gbit/s = 1.5 GB/s) it has plenty of headroom.

NVMe Gen5 at 14 GB/s per drive saturates a single-card RAID controller after two drives. Eight drives behind a hardware RAID card produce more bandwidth than the card's PCIe uplink can move. The controller becomes the bottleneck, then the failure point.

The practical answers are software:

• mdadm (Linux software RAID). Mature, in-kernel, supports RAID0/1/10 well. RAID5/6 on NVMe works but the parity calculation eats CPU. For 4 NVMe in mirror or stripe: perfect.
• ZFS RAIDZ2. Better integrity story (end-to-end checksums), better tooling for scrubs and replacement, snapshot/clone semantics. CPU-heavier than mdadm. The right choice for archive and dataset tiers where data integrity matters more than maximum throughput.
• Per-drive replication / erasure coding at the application layer. For very large arrays, MinIO/Ceph EC or BeeGFS striping move the redundancy decision out of the block layer entirely.

For most Kentino builds:

• Boot: 2× M.2 NVMe in mdadm RAID1.
• Scratch: RAID0 across 2–4 NVMe (data is regenerable).
• Dataset/checkpoint: RAID10 in mdadm or RAIDZ2 in ZFS.

Hardware RAID has exactly one remaining use case in an AI server: a small boot mirror behind a basic Broadcom/MegaRAID card on a server where you absolutely want pre-OS bootable redundancy. Even there, mdadm RAID1 with EFI on both drives works fine.

Network-attached storage for the dataset tier

The takeaway: the dataset tier almost always belongs outside the AI server above 10 TB. Inside the chassis, you want fast scratch, fast checkpoint, and a model store; the dataset can be remote.

How PyTorch actually loads data

A useful model for sizing storage is to follow what PyTorch's DataLoader does. With num_workers=8 and pin_memory=True:

1. Eight worker processes each open dataset files in parallel.
2. Each worker reads, decodes (JPEG, video frames, audio resample), and tensorizes a sample.
3. The decoded tensor is placed in shared memory.
4. The main process pulls from the shared queue, pins to host RAM, and copies to GPU via DMA.

The practical implications:

• A vision pipeline is rarely storage-bound unless the dataset is so large the page cache cannot hold a working set.
• An LLM pretraining pipeline reading pre-tokenized shards is often storage-bound on Gen4. Gen5 helps. So does putting the active shard set on local NVMe instead of NFS.
• Inference is essentially never storage-bound after model load. The disk activity meter on a serving box is flat.

The checkpoint cliff

A 70B model checkpoint at FP16 is 140 GB of weights. Add Adam optimizer state (~8 bytes per parameter for FP32 master + first + second moment): ~560 GB more. Add gradients if you snapshot mid-step: ~140 GB more. A "complete" 70B checkpoint can hit 800 GB to 1 TB depending on what you save.

Synchronous, single-rank-writes-everything checkpoint to a single Gen5 NVMe at 13 GB/s sustained write: ~75 seconds for a 1 TB checkpoint. Training blocks for the entire duration. Doing this every 100 steps when each step takes 5 seconds means you have just made the training run 15% slower.

Two fixes, both essential:

1. Sharded checkpointing. Each FSDP rank writes its own shard to its own local NVMe. With 8 GPUs and 8 local drives, the per-rank write is 125 GB and writes parallelize. Wall-clock drops to ~10 seconds.

2. Async checkpointing. PyTorch DCP (torch.distributed.checkpoint) offloads the write to a CPU thread. GPU briefly stalls for the GPU→CPU staging copy (a few seconds), then training resumes and the actual write to storage happens in the background.

The combination — async sharded — turns checkpoint storage from "must absorb 1 TB in seconds" into "must keep up with steady-state background copy of a few hundred MB/s". Almost any storage tier handles that. The mistake to avoid: synchronous, full-state-per-rank checkpoints written directly to NFS. That was normal in 2022. It is malpractice in 2026.

Concrete K-AI storage layouts

4-GPU inference server (e.g. K-AI 192 Genoa, 4× RTX 5090 or 4× RTX Pro 6000 Blackwell):

Boot + model:    1× 2 TB Gen5 M.2 NVMe (consumer or read-intensive enterprise)
Scratch:         optional, often unnecessary for pure inference
Dataset:         not needed on the box
Checkpoint:      not needed on the box

Total: 2 TB NVMe. Cost: low. Sufficient for any inference workload.

8-GPU training server (e.g. K-AI 256 Turin Dual, 8× RTX 5090 or 8× RTX Pro 6000):

Boot:            2× 480 GB M.2 NVMe, mdadm RAID1, ext4 → /
Model storage:   2× 4 TB U.2 Gen5 NVMe, mdadm RAID1, ext4 → /models
Scratch:         2× 8 TB U.2 Gen5 NVMe, mdadm RAID0, XFS → /scratch
Checkpoint:      2× 8 TB U.2 Gen5 NVMe, mdadm RAID10, XFS → /checkpoints
Dataset:         NFS mount from external storage server, 100 GbE

Total local NVMe: ~36 TB. Cost: meaningful but matched to GPU spend.

Cluster shared storage node (separate from compute):

Boot:            2× 960 GB M.2 NVMe, RAID1, ext4
Hot tier:        12× 7.68 TB U.2 Gen5 NVMe, ZFS RAIDZ2 or BeeGFS storage target, XFS
Cold tier:       8× 16 TB SAS HDD, ZFS RAIDZ2 → /archive
Network:         2× 100 GbE, RoCE or NVMe-oF capable

Serves the cluster's dataset tier. See K04 for distributed FS choice.

Two things to notice. The inference server has one drive. The training server has four tiers but each drive is sized for its role, not over-spec'd to be the largest available. The cluster storage node is a separate machine entirely.

Honest take — most customers over-spec storage

Pattern we see often enough to warrant calling out: a customer builds an 8-GPU server "to be future-proof", specs eight 15.36 TB U.2 NVMe drives because the slots are there, then runs inference workloads that touch maybe 200 GB of model files and idle the rest of the array forever. €25,000 of NVMe sitting cold.

The honest sizing rules:

• Pure inference, single server: 2–4 TB total NVMe. One drive. Done.
• Inference plus occasional fine-tuning: 4–8 TB total. Boot + scratch.
• Serious training, single server: 16–40 TB across boot/model/scratch/checkpoint, with the dataset external on a NAS or object store.
• Cluster: dataset and checkpoint move to shared storage; per-node local NVMe shrinks back to the inference profile (2–8 TB) for boot, model, and write-back scratch.

Storage is the only place in an AI build where the "more is always better" instinct is consistently wrong. GPUs scale benefit roughly linearly with cost; RAM scales reasonably; storage often does not scale at all once you have enough.

What breaks

The failure modes we see in the field, in order:

• Boot drive is consumer M.2 with no redundancy. Drive fails 18 months in, server unbootable, models stay safe on RAID but the rebuild is a day. Fix: 2× M.2 in mdadm RAID1, even on inference boxes.
• Scratch fills mid-run. Dataset cache, decoded shards, framework temp files conspire. Fix: monitor, alarm at 80% full, size 1.5× the largest expected working set.
• Hardware RAID card behind NVMe. Performance is half what it should be, controller is single point of failure, replacement is exotic. Fix: rip out the card, mdadm or ZFS direct.
• Synchronous checkpoint to NFS, training stalls. Fix: async sharded checkpoint, local NVMe first, then async copy.
• Gen5 M.2 thermal throttle. Drive rated 14 GB/s does 4 GB/s after 3 minutes of sustained write. Fix: heatsink, better airflow, or move to U.2.
• Dataset on dataset tier saturates Gen4. Vision training pipeline is 30% storage-bound. Fix: Gen5 scratch, pre-decode to WebDataset, or both.
• ZFS ARC eats RAM intended for training. Default 50% allocation OOMs the training process. Fix: cap zfs_arc_max to 10–20% of RAM. See L04.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.