Reference Build: A Robotics + AI Lab in One Rack

The articles before this one describe the architecture, the software stack, the power budget, and the network design. This one puts a price tag and a parts list on all of it — drawn from real hardware Kentino has shipped and benchmarked, not a theoretical BOM assembled from a marketing sheet.

The deployment target is a research or integration lab running one to four robots alongside on-prem inference and a training runway. Mid-scale: not a hobbyist box that can barely serve a 7B, not a hyperscale cluster that requires a facilities team. A serious working environment that a team of two to six people can stand up, operate, and evolve without dedicated DevOps.

The article covers two tiers of the same architectural pattern. The 4-GPU development tier is where most labs start. The 8-GPU production tier is where they land after the first real workload. The hardware choices are different; the architecture is identical.

What the fleet data shows

Kentino has deployed a fleet of uniformly configured 4-GPU compute nodes. Each unit carries the same specification: AMD EPYC 7542 (32 cores / 64 threads, 2.9 GHz boost), 512 GB DDR4 ECC LRDIMM at 2666 MT/s across eight DIMMs, four RTX 4090 GPUs (24 GB VRAM each, 96 GB total), a 2 TB NVMe scratch drive, and a 512 GB SATA SSD for the OS. Motherboard is the ASRockRack ROMED8-2T/BCM. Ubuntu 24.04 LTS, kernel 6.8.0-generic, NVIDIA driver 590.48.01, CUDA 13.1/13.2.

All ten nodes produce identical benchmark numbers. That is not an accident — it is the point of a reference build. When every unit leaves the same config, the benchmark is a specification, not a lucky run.

GPU compute

FP16 tensor-core matmul across the four cards (8192 × 8192 workload):

GPU 1 runs about 6% lower than GPUs 0, 2, and 3 — within the normal silicon-lottery spread for consumer cards. All four passed 60-second GPU-burn and combined GPU+CPU burn with zero computation errors. Temperature headroom is comfortable: the thermal throttle threshold for RTX 4090 is 83 °C, and the hottest card in the fleet peaked at 73 °C under simultaneous GPU and CPU load. No throttling was observed on any node.

A practical note on power limits: the fleet ships with per-card limits set slightly differently (450 W, 480 W, 500 W depending on slot). For sustained production use, normalising all four to 450 W smooths the aggregate draw and reduces PSU imbalance. The TFLOPS penalty at 450 W vs 500 W is under 2% — not worth the asymmetric rail load.

vLLM inference — Llama 3.3 70B AWQ

Benchmarked with vLLM 0.19.0, tensor_parallel_size=4, gpu_memory_utilization=0.80, max_model_len=2048:

One important caveat: the benchmark used the awq quantization kernel, not awq_marlin. The awq_marlin path on Ada Lovelace (RTX 4090) is 2–3× faster for single-request latency. Production deployments should use awq_marlin or the FP8 path under vLLM 0.20+, which closes the gap further. With awq_marlin, the single-request number above moves from ~8 tok/s to approximately 16–20 tok/s on these cards — more consistent with what users experience in practice. The aggregate batch throughput benefit is proportionally smaller because the bottleneck at high concurrency shifts to memory bandwidth rather than compute.

These numbers are for Llama 3.3 70B at INT4 (AWQ). A 7B or 8B model at Q4 runs 4–6× faster per token on the same hardware; a 13B model runs roughly 2.5× faster. The 70B number is the relevant sizing constraint for VLM and large planning models in a robotics context.

llama.cpp — single-user, no concurrency

Benchmarked with llama-cpp-python 0.3.20 (CUDA/cuBLAS), Llama 3.3 70B Instruct Q4_K_M GGUF, all four GPUs:

llama.cpp loads faster (10.8 s vs 95 s for vLLM) and delivers higher single-user decode speed on these cards because it does not pay the continuous-batching scheduler overhead. The trade is zero concurrency — one user at a time. The right read of these numbers: llama.cpp on a 4-GPU node gives a developer a responsive 70B model for interactive use; vLLM is the right choice the moment you have more than one client.

See I02 for the full serving-stack decision matrix. The short version: start with vLLM in Docker, use awq_marlin quantization, and switch to llama.cpp only for single-user dev boxes or Jetson deployments.

GPU memory bandwidth

Device-to-device (on-card) bandwidth is consistent at 920.2–920.6 GB/s across all four cards — within 0.04% of each other, confirming GDDR6X is not the variable here. PCIe Host→Device bandwidth lands at 26.2–26.3 GB/s, confirming Gen4 x16 is fully negotiated under load (idle link-state shows Gen1 due to ASPM power saving — normal, not a fault). GPU-to-GPU peer transfers over PCIe (no NVLink) measure 19–22 GB/s, which is the expected ceiling for peer-to-peer over a shared PCIe root complex without NVLink (see N03 for context).

NVMe storage

Sequential bandwidth is near the Gen4 x4 ceiling. Random IOPS at queue depth 32 saturates well: 568K read IOPS is well above what LLM weight loading or dataset access patterns demand at single-node scale. Model load from NVMe to VRAM takes 10–30 seconds depending on model size; this drive is not the bottleneck.

The reference build

BOM — 4-GPU development tier

The development tier is the right starting point for a lab running one or two robots, one or two active models, and occasional LoRA fine-tuning on the side.

Development tier: 4-GPU compute node + managed ToR switch + 6 GHz AP + 5 kVA online UPS.

BOM — 8-GPU production tier

The production tier doubles GPU count. The architectural pattern is the same; the CPU, RAM, and chassis scale accordingly to support eight full-bandwidth PCIe slots. An AMD EPYC Genoa or Turin platform provides the PCIe lane budget for 8× x16 slots without bifurcation compromise (see W02).

Note: the 25 GbE NIC add-on is only viable on 8-GPU chassis. The 4-GPU chassis does not have the slot clearance for an additional NIC card — the 10 GbE onboard ports on the ROMED8-2T/BCM are the practical limit on the 4-GPU tier (see N01 and the product page rules).

Storage beyond the compute node

For the dev tier, a 4-bay NAS with 4× 16 TB HDDs gives 48 TB usable at RAID-5 — enough for a year of episode data for a 2-robot lab. Connect it at 10 GbE to the ToR switch; do not put it on the same interface as the robot traffic.

Power and cooling for the rack

From I04:

The fleet stress-test data confirms the 4-GPU node draws ~1,914 W at full GPU load with all four cards hitting their power limits. Add CPU (~120 W sustained), NVMe, and fan overhead, and the node lands at approximately 2.1–2.2 kW sustained.

Electrical install for the dev tier: a 400 V 16 A three-phase feed is the minimum; 32 A three-phase gives the room to grow. PSU cords on two separate phases. See I04 and P02 for the wiring layout. For the production tier (10 kW full rack), you need a 400 V 32 A three-phase feed with at least 15 kW of cooling capacity.

UPS sizing: size for the server only, not the robots. For the dev tier, a 5 kVA double-conversion online UPS is the minimum sane choice. For the production tier, 10 kVA. Always specify online double-conversion. See P05.

Software stack on bare metal

Software stack: pinned Ubuntu + NVIDIA driver → container toolkit → vLLM + llama.cpp + nginx + monitoring.

Driver pinning is non-negotiable. The fleet runs driver 590.48.01; apt-mark hold nvidia-driver-590 is the first thing after bare-metal OS install. An unattended apt-get upgrade that pulls a newer driver has broken production inference exactly as many times as you would expect. See L01 for the full procedure.

Monitoring is not optional. The fleet data shows GPU 3 on one node had correctable PCIe AER errors (RxErr + BadTLP) during benchmarking — auto-corrected by hardware, but visible in DCGM counters. Without monitoring, this surfaces as "occasional weird inference errors" weeks later. Wire up DCGM and set an alert on correctable ECC errors above a baseline rate. See L05 for the Prometheus + Grafana stack.

Sizing variations

The 4-GPU development tier — what it can actually run

96 GB total VRAM (4× 24 GB) with TP=4:

The 72B VLM does not fit comfortably on 4× 24 GB at INT4 — the math is close, but context windows beyond 4K push it over. Either use the 32B VLM variant, or step up to the 8-GPU tier or a 4× RTX Pro 6000 Blackwell node (96 GB VRAM each) for the 72B class. See W07 for the VRAM vs. model tier discussion.

The dev tier handles: one 70B inference model serving 1–3 concurrent robot clients, one VLM (32B class) for visual queries, and occasional LoRA fine-tuning on a smaller base model. That covers 80% of robotics lab use cases in 2026.

The 8-GPU production tier — the extra headroom

192 GB total VRAM (8× 24 GB):

The 8-GPU tier also opens the door to simultaneous multi-model serving: a 70B LLM on 4 GPUs and a 32B VLM on the other 4, serving a mixed robot fleet with different model requirements at the same time, from one box. On the 4-GPU tier this requires a model-swap step with 30–120 s downtime per swap.

Note on TP=8 for a 70B: TP=8 over PCIe peer-to-peer (no NVLink) scales worse than TP=4 × PP=2 at concurrency above 8 because the all-reduce communication cost grows with tensor parallel degree. The benchmark data shows GPU-to-GPU PCIe bandwidth at 19–22 GB/s — enough for TP=4, but TP=8 all-reduce at high batch sizes saturates those links. The recommended config for a 70B on 8× 4090 is TP=4 × PP=2, as shown in I02.

Rack layout

The compute node always sits below the switch and UPS — front-to-back airflow requires the cold air to enter from the front of the rack (cold aisle or room AC supply), pass through the server, and exit into the hot aisle at the rear. See W05 and I04 for the airflow discipline.

The jump host is a 1U or small-form-factor machine with a display port breakout, used for initial bringup, driver installation, and maintenance when the primary server is unreachable over the network. It does not need a GPU. A used workstation or small 1U with 16 GB RAM and a Gen3 SSD is fine.

What this build is not

Not a desktop GPU server in a rack case. The open-air desktop tower build — GPUs mounted on a riser card, no case, fans pointed at the ceiling — is not what is described here. Desktop builds are fine for development; they are not rated for sustained 24/7 operation and do not provide the directed front-to-back airflow that keeps GPUs at steady operating temperature over weeks.

Not an NVLink cluster. RTX 4090 (consumer/workstation class) does not have NVLink. GPU-to-GPU transfers go over PCIe at 19–22 GB/s, not 900 GB/s. This is sufficient for TP=4 inference serving, and it is a real constraint for TP=8 training. If your workload is heavy distributed training on a 70B+ model, RTX Pro 6000 Blackwell (NVLink-capable in its multi-GPU edition) or a platform with NVSwitch changes the math — but also the price. See N03.

Not redundant PSUs. The dual-PSU configuration is split power delivery: PSU A feeds GPUs 0 and 1 (plus motherboard), PSU B feeds GPUs 2 and 3. If PSU A fails, you lose two GPUs and the motherboard. Nothing automatically fails over. This is honest dual-PSU — it balances rail load and reduces single-point risk vs. a single large PSU, but it is not N+1 redundancy. See W04.

Not a cloud alternative for everything. If your lab's primary workload is a single model call ten times a day, the TCO math does not justify on-prem. This build pays back against cloud API spend when you are running sustained inference for multiple clients, fine-tuning regularly, or operating under data-residency constraints. See T02 for the detailed math.

Why these specific choices

AMD EPYC 7542 as the CPU host. A 32-core EPYC provides 128 PCIe 4.0 lanes from the CPU alone, enough for 4× GPUs at x16 without bifurcation, plus a NIC, storage, and a PCIe switch for expansion. EPYC is the standard platform for multi-GPU AI servers at this scale for a reason: Intel Xeon at the same core count provides fewer PCIe lanes, which forces bifurcation compromises on 4+ GPU configs. See W02 for the lane count arithmetic.

512 GB system RAM. The fleet ships with 512 GB (8× 64 GB), which exceeds the nominal 256 GB spec. This is intentional: large model loading during container startup, dataset prefetching for training, and the operating-system page cache all consume RAM that the VRAM cannot. For vLLM with KV-cache swap space enabled, system RAM is directly usable as overflow — a 4 GB swap space per GPU requires 16 GB of system RAM headroom. 512 GB provides that headroom with room for Isaac Sim, which can hold 10–30 GB of simulation state in system RAM during policy training.

2 TB NVMe scratch. The benchmark shows 4,589 MB/s sequential read — fast enough to load a 70B FP8 model (~75 GB) in under 20 seconds. For a lab that swaps models frequently (development workflow, A/B testing inference stacks), load time matters. A slower SATA SSD doubles or triples that time per swap.

Front-to-back rack airflow. The most common mistake on first-time builds: desktop orientation (side intake, top exhaust, rear exhaust) does not work in a sealed rack. The compute node must use directed airflow matching the rack's cold-aisle / hot-aisle convention. See I01 and W05.

The honest take on 2026 hardware

This is a reference build for May 2026. The RTX 4090 is not the newest Kentino GPU option — the RTX 5090 (32 GB VRAM, Blackwell architecture, sm_120) and RTX Pro 6000 Blackwell (96 GB VRAM) are available and change the VRAM-per-card calculus significantly. A 4-GPU node with RTX Pro 6000 Blackwell gives 384 GB of VRAM — enough to host Qwen 2.5 VL 72B comfortably at FP8 on four cards — at a proportionally higher per-card cost.

What does not change between generations: the architectural pattern. AMD EPYC host, PCIe multi-GPU, front-to-back chassis, dual PSU, Ubuntu 24.04, vLLM in Docker, Prometheus + DCGM exporter. That pattern has been consistent across two GPU generations and will survive the next one. The parts list gets cheaper and faster; the pattern holds.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.