InfiniBand vs RoCE vs Plain Ethernet: Choosing the Fabric in 2026

Three transport choices show up on every multi-node AI cluster RFQ: InfiniBand, RoCE, and plain TCP/IP Ethernet. They are not three points on the same line. They are three different engineering philosophies — lossless-by-design HPC fabric, lossless-tuned Ethernet, and the everyday network stack the rest of computing uses — and they have different cost curves, different operational footprints, and very different right answers depending on what you are actually building.

This article walks through what each one is, where the latency and money actually go in 2026, and where the line sits between "we need RDMA" and "we need RDMA and InfiniBand specifically." We end with a flat decision matrix.

Audience: people sizing a 2- to 32-node AI cluster and trying to decide whether the line item for an InfiniBand switch is honest or theatre. Cross-reference N01 for the 25/100 GbE basics, N06 for the underlying latency budget, N07 for the routing knobs, and N08 for the practical RDMA bring-up.

Three fabrics, what each actually is

Native InfiniBand is a purpose-built HPC fabric. The link layer is lossless by design — credit-based flow control means a sender never transmits unless the receiver has buffer space, so packets do not get dropped under congestion the way Ethernet drops them. Routing is centralized: every fabric has a subnet manager (an opensm process or an embedded SM on a managed switch) that assigns LIDs, computes paths, and reconfigures on topology change. The application surface is the verbs API — ibv_post_send, queue pairs, completion queues, memory regions — the same API the rest of the RDMA world borrowed.

In 2026 the deployed generation is NDR (400 Gb/s per port, ConnectX-7, Quantum-2 switches). XDR (800 Gb/s, ConnectX-8, Quantum-X800) is shipping into hyperscaler and AI-lab orders this year; the IBTA published the XDR spec in 2023 and silicon is now in volume. Switch hop latency is sub-100 ns on NDR cut-through silicon — the lowest commercially available, and it has been the lowest by roughly a factor of four for two decades.

RoCEv2 (RDMA over Converged Ethernet, version 2) is the same RDMA verbs API running on UDP/IP over Ethernet. The NIC silicon is, on a ConnectX-7 or ConnectX-8, the exact same chip as the InfiniBand version — a firmware flag (LINK_TYPE_P1 = 1 for IB, 2 for Ethernet) flips it between modes. What is not free: Ethernet is lossy by default, and RDMA assumes lossless transport. So a RoCE deployment needs the fabric to behave like InfiniBand at the link layer, which means Priority Flow Control (PFC) for the short-term emergency brake, ECN for the long-term congestion signal, and a control loop like DCQCN that ties them together. That is the configuration work — and the operational hazard — that separates RoCE from plain Ethernet. Done right, RoCEv2 sits within striking distance of InfiniBand latency: 1.5–2.5 µs same-switch RDMA RTT versus IB's ~1 µs.

Plain TCP/IP Ethernet is the rest of the planet. Sockets API, kernel network stack, no RDMA, no lossless guarantee. Performance lives in the 10–30 µs/way latency band per N06, throughput is fine to NIC line rate for bulk transfer, and CPU is on the data path. This is the right fabric for management traffic, model downloads, telemetry, and the gigabit between your robot and your inference server. It is the wrong fabric for synchronous gradient AllReduces across 16 GPUs.

The headline comparison

Read the cost row as relative, not absolute. The real spread on switches is roughly 2–3× per port for an InfiniBand NDR switch versus an equivalent AI-grade Ethernet switch (Spectrum-X, Tomahawk 5/6). The NIC itself is essentially the same hardware in both modes; the difference is in licensing and in the support contract.

Where the latency goes

N06 takes a single RDMA round trip apart microsecond by microsecond. The condensed version, relevant to this comparison:

InfiniBand wins on switch ASIC latency — that is the gap that has not closed. Modern AI Ethernet switches (Spectrum-X SN5600 / Tomahawk 5) get the per-hop number into the 400–600 ns range, which is competitive enough that across a two-hop leaf-spine the absolute difference is on the order of 1 µs. That matters for tight tensor-parallel collectives. It does not matter for storage reads or batched inference.

The plain TCP column is the one that scares people who only know the latency from sockets. The 10–30 µs/way kernel cost is per packet; on a small-message collective with many round trips, it accumulates fast. NCCL over TCP is functional but it is the option of last resort.

Bandwidth and the NIC reality

In 2026 the same family of cards covers both InfiniBand and RoCE:

The headline: a ConnectX-7 card you buy for IB is the same card you would buy for RoCE. List price in early 2026 sits around €1,500–€2,200 for single-port NDR; the variant SKU just preconfigures the firmware. That is why "switch from IB to RoCE" is not a NIC re-buy in most clusters — it is a firmware flag, a transceiver swap, and a switch fabric change.

On the AMD side, Pensando Pollara 400 (now shipping with the MI300X/MI325 platforms) is the first credible non-NVIDIA RDMA NIC with production AI deployments. Broadcom announced an 800G AI NIC in 2025 (Thor Ultra / Thor 3 generation). For the first time in a decade, the RDMA NIC market is not a one-vendor town.

Switches: where the money actually is

The NIC is roughly the same price across modes. The switch is not.

The 2–3× per-port premium for InfiniBand has been stable for years. It is a real cost: a 64-port Quantum-2 NDR switch lists in the €120k–€180k band depending on optics and support; a 64-port Spectrum-X SN5600 with similar port count lands around €60k–€90k; a generic Tomahawk 5 whitebox from a Taiwanese OEM can be half of that. Add optics — OSFP NDR transceivers are €600–€1,200 each at volume — and the per-port "all in" delta widens further.

Where it does not widen as much as people expect: AOC (active optical cables) and DAC (direct-attach copper) for in-rack runs. A 3 m DAC is ~€100–€200 in either ecosystem. The big optics bill shows up only when you go past ~5 m and need SR/LR transceivers on both ends.

The Spectrum-X angle: NVIDIA's "AI Ethernet" pitch

NVIDIA's strategic answer to the Ethernet shift is Spectrum-X — a bundle of Spectrum-4 (and now Spectrum-5) switch silicon, BlueField/ConnectX NICs, and an end-to-end congestion control story they call "adaptive routing + telemetry-based load balancing." The marketing claim is that Spectrum-X delivers InfiniBand-class AllReduce performance on Ethernet, validated on real AI training workloads.

The measured reality, from NVIDIA's own benchmarks and from third parties:

• Spectrum-X closes roughly 80–90% of the AllReduce performance gap with InfiniBand NDR on NCCL workloads. The remaining gap is small messages and worst-case tail behaviour.
• The per-packet adaptive routing (sprays packets across multiple paths and re-orders at the receiver) is the secret sauce. Standard ECMP hashes whole flows, which on a small-flow-count AI workload produces hot spots. Spectrum-X breaks the flow-hash assumption.
• It only works as a stack. Mix a non-NVIDIA NIC with a Spectrum switch and the adaptive routing telemetry path breaks. This is the lock-in trade-off — you escape from the InfiniBand ecosystem only to re-enter it on Ethernet.

The honest read: Spectrum-X is a real engineering answer to a real problem. It also keeps NVIDIA in the loop on every layer of an AI cluster, which is exactly what hyperscalers chose Ethernet to escape. Whether it is the right call depends on whether the 10–20% remaining performance gap is worth the extra integration freedom you trade away.

The non-NVIDIA answer is Ultra Ethernet — the UEC 1.0 spec published in 2025, backed by AMD, Broadcom, Cisco, HPE, Intel, Meta, Microsoft. Multipath delivery, packet-level load balancing, in-network congestion control, open API. The Pensando Pollara and Broadcom Thor 3 are the first NICs shipping UEC. As of mid-2026 this is the credible cross-vendor Ethernet-for-AI path that does not require the NVIDIA stack end-to-end.

Hyperscaler context (and what it means for the rest of us)

The headline from the past two years: the hyperscalers have largely left InfiniBand. Meta trained Llama 3 over RoCE. AWS's AI fabric is Ethernet end-to-end. Google's TPU pods have always been custom, but their GPU fleets sit on Ethernet. Microsoft Azure runs a mix and is investing heavily in Ethernet on the new builds. Roughly 70% of new AI infrastructure deployments in early 2026 chose Ethernet over InfiniBand, per Broadcom's earnings disclosures, a reversal from the ~80% InfiniBand share in 2023.

Meta's published conclusion was unambiguous: properly tuned RoCEv2 and InfiniBand deliver equivalent training performance on their workloads. The shift is not about raw performance — it is about who owns the silicon, who staffs the SREs, how the parts get sourced, and whether the operational model integrates with the rest of the data centre.

Three things matter for the rest of us, who are not running 10,000-GPU training fabrics:

1. The "Ethernet can do AI" question is settled. It can. RoCEv2 with modern AI switch silicon and proper congestion control is production-grade. The hyperscalers proved it. You no longer need to defend the choice in an RFQ.
2. Operational expertise has shifted with the silicon. Five years ago there were three people in your country who could turn up an InfiniBand fabric and one who could properly tune PFC/DCQCN on RoCE. Today the RoCE skills are far more widespread, the InfiniBand ones are concentrated in HPC labs and at NVIDIA partners. Plan staffing accordingly.
3. The vendor lock-in story has inverted. InfiniBand has always been single-vendor (NVIDIA, formerly Mellanox). Ethernet was the open alternative — until Spectrum-X made the NVIDIA-end-to-end pitch on Ethernet too. The truly vendor-neutral answer in 2026 is UEC-class RoCE with Broadcom or AMD silicon.

When InfiniBand is the right call

InfiniBand is the right answer when most of these are true:

• Dedicated training cluster, 16+ nodes, running synchronous gradient AllReduces at sustained rates. The latency advantage on the small-collective end compounds across thousands of iterations.
• You are buying NVIDIA HGX or DGX systems anyway. Those baseboards are designed around NDR/XDR. Mixing Ethernet on a DGX node is operationally awkward.
• You have on-staff expertise that knows the subnet manager, can read ibtracert output, and is comfortable with NVIDIA's UFM management.
• The budget supports a 2–3× per-port switch premium without breaking the cluster economics.
• The cluster is the entire purpose of the building. A pure HPC/AI deployment with no shared storage tenants and no application traffic — InfiniBand simplifies the operational model because there is no QoS argument to have.

If three of those five describe your build, InfiniBand earns its price tag. If only one does, you are paying for a fabric optimized for someone else's problem.

When RoCEv2 is the pragmatic call

RoCE is the right answer for the largest band of Kentino's customer footprint:

• 4 to 16 nodes, mixed workload, training and inference and storage on the same fabric.
• Budget-conscious but RDMA-required. The cluster does enough multi-node training or distributed inference that the kernel-stack tax of plain TCP is unacceptable, but the InfiniBand switch premium is.
• Shared fabric with non-AI traffic. Operations, storage, and developer access all ride the same Ethernet plant. RoCE coexists with that; InfiniBand needs its own physical layer.
• You want vendor optionality. Mix Broadcom switches with NVIDIA NICs with AMD MI300 nodes. The Ethernet ecosystem allows it; the InfiniBand ecosystem does not.
• The team has Ethernet operational depth already. PFC/ECN tuning is real work, but it is work on a substrate everyone already knows. The SM in your head is the L2/L3 control plane you have run for years.

The trap to flag: RoCEv2 misconfigured is worse than plain TCP. PFC pause storms across an unsegmented fabric can take down the management plane. ECN thresholds set wrong produce a fabric that looks healthy on a perftest but collapses under real AllReduce incast. Per N08, this is the "rolling PFC config by hand on a generic whitebox is a project" warning. Buy from a vendor with vetted RoCE templates (NVIDIA Spectrum-X, Arista, Cisco Nexus 9000 with the AI fabric service profile), or hire someone who has done this before.

When plain TCP/IP Ethernet is enough

Plain Ethernet — no RDMA, no PFC, no DCQCN tuning, just sockets and the kernel stack — is the right answer more often than people admit:

• Single-node 4× or 8× GPU server. Inter-GPU traffic stays on PCIe or NVLink inside the chassis. The only network role is shipping training data in, model weights out, telemetry around. 25 GbE or one link of 100 GbE is fine.
• Inference clusters with light inter-node coordination. Per-request latency is dominated by GPU compute; 10 µs of network is noise.
• Mixed-use lab, robotics workbench, dev environment. The cluster does not run sustained synchronous training. RDMA is overkill.
• Robotics edge-AI setups as described in I01 — the robot talks to one inference server, latency budget is in the hundreds of milliseconds, the fabric is invisible.

The honest call for most of Kentino's customer base: if you are buying a single K-AI 256 Turin Dual with 8× RTX 5090, you do not need InfiniBand and you do not even need RoCE. A pair of 100 GbE links to a standard data-centre switch, jumbo frames enabled, default kernel tuning — that is the right answer and it costs five figures less than the alternative.

Honest Kentino take

We sell to customers building 1- to 8-node clusters with consumer and workstation GPUs (5090, 4090, RTX Pro 6000 Blackwell). Not the hyperscale market, not the HPC market. For that footprint:

• Single-node 8-GPU box: plain 25/100 GbE. Done. No RDMA, no fabric tax. The interesting bandwidth is inside the chassis on PCIe Gen5; the outside link is for data ingest and management.
• 2- to 4-node cluster with occasional distributed training: RoCEv2 on 100 GbE with a vendor-supported AI switch template. Spectrum-X if you want the NVIDIA-validated path; Tomahawk 5 / Arista if you want the multi-vendor path. Plan two weeks for proper PFC/ECN bring-up and nccl-tests validation.
• 4- to 8-node sustained training cluster: RoCEv2 on 200 or 400 GbE, with the management plane fully separated. This is the sweet spot for the per-port cost curve; the difference vs IB at this scale is not enough to justify the switch premium unless the workload is unusually latency-sensitive.
• Above 8 nodes, sustained AllReduce-heavy training: InfiniBand starts to deserve consideration. Not automatic — measure it. But at this size the operational simplicity of "the fabric is lossless, the SM handles routing, NCCL just works" is worth real money.
• Anything genuinely DGX-scale: not us. Buy HGX, accept the InfiniBand bundle.

---

This is part of the Kentino Wiki, a reference series on AI compute and the systems that connect it. Corrections welcome at info@kentino.com.