Ethernet for AI: 25/100/400 GbE Fundamentals

Ethernet is the default fabric for any AI cluster built outside a hyperscaler. It is not always the fastest option on paper — N02 will make the case for InfiniBand where it actually wins — but it is the option that exists everywhere, runs every adjacent tool, costs the least per gigabit, and locks you into nothing.

For Kentino-tier clusters — single-node inference servers, 2-to-8-node training rigs, small research labs — Ethernet is essentially always the right answer. The interesting questions are: which speed tier, which NIC, copper or fiber, and what does each step up actually cost. This article answers those, honestly, with numbers from mid-2026 pricing.

If you are deploying a single 4-GPU or 8-GPU box for inference and nothing else, you can skim to the cost table at the bottom and stop. The rest of this article is for multi-node planners.

Why Ethernet wins by default

Three reasons, in order of how often they actually drive the decision:

1. Ecosystem. Every server, switch, storage box, and orchestrator speaks Ethernet. Drivers ship in the kernel. Monitoring tools assume it. The first day of a new hire is not "let me explain InfiniBand."
2. Cost per gigabit. A 100 GbE switch port is roughly a quarter the price of comparable HDR InfiniBand. DACs and transceivers are commodity. NIC supply is multi-vendor — NVIDIA, Broadcom, Intel — which keeps prices honest. InfiniBand is single-vendor.
3. No fabric lock-in. Ethernet is a transport. Modern switches handle TCP, RoCE, NVMe-oF, and management plane traffic on the same wire with sane QoS. InfiniBand commits the fabric to NVIDIA's roadmap.

The trade is that Ethernet for AI needs more tuning to hit the same wire-rate behavior InfiniBand gives you out of the box. RoCEv2, PFC, ECN, DCQCN — all configurable on Ethernet, all preconfigured on IB. N02 and N08 cover that fight.

Speed tiers and when each makes sense

The biggest single mistake on Kentino-tier quotes is over-specifying the link. A 400 GbE switch under an 8-GPU inference box is wasted money. A 10 GbE link between two training nodes is wasted GPU time. The map below is the one that matters.

Three things to internalize:

• 25 GbE is not for the AI fabric. It is for everything around it — admin, telemetry, the link to the storage server when storage is not on the same RDMA fabric. Calling 25 GbE "the AI tier" is a tell that the vendor doesn't know the workload.
• 100 GbE is the honest baseline for multi-node. Below it, gradient sync stalls every iteration. Above it, the question is whether your GPUs can produce data fast enough to saturate the wire. For 5090 / RTX Pro 6000 Blackwell single-node systems doing fine-tune-class work, 100 GbE is usually enough.
• 400 GbE pays back only on real multi-rack training. Two 8-GPU nodes doing 70B fine-tune work do not need it. Sixteen nodes doing pretraining on tokenized text do.

800 GbE exists, the silicon is shipping (ConnectX-8, Thor Ultra), but it is currently a hyperscaler and B200-class deployment story. Mentioned for completeness; you are unlikely to spec it on a Kentino build in 2026.

NIC families that actually ship

The NIC market in 2026 is dominated by NVIDIA's ConnectX line, with Broadcom and Intel covering the rest. Marvell, AMD/Pensando, and a handful of OEM-rebadged Broadcoms exist but rarely show up on Kentino BOMs.

The honest picks for Kentino-tier work:

• Intel E810-CQDA2 when the customer wants 100 GbE on a budget and accepts that RoCE tuning is more work than on Mellanox. Roughly €700–€900 street.
• ConnectX-6 Dx for any 100 GbE link that needs to run RoCE reliably. €1100–€1500 add-on. The 90% answer.
• ConnectX-7 when the platform is PCIe Gen 5 and 200 or 400 GbE is justified by real multi-node training. The single-port OSFP NDR400 card is around €1650 street, and it switches between IB and Ethernet in firmware — useful if you might migrate fabrics later.
• BlueField-3 and ConnectX-8 exist; if you are speccing them you are not reading this article.

PCIe is the silent ceiling. A 100 GbE NIC in a Gen 3 x8 slot trains at link rate and runs at half speed. The rule:

On Kentino's 8-GPU EPYC platforms, the GPUs eat most of the Gen 4/5 x16 slots; the NIC ends up in a Gen 4 x8 in some configs. That is fine for one 100 GbE port, marginal for two. Verify before quoting. W02 (PCIe lanes & topology) is the deep version of this story.

The metallic and optical options — DAC, AOC, optics

Three ways to make a link physical, separated by an order of magnitude in cost. Buyers waste real money on optics where DAC would have worked.

Rules that hold across vendors:

• DAC inside a single rack. Passive copper at 100GBASE-CR4 is reliable to 3 m, spec'd to 5 m with active variants. At 400G (100G PAM4 per lane) the practical limit is 2 m passive — 3 m DACs exist but signal integrity gets fragile with cable bends.
• AOC for 5–30 m runs, especially across adjacent racks. The transceiver is bonded to the fiber, so a bad end means scrapping the whole cable.
• SR optics + MMF for in-room runs beyond 30 m or where structured cabling already exists. OM5 buys ~50% more reach over OM4 at 100 GbE SR and is what you should pull for new installs today.
• LR optics + SMF for crossing buildings or rooms. Overkill inside a single 4-rack cluster.

Cable-length math for a 4-rack cluster. A row of four 42U racks, each ~600 mm wide with ~100 mm of inter-rack gap. Top-of-rack switch at the top of rack 1 to a NIC at the bottom of rack 4: ~2.1 m horizontal + ~2 m vertical = ~4 m run with slack. At 100 GbE you are at the edge of DAC; at 400 GbE you are past it. For any cluster wider than ~2 racks at 400 GbE, plan AOC or optics. This is the most common spec error on multi-rack quotes — assuming DAC will reach because "the racks are right next to each other."

End-to-end 100 GbE link, 2 m intra-rack hop, mid-2026: ConnectX-6 Dx pair ~€2400 + two switch ports amortized ~€1500 + 2 m QSFP28 DAC ~€90 = ~€4000 per link. The same link at 400 GbE (ConnectX-7 pair, 400 GbE switch ports, 2 m QSFP-DD DAC) runs roughly €14k — 4× the bandwidth for ~3.5× the cost. The economics are reasonable; the question is whether the workload needs it.

Switch landscape

The 2026 AI Ethernet switch market sorts into three obvious buckets.

Two opinionated calls:

• For 4-to-8-node training clusters at 100 or 400 GbE, a single Spectrum-X SN5600 or Arista 7060X4 is the right answer. One switch, one fault domain, no Clos, vendor-shipped RoCE templates. The SN5600 is 64×800 GbE in 2U with 51.2 Tb/s aggregate and AI-tuned deep buffers; the 7060X4 gives you 32×400 GbE in 1U at ~700 ns cut-through latency for noticeably less money when you do not need 800 GbE.
• For 16+ nodes, plan routed L3 Clos with two spines, BGP unnumbered between leaves and spines, ECMP across the spines. N08 covers the uplink mechanics; N04 covers the topology choice.

MTU and jumbo frames

Default Ethernet MTU is 1500 bytes. At 100 GbE that is ~8.3 million packets per second per stream, each one paying kernel per-packet cost. MTU 9000 cuts packet rate by 6×, drops CPU overhead correspondingly, and is essentially mandatory for any AI fabric.

For RoCEv2 specifically, the IB-style MTU is picked from the largest size that fits inside the Ethernet MTU. 9000-byte Ethernet gives RoCE a 4096-byte MTU — the value every NCCL and RDMA stack expects. Stay at 1500 and you get a 2048-byte RoCE MTU and noticeable performance loss on every collective.

Three things to know:

• Must match end-to-end. A single 1500-MTU device in the path silently fragments and tanks throughput. Set it on every NIC, every switch port, every router hop on the AI VLAN.
• Keep jumbo confined to the AI VLAN. Your corporate network is not going to run MTU 9000 cleanly. The management plane stays at 1500.
• Verify with ip link and tracepath, not by trusting the config. The number of clusters where someone set 9000 on the NICs and forgot the switch ports is embarrassing.

Lossy vs lossless Ethernet — a preview

Plain TCP/IP over Ethernet is lossy: switches drop packets when queues fill, TCP retransmits, throughput backs off. Fine for streaming web traffic, fatal for RDMA — a single packet loss in a RoCE allreduce kills the whole transfer and forces NCCL to time out.

The fix is lossless Ethernet, achieved with two mechanisms working together:

• PFC (Priority Flow Control, 802.1Qbb) — per-priority pause. When the switch's buffer for priority 3 (the RDMA priority by convention) fills, it sends a pause frame to the upstream port. Short-term emergency brake.
• ECN (Explicit Congestion Notification) — switches mark packets in queues that are filling, the receiver echoes a CNP back, the sender slows down. Long-term congestion control.

Together they form DCQCN — Data Center Quantized Congestion Notification — which is the standard RoCEv2 control loop on modern Ethernet AI fabrics. ECN does the work most of the time, PFC fires only when ECN can't react fast enough.

The takeaway for this article: if you are running RoCE, your switch must support PFC and ECN on the RDMA priority, and your NIC must be configured to mark/respect them. Buying a switch from a vendor with documented RoCE templates (NVIDIA Spectrum, Arista, Cisco Nexus 9000) saves a week of config work. N02 covers the comparison with InfiniBand; N07 goes deep on the congestion-control side.

Buffer depth — shallow vs deep

The other axis on AI switches is buffer depth. AllReduce is many-to-one in every iteration — every node in the ring or tree converges on one node at the same instant. That is incast, and it crushes shallow-buffered switches.

• Shallow-buffered switches (older Tomahawk, generic enterprise) have 4–32 MB of buffer total. Cheap, low-latency, fine for east-west traffic on a corporate network. Drop packets the first time incast hits.
• Deep-buffered AI switches (Jericho3-AI, Spectrum-X, some 7800R3 line cards) have hundreds of MB to several GB of buffer. Absorb the incast burst, hold packets while DCQCN ramps down the senders.

For a 2-node cluster, buffer depth is irrelevant. For 8+ nodes doing dense allreduce, deep buffers are the difference between a healthy fabric and one that drops PFC pauses and stalls. This is a real reason to buy AI-class switches rather than generic ones.

Honest take — does this apply to your build?

A single-node Kentino K-AI server doing inference or single-node fine-tuning does not need any of this. The GPUs talk over PCIe and NVLink; the Ethernet link is for prompts in, tokens out, and dataset reads. A pair of 25 GbE SFP28 ports on the host's onboard NIC is sufficient.

You are reading this because you are planning two or more nodes. The matrix:

• 1 node: 10 or 25 GbE onboard. No upgrade.
• 2–4 nodes, fine-tuning: 100 GbE via ConnectX-6 Dx, one switch, RoCE configured. ~€4k per link.
• 4–8 nodes, 70B+ training: 100 or 200 GbE with ConnectX-7 if PCIe Gen 5 is available. Spectrum-X SN5600 or Arista 7060X4 single switch.
• 8–16 nodes, dense training: 200 or 400 GbE, routed L3 Clos with two spines, deep-buffer switches.
• 16+ nodes: past Kentino's typical lineup. Use NVIDIA reference architecture (Spectrum-X, BlueField-3 or ConnectX-8) and a consultant.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.