Switchless Topologies: Mesh, Ring, and Direct-Connect for Small AI Clusters

A 32-port 400 GbE switch with the optics, breakouts, and software entitlements to actually use it lands somewhere between €40k and €80k in mid-2026, and a 64-port NDR InfiniBand switch is worse. For a customer building a two- to four-node training rig, the switch can cost more than the GPUs in one of the nodes. It also adds a hop of latency, a single point of failure, a separate firmware lifecycle, and a small project's worth of PFC/ECN tuning if you are running RoCE.

The fact that almost nobody talks about is that below roughly eight nodes you do not need a switch at all. You can cable the NICs to each other directly, skip the switch entirely, and end up with something simpler, cheaper, and a little faster. The fact almost nobody admits is that above roughly eight nodes switchless falls off a cliff, and the cabling, port count, and operational story stop being defensible. This article maps that range honestly.

It is the companion to N04 (switched topologies). Read N06 for the latency math the switchless win is built on, and K07 for what a single base node looks like — the building block this article connects.

The case for switchless

Four things you get for free by removing the switch:

1. Zero switch latency. A modern cut-through Ethernet AI switch eats 400–600 ns per hop. An NDR InfiniBand switch is under 100 ns. A direct NIC-to-NIC cable adds wire delay (~5 ns/m on copper, the same on fibre) and nothing else. On a two-node ping-pong, this drops one-way latency from ~2 µs to ~1.2–1.5 µs.
2. Zero switch cost. A two-node direct connect is two NICs and one DAC. A three-node triangle is three NICs and three DACs. The capex saving versus even a small 100 GbE switch with QSFP28 optics is real money on a small build — €10k–€30k that goes back to GPUs.
3. No PFC/ECN headaches. A direct NIC-to-NIC link is point-to-point — flow control is a two-party conversation, PFC degenerates to "tell the peer to stop." There is no fabric-wide pause propagation pathology because there is no fabric.
4. One device class to debug. When something breaks on a switchless fabric, the suspect list is two NICs, one cable, and the kernel drivers on both ends. That is a small, finite search space.

The two-node case: just plug them together

This is the cleanest possible AI-cluster setup, and the one where switchless is unambiguously correct.

Two-node direct connect: one passive OSFP DAC, no switch, ~0.8–1.2 µs RDMA latency, ~50 GB/s usable throughput per direction.

A single OSFP DAC between two ConnectX-7 NICs at 400 Gb/s. That is the entire inter-node fabric. The same NICs that would face a switch in a larger build face each other instead. RDMA verbs work, NCCL picks it up automatically, GPUDirect RDMA runs unchanged.

What you get: ~50 GB/s per direction usable, sub-microsecond raw RDMA latency (ib_send_lat lands around 0.8–1.2 µs), one cable. No aggregation problem because there is nothing to aggregate. No oversubscription because there is no fan-out point.

For a two-node training pair — the most common "I have outgrown one box" build in our customer base — this is the right answer. Skip the switch. Plug them together. Spend the saved money on a bigger NVMe tier or a second NIC port for redundancy.

A practical refinement: use a dual-port ConnectX-7 and run two parallel 200 Gb/s DACs between the boxes, with NCCL configured to use both HCAs (NCCL_IB_HCA=mlx5_0,mlx5_1). You lose a little bit of per-flow peak but you get path redundancy and slightly better small-message behaviour from parallel queue pairs. We default to this on two-node builds.

Three and four nodes: triangle and K₄

Three nodes is the smallest case where the topology starts to matter. The options are:

• Linear chain (A-B-C). Two cables. Diameter 2. Node B is a hotspot — all A-to-C traffic crosses it. Avoid.
• Triangle (full mesh). Three cables. Diameter 1. Every node has two ports. Every flow is one hop. This is the right answer.

Left: triangle (K₃) — 3 nodes, 3 cables, diameter 1. Right: K₄ full mesh — 4 nodes, 6 cables, diameter 1. Every pair directly connected.

Four nodes is where it gets interesting. The full mesh — the complete graph K₄ — has six links total, three ports per node, and diameter 1. Every node reaches every other in exactly one hop. The cabling math:

Full mesh ports-per-node is N-1, which is why this approach implodes fast. At eight nodes you need seven ports per box, which is the end of the practical road on a single PCIe Gen5 x16 slot.

When does four-node full mesh beat a small switch? Specifically when you have 4× K-AI 128 nodes for inference and want them tightly coupled, you are running RoCE and do not want PFC across a switch, and the marginal €15k–€25k of a 100 GbE switch with optics is meaningful in the budget.

When does a small switch win even at four nodes? When you might add a fifth node next quarter. Adding one node to a K₄ mesh requires recabling every existing node to add the new ports. A switch has spare ports; you just plug in.

The 8-node case: hypercube, with an asterisk

The 3-cube (Q₃) — a hypercube of dimension 3 — is the textbook switchless layout for eight nodes. Each node sits at one corner of a cube; each edge of the cube is a direct link. Three ports per node, twelve links total, diameter 3.

The honest take: this is rare in production. It works, the diameter-3 worst case is acceptable for most collectives, but the cabling diagram is genuinely confusing to anyone who did not build it, troubleshooting requires understanding the Gray-code labelling, and a small 16-port 200 GbE switch is now in the same price band as the extra NIC ports and cables. The 8-node hypercube is more interesting as a teaching example than as a thing we ship. At eight nodes, our default recommendation is a switch.

The ring: dumb, simple, and surprisingly relevant

Forget about minimizing diameter. The ring connects each node only to its two neighbours: A-B-C-D-...-A. Two ports per node regardless of cluster size. N links total. Diameter N/2.

This sounds terrible — diameter 4 on 8 nodes, diameter 16 on 32 nodes. Why is it not always wrong?

Because NCCL's ring allreduce maps onto a physical ring exactly. The algorithm sends each chunk of data once around the ring per phase; if the physical topology already is a ring, the algorithm runs at line rate of a single link, with no wasted bandwidth. NCCL's default for medium-to-large messages is ring, not tree, because ring achieves the optimal bandwidth bound: 2(N-1)/N × link bandwidth for allreduce. The diameter of the physical topology does not matter at large message sizes — what matters is that every link is used in parallel, and the ring does that perfectly.

The pragmatic place where a physical ring is the right switchless answer is 4–8 node training rigs where every node has exactly two RDMA ports already. The catch: ring has no path redundancy. One bad cable splits the cluster into two pieces.

When switchless beats a small switch, in numbers

Approximate price comparison for an 8-node fabric build, mid-2026 (EUR ex VAT):

The crossover where the switch pays for itself is around 6–8 nodes, depending on bandwidth tier and whether you intend to grow.

Uplink: the part people forget

A switchless data fabric is internally self-contained. It is not, by itself, connected to anything. The cluster still needs an uplink for dataset and model pulls from corporate storage, SSH from developer workstations, telemetry to Prometheus/Grafana, IPMI/BMC management, and container registry traffic.

Pattern A — every node has a separate management NIC. Each node carries one small 25 GbE (or even 10 GbE) port to a cheap management switch, completely independent of the RDMA fabric. This is the right answer almost always. The RDMA fabric is a sterile, lossless, tuned environment; the management plane is a normal Ethernet network with normal traffic. Mix them and the management traffic disrupts your collectives.

Pattern B — dedicated uplink node. One node in the cluster has an extra port that connects out. Other nodes reach the outside world by routing through this node. Works for tight budgets and small lab setups, but the uplink node becomes a bottleneck for dataset reads and a single point of failure for management access.

The hard wall at ~16 nodes

Switchless dies above 16 nodes for three independent reasons, any one of which is sufficient:

1. Port count per node. Full mesh wants N-1 ports per node. Hypercube wants log₂(N). Even the log scaling means 16 nodes need 4 ports per node, which is at the edge of practical NIC density on a single PCIe Gen5 x16 slot. 32 nodes need 5 ports per node — multiple slots, multiple NUMA placements to manage.
2. Cabling combinatorics. A 16-node K₄ full mesh has 120 cables. A 16-node Q₄ hypercube has 32. Either way, labelling, documentation, and physical access to each cable matter. One miswired cable in a 32-cable hypercube takes hours to find.
3. Operational story. Replacing a failed NIC in a switchless fabric requires identifying the N-1 (or log N) cables that connected it, re-routing each one to a specific port on the replacement. The MTTR difference versus switched is real.

The honest summary: switchless is right for 2 to 4 dedicated nodes, defensible for 5 to 8 nodes with a clear "we will not grow" commitment, and a mistake for 9 or more nodes. At 9+, buy a switch.

Two concrete builds worth describing

2× K-AI 256 Turin Dual, direct-connected, 400G. Two 8-GPU EPYC Turin nodes (5090 or RTX Pro 6000 Blackwell), each with a single-port ConnectX-7 400 GbE / NDR, one 3 m passive OSFP DAC between them. Total inter-node hardware cost: ~€4k. NCCL allreduce busbw on large messages: ~45 GB/s. Suitable for two-way tensor-parallel inference of a 405B dense model (split layers across the two boxes), or fine-tuning a 70B that does not quite fit on one box. We have shipped variants of this build several times. It is boring, it works, it costs an order of magnitude less than the equivalent switch-attached setup.

4× K-AI 128 in K₄ full mesh, 100G. Four single-socket EPYC nodes with 4× RTX Pro 6000 Blackwell each. Each node carries a tri-port-equivalent NIC layout (one dual-port plus one single-port, or one quad-port with one port unused), 100 GbE DAC fabric. Six cables total. Bisection bandwidth 400 Gb/s. Used for tensor-parallel inference of a 70B-class model with 4-way splitting and full activation passing between every pair. Eliminates the switch as a single point of failure for the inference service, and the customer's budget went to GPUs instead of switching gear. Trade-off: locked at four nodes; growing requires re-architecting.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.