Tokens per Second per Euro: GPU Economics for Inference

Most buyers ask the wrong question. They want to know which GPU is "fastest." The right question for a production inference workload is "which GPU produces the most tokens per euro spent over the next three years for the model I actually serve." Those two questions have different answers, and the gap is large enough that picking on raw speed routinely costs Kentino customers 30–60% of their budget.

This article works the math. The audience is somebody sizing an inference build, has read W01 and N03, and now needs to decide between an RTX 5090, an RTX Pro 6000 Blackwell, an L40, or an L4 — and how many of them.

The metric: why tok/s per euro, not tok/s

Two performance regimes matter, and conflating them is the most common sizing mistake.

Single-stream tok/s is what one user sees while waiting for one response. It is bandwidth-bound on token generation and almost entirely a function of VRAM bandwidth divided by model weight footprint (see W01). Sustained throughput tok/s is what a server pumps out across many concurrent users with continuous batching enabled. On Blackwell-class hardware this is compute-bound once batches grow past 16. It is what production cares about.

The two diverge by 10–25× on the same card. Single-stream Llama 70B INT4 on an RTX Pro 6000 Blackwell lands around 32 tok/s. The same card at batch 32 with vLLM continuous batching pushes 600+ tok/s aggregate. Picking on single-stream under-sizes throughput deployments by an order of magnitude; picking on throughput over-sizes latency-sensitive ones. The TCO math here uses sustained throughput because that is what dominates the bill.

Current EU retail prices (May 2026)

Prices below are EU retail, ex VAT, on the open channel — not OEM volume contracts and not cloud rental. Sourced from EU price trackers and distributor list in May 2026. Treat as ranges; quarter-by-quarter swings of 10–20% are normal.

Two observations from this table alone. The Pro 6000 Server Edition is roughly 3× the price of a 5090 for 3× the VRAM and equivalent bandwidth — pricing tracks memory capacity, not raw speed. The L4 is the only card here under 100 W. That fact alone reshapes TCO once electricity enters the equation.

Single-GPU benchmark numbers

Numbers below are from public benchmarks (vLLM, llama.cpp, SGLang) collected from Spheron, CloudRift, RunPod, and the vLLM project's own published runs, May 2025 through May 2026. They are not Kentino's bench — we cross-checked against our internal 4×5090 and 4×Pro 6000 boxes for the cells we serve regularly; the public numbers track within ±15%.

Llama 3.3 70B FP8 — single-stream decode

Llama 3.3 70B INT4 (AWQ) — single-stream decode

Qwen 2.5 32B INT4 (AWQ) — single-stream

Llama 3 8B FP16 — single-stream and aggregate

A few honest reads from these tables. The 5090 and the Pro 6000 Blackwell have identical bandwidth (1.79 TB/s) and identical decode rates per token — the differences are VRAM capacity and ECC, not speed. The L40 sits roughly half-speed on inference relative to the Blackwell pair because GDDR6 ECC is 860 GB/s versus 1,790 GB/s. The L4 is a quarter-bandwidth card; it makes up for it with density and price.

For 70B-class single-stream work, the Pro 6000 Blackwell is the only card that runs the model at FP8 in one slot. The 5090 forces you into INT4 and 2-way tensor parallel — and TP=2 on PCIe loses 30–35% to all-reduce (see N03).

Multi-GPU scaling: the PCIe tax

PCIe-only tensor parallel does not scale linearly. We covered the mechanism in N03; here are the empirical multipliers you should plug into a TCO model.

Pipeline-parallel and data-parallel paths scale near-linearly; tensor-parallel pays a tax that grows with GPU count. For a 70B-class model the right configuration on 8× 5090 is two independent replicas (DP=2 × TP=4) or TP=4 × PP=2 — not TP=8. See K03 for the configuration rules.

This matters for cost because doubling the GPU count never doubles the throughput. A naïve "twice the GPUs, twice the tok/s" budget projection over-promises by 30–50%.

The 3-year TCO model

GPU sticker price is roughly half the real cost of running an inference card over three years. The other half is electricity, chassis share, and floor space. The model below is what we use internally for K-AI sizing.

Per-GPU 3-year TCO =
    GPU cost
  + Chassis share (chassis + CPU + RAM + PSU + NIC) / GPU count
  + Electricity:  TDP × utilization × 8,760 h × 3 × €0.20/kWh
  + Colo/rack space share
  + Maintenance & spares (~5% of GPU cost / year)

Assumptions used below:

• Electricity: €0.20/kWh blended (typical EU industrial colo or owned-DC). Czech retail business is often €0.18–€0.22; Germany higher.
• Utilization: 60% sustained (a realistic production load — not the 100% benchmark figure, not the 5% lab figure).
• PUE: 1.4. Datacenter overhead (cooling, distribution losses). This pushes effective electricity to 0.28 €/kWh of GPU draw.
• Chassis share: €4,000 amortized over 4 GPUs (€1,000/GPU) for a Kentino 4-GPU 5090 build; €8,000 over 8 (€1,000/GPU) for the 8-GPU Pro 6000 chassis.
• Maintenance: 5%/year of GPU cost (spare card sinking fund + driver/ops time).

Per-GPU TCO table

Electricity rows: TDP × 0.60 × 8,760 × 3 × 0.20 × 1.4 (PUE) / 1,000. The L4 line is striking — its sustained 3-year electricity bill is €318. The 5090's is €2,540. Eight times the power draw, eight times the bill.

Cost per million tokens

Now combine TCO with measured sustained throughput on a representative workload — Llama 3.3 70B INT4 at batch 32, or Qwen 32B at batch 32, or Llama 8B at batch 64. Take a card's sustained tok/s, multiply by 0.60 utilization × 3 × 8,760 × 3,600 to get total tokens served, divide TCO by that. Numbers below are illustrative; tune for your actual model and batch.

Read the third column as cost per million served tokens, fully loaded TCO. Open-router and cloud APIs for the same models charge €0.10–€2.00 per million tokens depending on tier; this puts on-prem in the same ballpark or cheaper at sustained load. The math falls apart below 30% utilization — TCO is dominated by capex you paid for whether or not you used.

The 5090 case

On paper, the 5090 has the best raw tok/s per euro of any card in the Kentino lineup for sub-72B inference. €2,800 for 1.79 TB/s of bandwidth and 32 GB of GDDR7 is a lot of card. For single-user serving of a 32B-class model, nothing else is in the same ratio.

The 5090 falls apart in two places:

1. Models that don't fit in 32 GB at the quant you want. A 70B FP8 (~75 GB) needs 3–4 cards; a 70B INT4 (~40 GB) needs 2. Once you tensor-parallel across PCIe, you pay the 30–50% all-reduce tax and your tok/s-per-card collapses.
2. Multi-GPU scaling beyond pipeline parallel. 8× 5090 is a real product (we build it) but the throughput per card on a tensor-parallel 70B is roughly 0.55× the single-card figure. You bought eight cards and got 4.4× the throughput. Not 8×.

The 5090's strength is hosting multiple replicas of smaller models. 4× 5090 running four independent Qwen 32B replicas behind a load balancer beats 4× 5090 tensor-parallel a single Llama 70B on both throughput and latency — and avoids the PCIe penalty entirely.

The Pro 6000 Blackwell case

96 GB of ECC GDDR7 at 1.79 TB/s is what you buy when 70B FP8 needs to fit on one card and you do not want to argue with PCIe about it. The Workstation and Server Editions share the same silicon, same memory, same bandwidth — the difference is form factor (active vs passive cooling), max-power BIOS, and validation for OEM server chassis. The Server Edition costs €700–€1,500 more and is the right pick for any rack-mount build that runs 24/7.

Where the Pro 6000 wins on tok/s per euro:

• One-card 70B FP8 hosting. No tensor parallel, no PCIe tax. Single-stream at 30+ tok/s, batched aggregate at 480+ tok/s.
• Scale-out via replicas. 4× Pro 6000 = four independent 70B replicas behind a router. Linear throughput scaling, graceful failure isolation.
• ECC for training. If the workload includes LoRA / QLoRA fine-tuning or full retraining, ECC earns its keep (see W01).

Where the Pro 6000 is overkill: any deployment that does not need >32 GB per card. If your models all fit on a 5090, you are paying 3× the price for VRAM you will not use. We see this regularly — buyer specs the Pro 6000 because it is the "professional" card, then runs a 13B model on it.

The L40 case

The L40 sits in an awkward spot. 48 GB ECC, GDDR6 not GDDR7, 860 GB/s bandwidth — roughly half a Pro 6000 Blackwell on inference, at 75–90% of the price. On pure tok/s per euro at inference, the Pro 6000 wins.

The L40's case is built on three other axes:

1. TDP. 300 W versus 600 W. Half the electricity over three years (€1,325 vs €2,650 in our model). In thermally constrained racks or single-phase circuits, this is the difference between four cards or two.
2. Datacenter validation. L40 ships with passive cooling, OEM validation across Supermicro/Dell/HPE, and the standard 5-year datacenter warranty path. The Pro 6000 Workstation is not validated for 24/7 chassis operation; the Server Edition is, but it is newer in the channel.
3. Reliability ceiling. L40 is purpose-built for sustained server load. Mean time between failure on a 24/7 inference workload is measurably better than the consumer-derived 5090.

The honest read: if your customer cares about uptime more than raw tok/s — regulated industries, on-call SLA contracts, anything that loses money when a card dies in the middle of the night — the L40 is worth the tok/s-per-euro hit. For raw price-performance, it isn't.

The L4 case

The L4 is the one nobody asks about and the one most customers should buy. 24 GB, 72 W single-slot, no power connector required, passively cooled. Eight of them fit in a 1U server. Sixteen fit in a 2U.

What the L4 does well:

• 8B-class model serving at scale. Llama 3 8B FP8 at batch 64 lands around 650 tok/s per card. Eight L4s in one chassis = 5,000 tok/s aggregate for a Llama 8B endpoint. The same chassis pulling 600 W of GPU.
• Multi-replica density. Each L4 hosts an independent 8B model. Eight replicas behind a load balancer with no inter-GPU communication at all. Failure of one card removes 1/8 of capacity, not the whole service.
• Power-constrained sites. A 16 A circuit can run twelve L4s plus host overhead. The same circuit barely runs three 5090s.
• Cost per million tokens. At €0.12/Mtok for 8B FP8, the L4 is cheaper than every cloud API for the same model class.

What the L4 does poorly: anything above 13B, anything that needs real bandwidth, training of any kind. The L4 is a fixed-function inference card for small to mid-class models. If your workload doesn't fit that, skip it.

Honest comparison vs cloud

The cloud math for a like-for-like build. AWS p5.48xlarge (8× H100 SXM) lists at $98.32/hr on-demand, normalized to $12.29 per GPU-hour. Specialized AI clouds (Lambda, RunPod, CoreWeave) run $2.00–$4.00 per H100-hour on-demand and below that on 1-year commits.

Three-year on-demand cloud cost for one H100-equivalent inference slot:

• AWS on-demand: $12.29 × 8,760 × 3 = $323,000
• Spot/discount AI cloud: $2.50 × 8,760 × 3 = $65,700
• 1-year reserved AI cloud: ~$1.80 × 8,760 × 3 = $47,300

Kentino on-prem 8× Pro 6000 Server Edition: TCO ~€115,000 over three years for eight cards. Per-card €14,400. That is roughly a quarter of the cheapest cloud H100 reserved for inference-comparable throughput (Pro 6000 BW lands at ~60–70% of H100 SXM on non-NVLink-dependent inference workloads).

Break-even versus AI-cloud reserved H100: the on-prem 8× Pro 6000 box pays for itself at roughly 50% sustained utilization over three years. Below that, rent. Above that, own. This is the same math every serious on-prem buyer arrives at, and the reason on-prem inference is not dead in 2026 despite the cloud cost cuts.

Workload to GPU recommendation matrix

The honest take

Most Kentino customers walk in asking which card is fastest. For the workload most of them actually have — a 7B or 8B chat model, maybe a 32B reasoning model, maybe a vision model in the 7B–13B class — the right answer is L4 or 5090, not the flagship. The Pro 6000 Blackwell cost premium at our pricing is €5,000–€7,000 per card. At four cards per build, that is €20,000–€28,000 of VRAM the customer is paying for and not using.

The job is to ask which models, at which concurrency, at which context window — and size accordingly. Smaller deal, customer comes back when they scale.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.