Cost per Million Tokens: On-Prem vs Cloud

There are three honest ways to buy LLM inference in 2026: call somebody else's hosted endpoint and pay per token (cloud LLM API), rent a GPU by the hour and run your own model (cloud GPU rental), or buy a server and run it in your own rack (on-prem). The price of a million tokens looks dramatically different across the three, and not in the order most buyers assume.

This article works the math in EUR ex VAT. Audience: somebody deciding whether to keep paying OpenAI monthly, switch to a hosted open-model provider, rent an H100, or buy a 4-/8-GPU box. We quote May 2026 prices and end with a five-minute decision flow. Cross-references: T01 per-GPU TCO, W07 card selection, I04 wall-side numbers.

The three platforms

Cloud LLM API. OpenAI, Anthropic, Google, Together, Fireworks. Pay per token in and per token out. No idle cost, no capex, no ops. Also no model choice beyond their catalogue, no LoRA, no quant control.

Cloud GPU rental. AWS p5/p5e/p6, Azure ND H200/B200, GCP A3 Ultra, plus specialist clouds (Lambda, CoreWeave, Nebius, RunPod). Rent a GPU instance by the hour and run anything. You bring the model, the serving stack, the ops. You pay whether you serve one token or one billion that hour.

On-prem. Buy a 4-GPU or 8-GPU box. Rack it, your building or a colo. You bring power, cooling, networking, ops. Amortize over three years. Marginal cost of a token is electricity. Fixed cost is everything you already paid.

Three different cost shapes — pure variable, pure fixed-with-cliff, capex-plus-marginal — that break even against each other at different points along the throughput axis.

Cloud LLM API pricing

USD per million tokens, public pricing pages, May 2026. EUR at EUR/USD ≈ 1.08.

Two honest reads. Frontier closed models (Claude Opus 4.7, GPT-5, Gemini 2.5 Pro) are an order of magnitude more expensive than hosted open-model APIs serving Llama 70B or DeepSeek-V3 — the price of "smartest now" versus "best open weights now." For most production workloads (RAG, agent steps, code, classification), Sonnet 4.6 or GPT-5 mini is the operating point, not the flagship. Output tokens cost three to seven times input on closed providers and roughly equal on open-model hosts. Hyperscaler open-model SKUs (Bedrock, Azure) charge a 30–80% premium over Together or Fireworks for the same model — you pay for IAM integration and a single bill, not cheaper tokens.

On-prem cost-per-Mtok

Cost-per-Mtok is three-year total cost divided by tokens served over three years. Denominator depends on utilization; numerator is roughly fixed.

On-prem €/Mtok =
    (capex + 3y electricity + 3y maintenance + 3y ops) /
    (sustained tok/s × utilization × 3 × 8,760 × 3,600 / 1e6)

Plugging in per-GPU TCO from T01 at 60% utilization, €0.20/kWh, PUE 1.4:

Drop utilization to 30% and every figure doubles. Push to 80% and L4-served 8B lands at €0.09/Mtok, cheaper than every cloud API for that class. Math collapses below 25% utilization because TCO is dominated by capex you paid whether or not you used.

The break-even

Overlay the cloud line. Llama 70B at 100 M tokens/month, balanced 50/50, on Together: 100 × €0.81 = €81/month → €2,916 over three years. Against a 4× 5090 on-prem at €27,000, Together wins by €24,000. At 100 M tokens/month, on-prem is wildly wrong.

Push to 1 B tokens/month: 1,000 × €0.81 = €810/month → €29,160 over three years. Together and the 4× 5090 are within €2,000. At 1.5 B/month, on-prem wins materially.

Break-even for 70B-class inference vs the cheapest open-model hosted API lands around 1.0–1.5 B tokens per month sustained on a 4× 5090. For closed frontier APIs (Claude Sonnet 4.6 at €8.34/Mtok blended, GPT-5 at €5.21/Mtok blended), break-even moves to 80–150 M tokens/month — the cloud line is an order of magnitude steeper.

That is the most important number in this article: break-even is not "100 M tokens/month for everything." It is workload-specific, and depends entirely on which cloud line you are competing with. Buyers who say "we hit 50 M, we should go on-prem" are usually comparing against Claude Opus, where they are right — ignoring that a workload move from Opus to Sonnet 4.6 saves them five times more than a move from Sonnet to on-prem at that volume.

Cloud GPU rental — the middle option

Hourly rates per GPU, May 2026 on-demand:

Reserved 1-year commits take 25–35% off. AWS p5 on-demand is roughly 3× Lambda for the same H100 silicon. Translate to per-token: 8× H100 at Lambda running Llama 3.3 70B FP8 at ~3,500 tok/s sustained → $16/hr × 8,760 × 3 = €389,000 over 3y → €1.96/Mtok at 60% utilization. That is 4× an 8× Pro 6000 SE on-prem build (€0.53/Mtok) and 2.5× just calling Together (€0.81/Mtok).

Renting H100 to serve open-model inference is the worst of both worlds: rental rates for the box and ops to run it. The math only works when you need a specific model nobody hosts and lack the volume for on-prem. Cloud GPU rental does have one honest use case: burst training. A LoRA fine-tune on 8× H100 for 8 hours costs ~$130 at Lambda. No on-prem TCO beats that for a one-off. The logic does not transfer to inference, which is sustained and runs 24/7. The "spot" trap: spot/preemptible/community tiers offer 50–80% off but are interruptible. Fine for stateless inference replicas. For an 18-hour training job that checkpoints every step, two interruptions double your cost-per-completed-step and you are no longer cheaper than on-demand.

Hidden costs nobody quotes

Cloud: egress at $0.05–0.09/GB hits when you ship images, audio, or video back. Logging adds 5–10% at scale. Re-billed context is the worst — every API charges full prompt every call, so a 50K-token agent context costs 50K input tokens per step of a 20-step loop. Prompt caching (Anthropic, OpenAI, Google all support it) cuts this 50–90% on cached prefixes, but only if you architect for it. Production rate limits start at 30K–500K TPM and require usage history; new customers cannot run at scale on day one. EU-region SKUs are 10–20% more than US East with lower availability.

On-prem: ops time at 0.1–0.3 FTE per server is €8k–€24k/year not in the per-Mtok math. Downtime needs a redundant box or graceful cloud fallback. Electricity is not €0.20/kWh everywhere — EU industrial 2026 ranges from €0.09/kWh (Sweden, Norway hydro) to €0.35/kWh (Italy, Ireland), Germany at €0.18–€0.22/kWh on subsidized industrial, Czechia at €0.18–€0.25/kWh open commercial. For Italy or Ireland, every electricity row in T01 goes up 50%. Colo rack is €100–€300/month for 4U — €3,600–€10,800 over three years on top of hardware.

Published cloud price is roughly 1.2–1.5× the actual bill; published on-prem TCO is roughly 1.2–1.5× sticker. Multipliers cancel for the comparison.

Non-cost factors

Three things override cost-per-Mtok for real buyers.

Data sovereignty. The biggest reason European buyers go on-prem in 2026 is GDPR plus sectoral rules (DORA, NIS2, the AI Act, MDR). Sending production prompts with personal data, medical records, or industrial secrets to a US-hosted endpoint is a contractual problem regardless of price. EU-region SKUs do not mean EU-controlled. For a regulated buyer, the on-prem premium is not a cost, it is the price of being legally allowed to deploy.

Latency floor. WAN to the closest hosted endpoint adds 15–40 ms RTT inside the EU and 80–120 ms transatlantic. On-prem on a 10 GbE LAN is sub-millisecond. For interactive chat the WAN delta is invisible against 800 ms TTFT. For a robotics control loop (I01) or a sub-300 ms voice agent, WAN is fatal.

Customizability. Cloud APIs ship the model and quant the provider chose. On-prem you pick the model, quant, serving stack (vLLM, SGLang, llama.cpp), LoRA adapters, speculative decoder. For research labs or specialized products this is what you are buying — cost-per-Mtok is secondary.

If none apply, the buyer belongs on cloud below break-even. Not every prospect should buy a server.

Worked decision: 70B-class workload at two volumes

Customer profile: SaaS doing document classification and summarization, 70/30 input/output, open-weight 70B-class model.

At 100 M tokens/month, Bedrock wins by an order of magnitude over on-prem; the customer has not earned the box. At 2 B tokens/month, on-prem wins decisively by 1.8–2.2× even against the cheapest hosted alternative. Break-even for this workload landed around 800 M – 1.2 B tokens/month. Below that, rent. Above that, own.

Decision matrix

Conservative — a regulated buyer pushes the on-prem column one row earlier; an experimental buyer one row later. The frontier column is special: no on-prem build replaces Claude Opus 4.7 or GPT-5. Those are quality SKUs you call when needed, not workloads you host.

The honest take

Most prospects who walk in thinking they need a server do not. At typical volumes — 5–50 M tokens/month for an early product, 50–500 M for mid-stage — cloud APIs win for another twelve to twenty-four months. The right answer is "use Together or Bedrock, watch your monthly spend, call us when it crosses €1,500/month sustained."

Buyers for whom on-prem makes sense in 2026 fall into three shapes. Regulated: data cannot leave the building. Sustained at scale: 1 B+ tokens/month on a 70B-class model where the math wins by 2–3×. Robotics or low-latency: WAN floor breaks the application. About 30–40% of our prospects fit one. For the other 60–70%, cloud APIs in 2026 are a remarkable product and they should keep using one until the math changes.

Training is a different conversation. Almost every workload at K-AI scale (LoRA, QLoRA, fine-tune of 7B–32B) is cheaper on-prem than cloud GPU rental — the box runs 100% during the run. The exception is hyperscale training (full pretraining of 70B+, RLHF at scale) where you need 8× H100 with NVLink, which we do not sell (W07 covers why). Rent it, run it, bring the weights home.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.