Sustained vs Burst Inference Economics: When On-Prem Wins, When Cloud Bails You Out

A four-GPU inference server burns the same electricity, occupies the same rack space, and depreciates on the same schedule whether it serves twenty million tokens a day or twenty thousand. The cards do not know they are idle. The accountant does. This single asymmetry — fixed cost on the on-prem side, marginal cost on the cloud side — is the entire reason the right answer to "should we go on-prem?" depends as much on the shape of your traffic as on the volume.

T01 worked the per-card math at a single utilization point. T02 maps cost-per-million-tokens against cloud APIs at full load. This article (T03) is the one your CFO actually needs to read: what happens to those numbers when traffic is bursty, when it is sustained, and how to architect a system that does not over-spend on either side.

The audience is somebody who has already concluded the workload might fit on-prem and now needs to decide how much on-prem, and what bails out the rest.

The fixed-cost trap

Cloud inference scales cost with usage. A single Llama 70B call on an OpenRouter provider costs you roughly €0.50–€2.00 per million tokens (varies by tier and provider). Send no traffic, pay nothing. Send 10× the traffic, pay 10×. The relationship is linear and starts at zero.

On-prem inference is the opposite. A 4× RTX Pro 6000 Blackwell box (T01 puts the 3-year TCO around €58k all-in — capex amortised, electricity, chassis share, maintenance) costs that €58k whether it serves one million tokens or one trillion. The marginal cost of the 200-millionth token in a month is the same as the marginal cost of the second: the kWh of electricity it took to compute it and nothing more. Capex is a sunk cost the day the box ships.

The accounting consequence is brutal and rarely internalised by buyers:

(Same hardware, same TCO of €14.4k per card over three years; only the denominator changes.)

The cloud rate for Llama 70B FP8 in 2026 sits in the €0.50–€1.50/Mtok band depending on provider tier. Read the table again with that in mind. At 5% sustained utilisation the on-prem box is more expensive than every cloud option that exists. At 30% it ties the most expensive providers. At 60%+ it walks away with the math.

The trap is that customers benchmark capex against current load, then discover six months later that current load was 8% of the box. The hardware did nothing wrong. The sizing did.

What real production utilisation looks like

We have public data plus our own bench across the K-AI installs we operate or visit, and the picture is consistent across the industry in 2026:

The headline numbers from production deployments and operator surveys: most teams running their own inference report 20–40% average GPU utilization, with mature setups creeping toward 40–65%. Anything above 65% sustained is either pure batch work or a cluster running hot enough that the next traffic spike will queue. Below 20% means the box is running mostly to pay for itself, not to serve users.

The number that always surprises buyers: an interactive workload — a chatbot, a coding assistant, a customer support bot — almost never sustains more than 30%. The reason is daylight. Even a global product has off-hours; a regional product has 14 hours of low load every day. The math punishes that hard.

Sustained workloads — the bread and butter

The on-prem economics work cleanly for any workload where you can keep the GPUs working most of the day. In rough order of how often we see them on K-AI installs:

• Coding assistant / IDE backend for a team of 30+ engineers. Steady weekday load 09:00–18:00, lighter evenings, near-zero weekends. Average 20–30% across the week, peaking 60–80% during the workday.
• Customer support / sales chatbot with 24/7 traffic. Diurnal pattern follows the served region; global products land 25–35% sustained, regional 15–25%.
• RAG / semantic search backend indexing and serving an internal knowledge base. Query rate is moderately steady; reindexing is a scheduled overnight job that pegs the box at 95%. Combined utilisation 30–50%.
• Batch processing — overnight summarisation of yesterday's documents, weekly autolabeling runs, monthly classification re-runs over a corpus. Pure queue work, 70–90% utilisation while the queue is non-empty, designed to run to completion.
• Robot fleet inference — multiple units pulling off the same VLM endpoint during operating hours, idle outside (I01, I06).

The pattern is that sustained workloads either span enough timezones to fill the day, or have batch components that intentionally fill the gaps. The "fill the gaps" technique is the single biggest TCO lever a customer can pull and the most overlooked one. A box that does interactive chat 12 hours a day at 30% and runs autolabeling overnight at 90% averages out around 60% — twice the effective utilisation, half the per-token cost.

Burst workloads — where on-prem hurts

The opposite shape is a workload that needs huge capacity for a short window and almost nothing the rest of the time. Examples we see regularly:

• Marketing campaign / content generation push. Generate variants for an ad campaign across N markets in 48 hours, then nothing for a month.
• Viral content moments. A consumer app launches a feature, gets featured on a tech newsletter, traffic 50× normal for six hours, back to baseline by the next morning.
• Demo / pilot events. A trade show booth running live inference demos for three days, then back to zero.
• Periodic batch jobs that must finish in a window. Monthly compliance scan over an entire document corpus that has to complete by a regulatory deadline.
• Election-night or live-event analytics — a window of 12–48 hours where the volume is 100× normal.

Worked example. A workload needs 10 million tokens generated. Spread across 24 hours that is roughly 116 tok/s aggregate — a single L4 handles it. Compressed into a one-hour window it is 2,778 tok/s — needs a 4-GPU Pro 6000 box running flat out, or roughly six L4s.

So the same 10 million tokens demands either one card 24/7 or six cards for one hour. For burst workloads, on-prem capacity sits idle 23 of every 24 hours. The TCO math collapses: that six-card box at 4% utilisation costs €13–25/Mtok in the equivalent of the table above. A cloud burst at €1.50/Mtok serves the same load for €15 total. The break-even is not even close.

The unavoidable conclusion: trying to size on-prem for peak burst is the most common over-spend mistake we see. Customers want the box to cover the worst-case Tuesday afternoon and so they buy 4× the GPUs they need on a normal week. The bill arrives in the form of capex they cannot recover and a 90% idle fleet.

The hybrid pattern

The honest architecture for a real production workload with both shapes is on-prem baseline plus cloud burst overflow.

Hybrid routing: on-prem first, cloud overflow only when queue depth crosses the configured threshold.

The right routing logic is not "send X% to cloud, rest to on-prem." It is on-prem first, cloud only when on-prem is at capacity. The signal is queue depth or KV-cache utilisation on the vLLM side (K03 covers the metrics), not a static split.

Concrete recipe with vLLM:

1. Front the on-prem cluster with a router that exposes /v1/chat/completions. NGINX, HAProxy, the vLLM Router, or a small custom gateway all work.
2. Expose vllm:num_requests_waiting as a queue-depth gauge. Threshold at, say, 8–16 waiting per replica before fallback kicks in.
3. On overflow, route to a cloud endpoint with the same model name. OpenRouter / Together / Fireworks all serve Llama-class models. Pin one as primary and one as backup so you don't go down with a single provider.
4. Tag overflow requests in your observability so you can see, daily, what percentage you are paying out on. If it is above 15% most days, your on-prem is undersized; if it is below 1% for weeks, on-prem is oversized.

The economic logic: pay capex once for the load that is always there, pay variable cloud rates for the spikes. Done well, this gets effective utilisation on the on-prem box to 50–70% (the routing threshold is what you tune for) while keeping tail-latency commitments intact during bursts.

Cold start: the autoscaling reality

The naïve cloud-style answer to bursty load is "autoscale the on-prem cluster" — keep replicas at zero when idle, spin up on demand. This breaks badly for LLM serving for one reason: model load time.

Loading Llama 70B FP8 (~75 GB of weights) from a Gen5 NVMe drive takes 30–60 seconds of pure I/O at theoretical 12–14 GB/s read speeds. Add CUDA kernel compilation, vLLM scheduler warmup, KV cache preallocation, and a fresh replica is ready to take its first request 60–120 seconds after the pod starts. Smaller models help proportionally — an 8B fits in 5–10 seconds — but anything north of 30B is in the "users will time out" range.

For batch workloads, this is fine. For interactive workloads, it is fatal. Two minutes of latency on the request that triggered the scale-up event is unacceptable.

The mitigations, in order of effort:

• Keep N replicas always warm. The minimum that holds your average load. Scale up on demand, never down to zero. The minimum-warm pattern is what mature operators build entire products around.
• vLLM Sleep Mode. Newer vLLM versions support a "sleep" state where weights stay in GPU memory but the engine releases compute resources. Wakeup is 18–20× faster than a fresh load. Useful when you have multiple models that share GPUs but only one runs at a time.
• Pre-warmed model cache on shared NVMe / NFS. Pull weights once to a fast local cache, all replicas mount it. First-load happens once at cluster startup; subsequent replicas come up in seconds.
• Time-of-day scheduled scaling. Pre-scale up at 08:30 in anticipation of 09:00 traffic. The simplest and most effective pattern for predictable diurnal workloads. CronJobs setting kubectl scale are unfashionable but they work.
• NVIDIA NIM Operator with prebuilt TRT-LLM engines. Engines built ahead of time, cached on disk, loaded on demand. Faster cold start than fresh compilation but still in the tens of seconds for 70B.

For the K-AI customer base — small fleets, predictable usage patterns — the right answer is almost always N warm replicas + scheduled scaling + cloud burst overflow. Scale-to-zero is for serverless platforms where someone else eats the cold start.

Autoscaling: HPA based on what?

Kubernetes HPA defaults to CPU and memory metrics. Both are wrong for GPU inference. A vLLM pod can be at 5% CPU and 100% overloaded, or 95% CPU and serving fine. The metric you actually need is one of:

The right tool to wire this into Kubernetes is KEDA (Kubernetes Event-Driven Autoscaling), which can consume Prometheus queries directly and feed them to an HPA. The vLLM production stack ships first-class KEDA integration, and the same pattern works with KServe on OpenShift AI.

A working ScaledObject for queue-depth scaling:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama70b
spec:
  scaleTargetRef:
    name: vllm-llama70b
  minReplicaCount: 2     # always-warm floor
  maxReplicaCount: 6     # cap before cloud overflow
  pollingInterval: 15
  cooldownPeriod: 300    # don't thrash on noisy queue
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      query: avg(vllm:num_requests_waiting)
      threshold: '5'     # avg 5 waiting per replica → scale up

Two production lessons we have collected the hard way:

1. cooldownPeriod matters as much as the trigger. A noisy queue causes scale-up/scale-down thrashing, and every scale-up pays the cold start. 5 minutes is a sane minimum.
2. minReplicaCount should be sized for the trough, not zero. The two pods sitting idle at 03:00 are the two pods that handle the 09:00 spike without timing out.

Scale horizontally vs scale vertically

A subtle architectural choice when sizing: do you buy more boxes (horizontal) or more GPUs per box (vertical)?

For inference workloads with bursty traffic, horizontal almost always wins. Two 4-GPU boxes can run as four DP=2 replicas, take a hit on one box without losing service, and let you scale capacity in halves rather than whole units. Vertical is the right call for training and for very large single-model inference where TP needs the GPUs in one electrical domain.

The "GPU is idle but I'm paying for it" psychological trap

Customers visiting the install for the first time often pull up the Grafana dashboard, see GPU utilisation at 18% at 14:30 on a Tuesday, and ask why they paid for hardware that does nothing 80% of the time.

The honest answer is that for any latency-sensitive workload, some idle is the price you pay for low-tail-latency response. A box at 95% sustained utilisation has a queue. A queue means request latency. The customer who wants 300 ms time-to-first-token does not also get 95% utilisation on the same box. Pick one.

The trade-off that can be optimised is moving fixed work into the idle windows:

• Batch jobs (autolabeling, document extraction, embedding generation for new content) scheduled overnight when the chatbot is quiet.
• Fine-tuning runs on weekends when no users are on.
• Periodic re-indexing of vector stores during early morning hours.

This is the discipline of treating the on-prem box as a dual-purpose asset: latency-sensitive interactive serving in the day, batch work overnight. Done well, the same hardware that posts 25% utilisation on the interactive dashboard posts 65% utilisation on the all-workloads dashboard. The CFO sees the latter.

What is not a fix: artificially cranking the request rate to "look busy." Customers occasionally try this. It makes tail latency worse, makes monitoring useless, and does not change the bill.

The decision matrix

The shape of the workload, more than the volume, dictates the right architecture. Map your situation to the closest row:

Two things this matrix gets right that ad-hoc planning gets wrong:

• Pure burst workloads belong in the cloud. Buying hardware for a workload that runs four hours a month is malpractice.
• The "sustained + burst" hybrid almost always wants both. Trying to pick one or the other usually picks wrong.

The honest take

Most Kentino customers should plan for 30–50% sustained utilisation on on-prem and use cloud overflow for everything above that. The customers who get to 60–80% on-prem utilisation are the ones running batch work alongside interactive — autolabeling overnight, fine-tuning on weekends, embedding generation in the small hours. The customers who sit at 10–15% utilisation are the ones who sized for peak.

The single most expensive mistake is buying for the worst-case Tuesday afternoon. The second most expensive is going pure cloud when sustained load would pay capex back in 14 months. The right answer is almost always in the middle: a smaller on-prem box than the customer initially specced, with a cloud overflow path for the spikes.

Cross-check the math: at €0.50/Mtok cloud rates and a 4-GPU Pro 6000 box costing €58k over 3 years, the box pays for itself at roughly 14 billion tokens served. Spread evenly that is ~150 tok/s aggregate sustained — well within reach of the box at 25–35% utilisation. Below that volume, rent. Above it, own. This is the same break-even point T01 and T02 reach from the per-card and per-token angles.

---

This is part of the Kentino Wiki, a reference series on AI compute, robotics, and the systems that connect them. Comments and corrections welcome at info@kentino.com.