The real cost of on-premise LLM inference
A structure for the comparison, and the three costs that most on-premise business cases leave out.
The on-premise versus API cost comparison is usually made badly in both directions. API advocates quote per-token pricing against nothing. On-premise advocates quote hardware cost against a year of API spend and declare victory.
Below is the structure we use. It will not give you a number — that depends entirely on your workload — but it will tell you which numbers you need and which ones people forget.
No prices here. GPU and token pricing move fast enough that any figure would be wrong by the time you read it. The structure is durable; plug in current numbers when you run it.
The API side is simple
Tokens in, tokens out, times price, times volume. Two things people miss:
- Retrieval inflates input tokens dramatically. A RAG system sends retrieved context with every request. Input volume is frequently 10–50× the user's actual question, and input tokens are where the cost sits.
- Retries, evaluation, and development traffic are real. Continuous evaluation against production traffic means you pay to check your own quality.
Model it on realistic context sizes rather than on the length of the user's question.
The on-premise side has five components
| Component | What to count |
|---|---|
| Compute | GPU purchase or reserved instance cost, amortized over the useful life you actually expect — not the accounting life. |
| Supporting infrastructure | Networking, storage for models and indexes, power, cooling, rack space. Frequently 15–30% on top of GPU cost for on-prem. |
| Platform engineering | Building and maintaining serving, scheduling, monitoring. Front-loaded, then ongoing. |
| Operations | On-call, incident response, capacity management, model lifecycle. This is the line that is almost always missing. |
| Opportunity cost | What the people doing the above would otherwise have built. |
The three omissions
Staffing is usually the largest line
Someone has to operate the cluster, manage the model lifecycle, respond at 2am, and keep evaluations current. At minimum that is a fraction of a platform engineer, ongoing, for as long as the system runs.
Over three years, that cost routinely exceeds the hardware it supports. If your business case has a hardware line and no staffing line, it is not a business case.
Utilization sets the real rate, not capacity
A GPU cluster at 15% utilization costs exactly what one at 85% costs. Enterprise AI workloads are bursty — heavy during business hours, near-idle overnight and at weekends — so effective cost per token is frequently several times the naive capacity calculation.
This is the single biggest error in on-premise business cases. The fix is to model against realistic duty cycles, and to plan multi-tenancy from the start so the investment is shared across teams.
Hardware depreciates against a moving target
You are committing capital on a three-to-five year cycle to a field that re-baselines annually. Model efficiency improves, serving techniques improve, and hardware generations land. The risk that your capacity plan is wrong in two years is real and belongs in the model rather than in a footnote.
Where the crossover actually sits
Directionally, and without pretending to precision:
- Low or highly variable volume — API wins clearly. Scale-to-zero is worth a great deal, and you are not paying for idle GPUs.
- Moderate, spiky volume — API usually still wins once staffing is counted honestly.
- High, sustained, predictable volume — on-premise or reserved private cloud starts to win, and the margin grows with utilization.
- Any volume, regulated data — the comparison is frequently moot, because the API was never an approvable option.
Choosing on-premise for cost reasons alone is usually wrong below a high sustained volume. Choose it for control, data boundary, or latency — and treat favourable economics at scale as a bonus rather than the justification.
Two things that change the maths
Optimization. Quantization, continuous batching, and right-sized models frequently reduce hardware requirements by a large factor. Many on-premise business cases are built on unoptimized inference and therefore price a cluster several times larger than needed.
Multi-tenancy. A cluster serving one use case is an expensive way to run one use case. The same cluster serving eight has a completely different cost profile — which is the actual argument for a platform rather than a series of point deployments.
How to run it
- Measure real token volumes including retrieval context, not estimates from user counts.
- Model API cost at that volume with current pricing, including evaluation traffic.
- Size on-premise hardware after optimization, not before.
- Add supporting infrastructure at a realistic percentage.
- Add staffing honestly, as an ongoing line for the full period.
- Model utilization from realistic duty cycles, not peak capacity.
- Run it over three years, and sensitivity-test the volume assumption in both directions.
If on-premise still wins after all of that, it wins. We have run this analysis and recommended the API more than once.
Related
AI for Manufacturing
AI for ManufacturingStuck at one of these?
The assessment diagnoses which constraint is actually blocking you, and produces a roadmap you own — executable by us, by another firm, or by your own team.
Direct response from an engineer. Typically within one business day.