Managed API
Inference runs on a vendor's infrastructure. Your data is processed by a third party under contract. Fastest to build on, no infrastructure to operate, and access to frontier models the moment they ship.
Private vs. Cloud AI
Where inference runs determines your data boundary, your cost curve, your latency floor, and which regulators you have to satisfy. An honest comparison.
This is a decision most organizations make by default rather than by analysis — usually by starting with whatever API was easiest, then discovering during security review that the architecture cannot be approved.
There is no universally correct answer. There is an answer that fits your data sensitivity, volume profile, latency requirements, existing infrastructure, and operational capacity. Below is how we evaluate it, including where each model genuinely loses.
We hold no reseller agreements and take no vendor commissions, so nothing on this page carries margin for us. We have recommended managed APIs to clients whose requirements pointed there.
The options
Five models
Private cloud
Models run inside your own cloud tenant or VPC — Bedrock, Azure OpenAI, Vertex, or open-weight models on your own GPU instances. Data stays within your cloud boundary and existing controls.
On-premise
Models run on hardware you own, in your datacenter. No external dependency for inference. The highest control, and the highest operational burden.
Edge
Inference at the point of use — plant floor, clinical device, retail, vehicle, field operations. Optimized models on constrained hardware, managed as a fleet.
Air-gapped
On-premise with no external connectivity at all. Model updates, dependencies, and monitoring all become offline logistics problems that must be designed for from the start.
Comparison
Side by side
| Managed API | Private cloud | On-premise | Edge | Air-gapped | |
|---|---|---|---|---|---|
| Data boundary | Third-party processor | Your cloud tenant | Your datacenter | Your device | Isolated enclave |
| Third-party in data path | Yes | Cloud provider only | None | None | None |
| Regulated data fit | Requires DPA & review | Often approvable | Fully controlled | Fully controlled | Fully controlled |
| Latency floor | Network-dependent | Low | Low | Lowest | Low |
| Cost model | Per token | Reserved + usage | Capital + operating | Capital per node | Capital |
| Cost at high volume | Scales linearly | Flattens | Flat | Flat | Flat |
| Frontier model access | Immediate | Provider-dependent | Open weights only | Small models only | Open weights only |
| Ops burden | Minimal | Moderate | High | High (fleet) | Highest |
| Time to first deployment | Days | Weeks | Months (procurement) | Months | Months |
| Scales to zero | Yes | Partially | No | No | No |
No row here is a verdict. A "high ops burden" is irrelevant if you already run a platform team, and "immediate frontier access" is worthless if legal will not clear the data path.
Framework
How we decide
Constraints first, preferences second. The regulatory question comes before the technical one, because it eliminates options rather than ranking them.
Economics
Total cost of ownership
Per-token API pricing looks cheaper than a GPU cluster until volume is sustained. But most on-premise business cases we review understate cost by omitting the same three things.
Staffing is the largest omitted cost
Someone has to operate the cluster, manage model lifecycle, respond to incidents, and keep evaluations current. That is a fractional platform engineer at minimum, and it usually outweighs the hardware line over three years.
Utilization, not capacity, sets the real rate
A GPU cluster running at 15% utilization costs the same as one at 85%. Enterprise AI workloads are bursty by nature, so the effective cost per token is frequently several times the naive capacity calculation.
Hardware depreciates against a moving target
You are committing capital on a three-to-five year cycle to a field that re-baselines annually. That risk is real and belongs in the model rather than in a footnote.
The honest summary: managed APIs win on cost until volume is both high and predictable. Below that threshold, choosing on-premise for cost reasons alone is usually wrong — choose it for control, and treat favourable economics at scale as a bonus.
Hybrid
Routing by data classification
These models are not mutually exclusive. The most cost-effective architecture we deploy for mid-size enterprises routes by data classification: sensitive workloads to private infrastructure, everything else to a managed API.
A classifier or policy layer at the gateway inspects the request, decides which tier may handle it, and enforces that decision. Public-facing content generation goes to the frontier model. Anything touching regulated records never leaves the boundary.
It costs more to build than either pure model, and it is worth it when you have genuinely mixed workloads. It is not worth it when 90% of your traffic falls on one side of the line — in that case, build for the majority and handle the remainder manually.
Candour
Where each model loses
Every comparison table has a bias. Here is ours, stated explicitly.
Managed API
The data-processor relationship is unresolvable for some regulators regardless of contract terms. Costs scale linearly and forever. You inherit the provider's deprecation schedule, rate limits, and outages. Model behaviour can change beneath you without notice.
Private cloud
Still a tenancy trust decision — you have moved the boundary, not removed it. Some auditors do not distinguish it meaningfully from a managed API. GPU capacity in-region can be genuinely hard to reserve.
On-premise
Capital risk against fast-depreciating hardware, months of procurement lead time, and a real operational burden that lands on a team that may not exist yet. Locked to open weights, so the hardest reasoning tasks stay out of reach.
Edge
Hard model size constraints force real capability trade-offs. Fleet management across hundreds of nodes is its own engineering discipline. Debugging a failure on a device you cannot reach is materially harder than debugging a server.
Air-gapped
Every update becomes a logistics exercise with a change-control process attached. No vendor telemetry means you build your own observability. Expect the slowest iteration cycle of any model here, by a wide margin.
Questions
Frequently asked
Can we start with an API and move to private later?
Yes, and it is often the right sequence — prove the use case cheaply, then move it once value is demonstrated. The migration is real work but not a rewrite, provided you build behind an abstraction from the start and keep your evaluation harness portable.
What does not work is proving a use case on data you were never permitted to send to an API in the first place. That is not a pilot, it is an incident.
Are open-weight models good enough?
For retrieval, extraction, classification, summarization, and most agentic workflows, yes. For the hardest reasoning tasks, frontier models still lead.
We evaluate against your actual workload rather than public benchmarks, because benchmark rank rarely predicts performance on a specific enterprise task.
How much GPU capacity do we actually need?
It depends on concurrency, token throughput, model size, and acceptable latency — not on user count, which is the figure most estimates start from. We model it from measured workload characteristics during the assessment.
Does private deployment mean worse quality?
Usually the constraint that matters is retrieval quality, not model quality. A well-engineered retrieval layer over an open model routinely outperforms a frontier model with a naive pipeline on enterprise tasks.
Which model fits your constraints?
The assessment models the realistic options against your data sensitivity, infrastructure, volume, and regulatory position — and recommends one, with the cost case for each.
Vendor-neutral. We hold no reseller agreements and take no vendor commissions.