AI TCO Calculator: 4 Ready Scenarios, Cloud vs On-Prem

Reading time: ~9 min · Cluster: vendor evaluation · Level: CIO / CFO

I covered the method of computing AI TCO separately: the full worked example and formulas and how to calculate over a 3-year horizon. This post is different: it is a library of ready scenarios. Instead of teaching the formula, it shows four typical company profiles priced with the same rates, so you can recognize yours and see the verdict at once. For your own decision, plug your numbers into the interactive TCO calculator.

Every scenario uses one open-weight model in the 70B class, a 3-year horizon and the same indicative 2026 rates: managed API at about 0.60 USD per million tokens, GPU rental at about 3 USD per hour, an on-prem node with two H100-class cards at about 285k USD of all costs over 3 years. The numbers show proportions, they are not a price list.

Table of contents

• Four variables that flip the result
• Scenario 1: pilot, low volume
• Scenario 2: mid-size service desk with heavy RAG
• Scenario 3: high 24/7 load, many workflows
• Scenario 4: moderate volume, data cannot leave
• Matrix: recognize your scenario
• The lever that flips the verdict
• What this post does not cover
• Disclosure and biases

Four variables that flip the result

Before you look at the scenarios, it helps to know what separates them. TCO is always flipped by the same four variables: volume of queries per month, context length per query (in RAG it is context, not the question, that drives the per-token bill), availability mode (round-the-clock vs working hours) and model class. The GPU price, counter to intuition, matters least. The four profiles below differ precisely on these variables, which is why they yield four different verdicts.

Scenario 1: pilot, low volume

A company tests one workflow, say ticket classification. Volume 8,000 queries per month, light context of 4,000 tokens per query, model available in working hours only.

Billing model	Compute (3 years)	Rest (3 years)	Total (3 years)
Managed API per token	~700 USD	~20,000 USD	~21,000 USD
GPU rental (working hours)	~45,000 USD	~100,000 USD	~145,000 USD
On-prem CAPEX plus OPEX	CAPEX 95,000 USD	~190,000 USD	~285,000 USD

Verdict: managed API, overwhelmingly. At this volume compute is pennies, and the whole cost is governance and integrations you would incur anyway. Owning hardware here is wasted capital: the card sits idle most of the day. Consider on-prem only once the pilot proves the volume, or when data cannot leave from the start (see scenario 4).

Scenario 2: mid-size service desk with heavy RAG

An internal service desk over technical documentation, the model answers from long context. Volume 150,000 queries per month, heavy context of 12,000 tokens per query, availability in extended working hours.

Billing model	Compute (3 years)	Rest (3 years)	Total (3 years)
Managed API per token	~39,000 USD	~70,000 USD	~109,000 USD
GPU rental (near-continuous)	~158,000 USD	~102,000 USD	~260,000 USD
On-prem CAPEX plus OPEX	CAPEX 95,000 USD	~190,000 USD	~285,000 USD

Verdict: API is still cheapest, but the threshold is closing. Heavy RAG context sharply raises the per-token bill, because you pay for every retrieval token. If context or volume doubled again, API and on-prem would start to meet. At this level, choosing on-prem is still a decision about data control and compliance, not price.

Scenario 3: high 24/7 load, many workflows

AI is the backbone of operations: service desk plus instruction generation plus quoting support, round the clock, across several plants. Volume 1.8 million queries per month, context 9,000 tokens, 24/7 mode.

Billing model	Compute (3 years)	Rest (3 years)	Total (3 years)
Managed API per token	~350,000 USD	~63,000 USD	~413,000 USD
On-prem (scaled, possibly 2 nodes)	CAPEX ~150,000 USD	~170,000 USD	~320,000 USD

Verdict: on-prem wins on both fronts at once. Above the threshold, which falls near 1.5 million queries per month sustained for three years, the marginal cost of owned hardware beats per-token billing, and the regulatory argument points the same way. A capacity note: this volume at steady concurrency may need more than one node, so CAPEX rises, but the bill still lands below API.

Scenario 4: moderate volume, data cannot leave

A NIS2 essential entity, documentation covered by trade secrecy that must not leave the perimeter. Moderate volume of 60,000 queries per month, context 9,000 tokens, working hours.

Billing model	Total (3 years)	Note
Managed API per token	~75,000 USD	Cheapest, but data leaves the perimeter
On-prem CAPEX plus OPEX	~285,000 USD	About 4x more, data stays with you

Verdict: on-prem, even though it is more expensive. This is the scenario where TCO is not the only criterion. The roughly 210k USD difference over three years is not a loss, it is the countable price of data control. The value of this calculation is that it turns the decision from "on-prem or cloud" into "data control costs us this much, and we choose it deliberately." On how to avoid locking yourself to one vendor while doing so, see the post on preventing vendor lock-in.

Matrix: recognize your scenario

Profile	Volume / month	What usually wins on cost	When the verdict changes
Pilot, light context	below 50,000	Managed API, decisively	Almost never at this volume
Mid service desk, heavy RAG	100,000 to 500,000	Still API	Heavy context or a frontier model brings on-prem closer
High 24/7 load	above 1.5M, sustained	On-prem CAPEX	Below the threshold API returns
Regulated, data cannot leave	any	On-prem (despite higher cost)	If data may leave, the cost calculus returns

The lever that flips the verdict

Each scenario is flipped by one variable, not the GPU price.

In the pilot it is flipped by volume: until it grows, no owned hardware amortizes. In the mid service desk it is flipped by context length: doubling RAG context doubles the per-token cost but does not move on-prem, so heavy retrieval brings the owned-hardware threshold closer. In the 24/7 scenario it is flipped by availability mode: if the model did not have to run at night, GPU rental in working hours would beat on-prem for a long time. And in the regulated scenario the verdict does not depend on cost at all, only on whether the data may leave.

Practical takeaway: before you compute anything, settle these four variables for yourself. Only then does the calculator turn an indicative scenario into your decision.

What this post does not cover

• The derivation of formulas and the full method, which live in a separate post on methodology and a worked example.
• Specific API operator pricing and GPU rental rates, because they change quarterly. I give orders of magnitude, not a price table.
• Per-token cost at the hardware level, and card sizing for a specific model.
• Full regulatory mapping of NIS2 and the AI Act.

// disclosure & biasesDisclosure and biases

I write from the view of someone working on AI systems run outside the public cloud, so I lean toward emphasizing data control. I tried to balance that: in three of the four scenarios on-prem loses on pure cost, and I say so plainly. All numbers are indicative, based on public market rates and tariffs from 2026, not on a specific deployment. For your own decision, plug in your own rates and real volume. This is not financial or investment advice.

Related notes

• Methodology: AI TCO calculator: methodology, worked example and FAQ — the formulas and a full derivation of one scenario.
• How to calculate: On-prem AI vs cloud TCO: how to calculate over 3 years — the three cost models and the break-even point.
• Lock-in: How to prevent AI vendor lock-in: clauses and exit plan — how not to lock yourself to a vendor when you choose cloud.
• Sizing: GPU sizing for Llama 3.1 70B inference: numbers from benchmarks — the hardware behind the on-prem scenario.