AI TCO calculator: methodology, a worked example and FAQ

Reading time: about 13 minutes. Cluster: vendor-eval. Author: Fryderyk.

The short answer first

An AI TCO calculator answers one question: at what level of utilisation your own hardware starts to be cheaper than the cloud in your specific case. To answer it sensibly, it has to cost three scenarios separately, with the same cost categories, over the same three-year horizon: a managed API billed per token, GPU rental billed per hour, and on-prem CAPEX with full OPEX.

The verdict the numbers produce is uncomfortable for both camps. For a moderate volume, on the order of tens of thousands of queries a month on an open-weight model, the cheapest option on pure cost is the managed per-token API, and by a wide margin. On-prem then earns its keep on data control and compliance, not on price. GPU rental only catches up to on-prem in a round-the-clock setup. And on-prem CAPEX only starts to beat per-token above roughly 1.5 million queries a month sustained across a full three years. Below I lay out the exact methodology and the full worked example that produces this.

This piece is a companion to the calculator and an extension of the note TCO of on-prem AI vs cloud: how to calculate over a 3-year horizon. That one explains why the price of the card is the least important number. Here we run one concrete scenario from inputs to result.

Contents

1. What a TCO calculator must cost, and what it usually skips
2. Methodology: three formulas you must not mix
3. Inputs: what you feed the calculator
4. Worked example: a manufacturing service desk, 3 years
5. Side by side: three scenarios
6. The crossover point and sensitivity
7. Why the naive calculation falls apart
8. How to read the result
9. FAQ
10. Disclosure and biases
11. What I do not cover here
12. Related notes

What a TCO calculator must cost, and what it usually skips

Most AI TCO calculators online cost one thing: compute. The price of a card, or a per-token rate, multiplied by time. That produces a number that looks concrete and is wrong, because it omits 30 to 60 percent of the real bill.

An honest TCO calculator has to cover five categories on each side: compute, energy and infrastructure upkeep, people, compliance, and project plus exit costs. Compute is the easiest to cost and the least differentiating. The real difference between on-prem and cloud sits in people and in compliance, and those items are exactly the ones that fall out of the calculation, because they do not have a convenient unit rate.

The second thing the calculator must police is a uniform horizon. Three years on both sides, because that is the typical hardware amortisation period and roughly how long you plan a production workflow for. A shorter horizon inflates the cloud advantage, a longer one inflates on-prem. You fix the horizon before the numbers, not after the fact to fit the answer you wanted.

Methodology: three formulas you must not mix

The word cloud covers two entirely different billing models, and on-prem is a third. Mixing them is the single most common reason a calculation does not survive a meeting with the CFO. Three formulas below, each on its own.

Formula 1. Managed API, per token.

Compute cost (month) = number of queries × average tokens per query × price per token

You count tokens end to end: input plus output, including the entire retrieval context. In RAG use cases it is the context, not the question itself, that makes up most of the bill. On top of compute, add vector storage, egress, and the cost of governance and due diligence of the external vendor.

Formula 2. GPU rental, per hour.

Compute cost (month) = number of cards × working hours per month × hourly rate

The key variable is working hours. A model available around the clock also pays for the hours when nobody is asking. A model exposed only during plant working hours pays a fraction of that. On top of compute, add storage, egress, an IP address and, as with the API, the cost of governing the external vendor. You still stand up and maintain the model yourself, so a fraction of a headcount comes with it.

Formula 3. On-prem, CAPEX plus OPEX.

TCO (3 years) = CAPEX + (annual OPEX × 3) + project costs − residual value

You amortise CAPEX, you carry OPEX every year, project costs are one-off, and after three years the hardware retains a value you subtract. The marginal cost of the next compute hour is the lowest of all three models here, but every unused hour is wasted capital.

Selection rule: do not compare on-prem CAPEX with per token, because you are measuring two different things, capital versus consumption. First decide which cloud model is your real alternative. If you would stand up the model yourself anyway, your reference is GPU rental. If a ready-made API would do, the reference is per token.

Inputs: what you feed the calculator

For the result to be defensible, the calculator needs eight input numbers. Four describe the load, four describe your conditions.

Load: number of queries per month, average input tokens per query (with the RAG context), average output tokens, availability mode (round the clock or working hours only). Conditions: your business energy rate on your tariff, the fully loaded cost of an engineering headcount in your organisation, the required level of audit and pen-testing, and the model class that actually does your job.

That last number is the most important and the most often overestimated. Many production workflows, ticket classification, extracting data from documents, service-desk answers grounded in internal documentation, are handled by an open-weight model in the 70B class, not the largest available frontier model. That shifts the whole bill, because a cheaper model means a lower per-token rate and smaller on-prem hardware.

Worked example: a manufacturing service desk, 3 years

We take one concrete scenario and run it through all three formulas. Every number is indicative, based on public 2026 market rates, and serves to show the method. For your own decision, substitute your own rates.

Assumptions:

• A manufacturing company, an internal service desk plus RAG over technical documentation.
• Volume: 40,000 queries a month.
• Context: on average 8,000 input tokens (the question plus retrieved fragments) and 1,000 output tokens, 9,000 tokens per query in total.
• Model: open-weight, 70B class.
• On-prem hardware: one node with two H100-class 80 GB cards.
• Horizon: 3 years. Business energy rate: about USD 0.24 per kWh. Fully loaded engineering headcount: about USD 80,000 a year.

Scenario A, managed API per token.

Per month: 40,000 queries times 9,000 tokens is 360 million tokens. At a blended rate of about USD 0.60 per million tokens for a 70B-class model, that is about USD 216 a month for compute, or about USD 7,800 over three years. On top of that, vector storage and egress at about USD 15,000 over three years, plus governance and due diligence of the external vendor, a fraction of a headcount, about USD 48,000 over three years.

Item (per token)	3 years (USD)
Compute (per token)	7,800
Vector storage, egress	15,000
Governance and vendor due diligence	48,000
Total	about 71,000

Scenario B, GPU rental per hour.

At a rate of about USD 3 per GPU-hour, the working mode is what matters. Round the clock: 2 cards times 8,760 hours a year times USD 3 is about USD 52,600 a year, or about USD 157,700 over three years. During plant working hours, say 10 hours across 250 working days, that is 2,500 hours per card a year, or about USD 45,000 of compute over three years. In both variants you stand up the model yourself, so add about 0.3 of a headcount (about USD 72,000 over three years), storage and egress (about USD 15,000) and vendor due diligence (about USD 15,000).

Item (GPU rental)	24/7 mode	Working hours
Compute (per GPU-hour)	157,700	45,000
People (0.3 FTE)	72,000	72,000
Storage, egress, IP	15,000	15,000
Vendor due diligence	15,000	15,000
Total	about 260,000	about 147,000

Scenario C, on-prem CAPEX plus OPEX.

CAPEX: two H100-class cards about USD 60,000, a server with RAM, NVMe and redundant power about USD 20,000, network, rack, UPS and cooling expansion about USD 15,000, about USD 95,000 in total. OPEX per year: energy for a node of about 3 kW running continuously is about USD 6,000, a service contract about USD 5,000, people about 0.4 FTE about USD 32,000, audit and pen-test about USD 5,000, platform or open-source support about USD 12,000, about USD 60,000 a year in total. One-off project costs, integration with ERP and MES, training, first-half-year iterations, about USD 25,000. Hardware residual value after three years about USD 15,000 on the minus side.

Item (on-prem, 3 years)	USD
CAPEX (cards, server, network, cooling)	95,000
OPEX (energy, service, people, audit, platform) times 3	180,000
Project costs (integration, training, iterations)	25,000
Hardware residual value	−15,000
Total	about 285,000

Side by side: three scenarios

Scenario	Compute (3 years)	Everything else (3 years)	Total (3 years)
Managed API per token	about 7,800	about 63,000	about 71,000
GPU rental, working hours	about 45,000	about 102,000	about 147,000
GPU rental, 24/7	about 157,700	about 102,000	about 260,000
On-prem CAPEX plus OPEX	CAPEX 95,000	about 190,000	about 285,000

For this volume the managed API is the cheapest, and not by a few percent but several times over. Compute is its smallest line. On-prem, at the same load, costs roughly four times as much on the pure bill. That is not an argument against on-prem, it is an argument for justifying on-prem with what actually justifies it: data control, an NIS2 or GDPR requirement, no consent to send documentation outside the perimeter. Not price, because on price, at this volume, on-prem loses.

The crossover point and sensitivity

One variable flips the result: volume. The rest of the numbers only shift the threshold.

When does on-prem CAPEX start to beat the managed API on pure cost? The three-year on-prem cost, about USD 285,000, divided by the rate of USD 0.60 per million tokens, is about 475 billion tokens over three years, or about 13 billion a month. At 9,000 tokens per query that is about 1.5 million queries a month sustained across a full three years. Below that threshold, on pure cost, the API wins.

Three levers that move the threshold:

• Context length. Doubling the RAG context doubles the per-token cost but does not touch on-prem. Heavy retrieval brings the on-prem break-even closer.
• Availability mode. GPU rental during working hours costs a fraction of the round-the-clock mode. If the model does not need to be up at night, rental beats on-prem for a long time.
• Model class. A pricier frontier model instead of an open-weight 70B can raise the per-token rate several times over and sharply lower the threshold at which on-prem pays off.

Utilisation (queries / month)	What usually wins on cost
Below 100,000	Managed API per token
100,000 to 500,000	Still the API, unless heavy context or a frontier model
500,000 to 1.5M	The border, decided by context, mode and model class
Above 1.5M, sustained, for 3 years	On-prem CAPEX

Why the naive calculation falls apart

Four mistakes that regularly flip the result and make a calculation fail in front of the board.

First: comparing the price of the card against zero on the cloud side. The card shows up in the on-prem bill, while the cloud has no visible spend at the start, so it comes out cheaper, until you add up eighteen months of invoices. Second: leaving out people. Even a productized on-prem setup needs a fraction of a platform-engineering and security headcount, and DIY realistically one to two engineers. Third: counting tokens by the length of the question instead of the full context. In RAG the context is often many times larger than the question, so the per-token bill grows with it. Fourth: no regulatory cost on the cloud side. An external vendor means annual due diligence, mapping transfers, DPA clauses and re-evaluation on every sub-processor change. That is real work you have to add in.

Fifth, and quiet: fitting the horizon to the answer. If someone costs on-prem over five years and the cloud over two, they get the number they wanted. The same horizon on both sides is a condition of honesty, not a detail.

How to read the result

The calculator does not say on-prem or cloud. It says how much more one costs than the other at your volume, and shows how far you are from the crossover point. That lets you separate the cost decision from the risk decision.

If you come out well below the threshold, as in the example above, you make the on-prem decision on the basis of compliance and data control, aware of what that control costs. If you are near the threshold, risk profile and horizon decide. If you are above the threshold, on-prem wins on both fronts at once and the bill is simple. In each case the next step is the same: substitute your own rates, tariff and real volume, because only your numbers turn an indicative example into a decision.

FAQ

How do you calculate the TCO of an AI deployment?

Add up five cost categories, compute, infrastructure, people, compliance, and project plus exit costs, on each side of the comparison, over the same three-year horizon. For the cloud pick the right billing model, per token or per GPU-hour, and for on-prem cost CAPEX plus three years of OPEX minus residual value. Compute is the smallest difference, the real difference sits in people and compliance.

Which cloud model do you compare on-prem against, the API or GPU rental?

The one that would be your real alternative. If you would stand up the model yourself anyway, the reference is GPU rental. If a ready-made API would do, the reference is per-token billing. These are two different calculations and two different break-even points.

From what volume does on-prem AI start to pay off?

In the example with a 70B-class model, the threshold against the managed API lands around 1.5 million queries a month sustained over three years. Against round-the-clock GPU rental, on-prem is competitive much sooner. The threshold depends heavily on context length, availability mode and model class.

What most often disappears from the bill?

On the on-prem side: people, energy and the project costs of the first half year. On the cloud side: the regulatory cost of the external vendor, egress, vector storage and GPU-hours paid for when nobody is asking. Add 20 to 40 percent to the plain compute sum for the items that do not have a convenient unit rate.

Does a lower cloud cost mean on-prem makes no sense?

No. It means on-prem has to be justified by data control and compliance, not by price, as long as the volume is moderate. At high, steady load the cost bill and the regulatory argument point the same way.

// disclosure & biasesDisclosure and biases

I write from the perspective of someone working on AI solutions run outside the public cloud, so I have a natural leaning towards stressing data control and the on-prem advantage where compliance is involved. I have tried to balance that: the worked example says plainly that for a moderate volume on-prem is more expensive than the managed API, and several times over. All numbers are indicative, based on public market rates and 2026 tariffs, not on the result of any specific deployment. For your own decision, substitute your own rates, tariff and real volume. This is not financial or investment advice.

What I do not cover here

• Specific API model pricing and specific operator rates, because they change every quarter. I give the method and indicative orders of magnitude, not a price table.
• Card sizing for a specific model, because that is a separate topic driven by benchmark numbers.
• The three on-prem operating models, DIY, productized and managed, and who is responsible for what.
• The full regulatory mapping of NIS2 and the AI Act, which affects cost indirectly.
• Model federation across plants in a capital group, where the TCO bill looks different.

Related notes

• TCO of on-prem AI vs cloud: how to calculate over a 3-year horizon
• GPU sizing for Llama 3.1 70B inference: numbers from benchmarks
• Bare-metal, colo or appliance: where to put on-prem AI (CAPEX and OPEX)
• DIY, productized or managed: three on-prem AI models and who maintains them
• Vendor lock-in in AI: three layers, two contractual traps