On-prem AI vs cloud TCO: how to calculate over three years

Reading time: about 12 minutes. Cluster: vendor-eval. Author: Fryderyk.

Answer first

Whether on-prem AI works out cheaper than cloud is not settled by the price of a GPU. It is settled by two numbers: utilization (how many hours a month the hardware actually computes) and the amortization horizon. On-prem wins when utilization is high and steady and the horizon is at least three years. Cloud, billed either per token or per GPU-hour, wins with low, irregular or hard-to-predict volume, during pilots, and wherever there is no team to run the infrastructure.

The raw break-even is simpler than it looks: a card around 25k USD against rental at roughly 3 USD per GPU-hour meets at about 8k hours, that is under a year of round-the-clock running. The catch is that this sum ignores everything else: the server room, energy, people and compliance. Once those are added, on-prem becomes cheaper only at genuinely high, sustained utilization held for a full three years. Below I show how to model both scenarios so the numbers survive scrutiny from the board and from an auditor, and where the line actually falls.

Table of contents

1. Why the hardware price is the least important number
2. Three cost models you must not blend
3. The on-prem model: what belongs in a three-year sum
4. The cloud model: per token and per GPU-hour
5. The break-even point: the one variable that decides
6. When cloud honestly wins
7. Table: what to include, what not to forget
8. FAQ
9. Disclosure and biases
10. What this does not cover
11. Related notes

Why the hardware price is the least important number

A TCO conversation about on-prem AI almost always starts from the wrong end: the price of the card. That is understandable, because the price is visible and concrete, and a fresh H100-class card runs 25k to 40k USD apiece depending on variant and vendor. The trouble is that the card is the least risky line in the whole sum, because it is one-off, known in advance and depreciable.

The real costs sit in the places that are easy to forget at decision time. A card at 25k needs a server to live in, power, cooling, rack space, updates and a person to maintain it. In practice a full production setup can run 30 to 50 percent above the card price alone. On the cloud side it is the reverse: there is no visible upfront spend, so it is easy to score it as cheaper, right up until you tally eighteen months of invoices at steady volume.

That is why the only honest comparison is a full sum of both scenarios over the same horizon, usually three years, because that is the typical hardware depreciation period and the realistic planning span for a production workflow. Anything shorter favours cloud, anything longer favours on-prem, so fix the horizon before the numbers, not after.

Three cost models you must not blend

The first trap in the comparison: "cloud" is not one thing. There are three billing models and they give three different answers. Mixing them is the most common reason a calculation fails to hold up in front of a CFO.

Model 1. Cloud per token (managed model API). You pay for every thousand input and output tokens. Zero infrastructure on your side. Cost rises linearly with query volume and context length. This is the model for APIs such as OpenAI, Anthropic, Azure OpenAI, Bedrock. Upside: no CAPEX, instant start, zero cost at zero use. Regulatory downside: data leaves the perimeter and the vendor enters the NIS2 supply chain.

Model 2. Cloud per GPU-hour (renting the card). You rent a card or cluster by the hour and stand the model up yourself. This is the variant most comparable to on-prem, because you pay for the same resource, only somebody else's. Market rates in 2026 for an H100 run roughly 2 to 3.5 USD per GPU-hour on specialist clouds, clearly more on hyperscalers. Upside: flexibility, no CAPEX. Downside: at 24/7 running the bill climbs fast, and the data still sits with the cloud operator.

Model 3. On-prem CAPEX (the card is yours). You buy the hardware, depreciate it, maintain it. High cost at the start, low marginal cost for every additional compute hour. In this model every idle GPU-hour is wasted capital, but every used one is cheap.

Rule: do not compare on-prem CAPEX with cloud per token, because you are measuring two different things (capital versus consumption). First decide which cloud model is your real alternative. If you would stand the model up yourself anyway, your reference is GPU rental (model 2). If you would settle for a ready-made API, your reference is per token (model 1). Two different calculations, two different break-even points.

The on-prem model: what belongs in a three-year sum

A full on-prem TCO is made of four groups, not one. Below are the lines that must appear in the calculation for the number to be honest.

CAPEX, usually depreciated over 3 to 5 years:

• Server with GPU cards, RAM, NVMe drives, redundant power, management card. The core of the spend.
• Network: switch, cabling, an interconnect for multi-node if needed.
• Backup power and a cooling upgrade for the extra kilowatts.
• Optionally a second, smaller node for redundancy or a test environment.

OPEX, annually:

• Energy. Here you take power draw (cards plus cooling) times server-room PUE times price per kilowatt-hour times hours of operation. For companies in Poland in 2026 the real, full business rate (active energy plus distribution plus system charges) sits roughly in the 0.20 to 0.25 EUR per kilowatt-hour range net, depending on tariff and consumption profile. A node with two H100 cards plus cooling is about 3 kW, which at 24/7 comes to a few thousand EUR a year.
• Hardware support contract (ProSupport or equivalent).
• Software upkeep: either a productized platform licence, or the team cost under DIY on open source.
• People. Even a productized deployment needs a fraction of a platform engineer and a fraction of a security role. DIY realistically needs 1 to 2 full ML engineers.
• Audit and pen-test, for NIS2 essential entities at least once a year.

Project costs, one-off, often omitted:

• Board, CISO, legal and compliance time on the decision and due diligence.
• Department time to integrate the source system (ERP, MES, PLM).
• User training and the first-half-year iterations (prompt tuning, retrieval tuning, content cleanup).

Residual value and exit:

• After three years the hardware still has book and useful value. The card does not vanish, it can move to another workload.
• Exit cost is low: the data is already with you, there is no egress and no re-platforming.

Practical rule: add 20 to 40 percent to the plain CAPEX plus OPEX sum for project costs and iterations. A calculation without that buffer always understates on-prem and overstates its edge over cloud.

The cloud model: per token and per GPU-hour

On the cloud side the sum looks simpler, but it has its own lines that vanish from a naive calculation.

Per token. Cost = queries per month times average tokens per query (input plus output, including all the retrieval context) times price per token. The key observation: in RAG use cases the context can be large, so the cost per query is often several times higher than the length of the question alone suggests. On top come the hidden lines: data egress, vector storage if you keep it in the same cloud, and the time to build the controls that let such a data flow pass NIS2 due diligence.

Per GPU-hour. Cost = number of cards times hours of operation times hourly rate. The trap here is the running mode. If the model must be available all the time (a plant service desk during working hours, and often beyond), you pay for hours even when nobody is asking. Spot rates can be much lower but carry interruption risk, so they rarely fit production. On top come storage, egress and IP-address fees, which differ between operators.

A line common to both variants and constantly skipped: the regulatory cost. An external vendor in the supply chain means annual due diligence, transfer mapping, DPA clauses and re-evaluation on every sub-processor change. That is real work to add on the cloud side, and it is lower on-prem, because the supplier is internal.

The break-even point: the one variable that decides

Once you sum both sides, almost everything turns on a single variable: utilization, that is how many GPU-hours a month you actually consume.

The raw rent-versus-buy maths is well known: a card at 25k USD against a 3 USD hourly rate meets at about 8.3k hours, roughly 347 days of continuous running. If only the hardware counted, the line would fall under a year. But on-prem is not only the card, so the real line sits higher. In practice buying starts to beat renting only at sustained loads on the order of 10k GPU-hours a month held for three years and more, a level a minority of organizations ever reach.

Below is a simplified map of where the line usually falls. The figures are indicative and depend on rates, tariffs and the specific model, but the order of magnitude holds.

Utilization (GPU-hours / month)	Profile	What usually wins on cost
Under 1,000	Pilot, irregular volume, few queries	Cloud per token or on-demand GPU rental
1,000 to 5,000	One steady workflow, moderate volume	Usually still cloud on pure cost, on-prem wins on control
5,000 to 10,000	Steady service desk, several workflows	Parity zone, horizon and risk profile decide
Over 10,000, sustained, over 3 years	High, predictable 24/7 load	On-prem CAPEX

The conclusion is uncomfortable for both camps. On-prem is not magically cheaper until you have high, steady load. Cloud is not magically cheaper once you have that load and hold it for years. So before you calculate anything, estimate the real query volume and context length, because those, not the card price, settle the result.

When cloud honestly wins

This is worth saying plainly, because a portal about on-prem AI has a natural pull towards promoting on-prem. There are situations where cloud is simply cheaper and more sensible, and pretending otherwise ends in a bad investment decision.

• Pilot and proof of concept. Until you know whether a workflow will even land, paying upfront for hardware is a bet in the dark. Cloud lets you measure the real volume before you calculate on-prem.
• Low or irregular volume. A handful of queries a day will never amortize a card. Here on-prem is wasted capital.
• No team to run it. If you have no platform engineer and no security competence for the model layer, and no plan to build them, the cost of standing that team up will exceed the hardware saving.
• Low data sensitivity. If the data is not covered by NIS2, by GDPR in a sensitive mode, or by NDA, the regulatory argument for on-prem weakens, and only the sum is left, which at low volume points to cloud.

On-prem holds up where three conditions meet at once: high steady load, a long horizon and data you must not or cannot afford to send outside the perimeter. Remove any one of them and the edge melts.

Table: what to include, what not to forget

Cost category	On-prem (3 years)	Cloud (3 years)
Compute	CAPEX for cards and server, depreciated	OPEX per token or per GPU-hour
Energy and cooling	Power draw times PUE times kWh rate	In the service price
Server room or colo	Rack, UPS, air conditioning	None
People	0.3 to 0.5 FTE platform for productized, more for DIY	Smaller but not zero: integrations and governance
Software	Open source or productized licence	In the API price
Compliance	Lower due-diligence cost, internal supplier	Annual due diligence of an external vendor
Exit and migration	Data stays, hardware keeps value	Re-platforming, data egress
Project costs	Decision, integrations, training, iterations	Integrations, training, controls for DD

Two columns, the same horizon, the same categories. Only such a table lets you compare like with like. A naive calculation compares the card price with zero on the cloud side and always comes out wrong.

FAQ

Is on-prem AI cheaper than cloud?

Not in a general sense. It is cheaper at high, steady utilization held for at least three years. At low or irregular volume cloud usually wins on cost. The difference sits more often in the regulatory risk profile than in the price itself.

What horizon should the calculation use?

Three years is the standard, because that is the typical hardware depreciation period and the realistic planning span for a production workflow. A shorter horizon favours cloud, a longer one on-prem, so fix the horizon before calculating, do not tune it after the fact to a desired result.

What should on-prem actually be compared with, API or GPU rental?

With the cloud model that would be your real alternative. If you would stand the model up yourself anyway, the reference is GPU rental. If a ready-made API would do, the reference is per-token billing. Two different calculations.

What most often vanishes from the on-prem sum?

People, energy, project costs and the first-half-year iterations. Add about 20 to 40 percent to the plain CAPEX plus OPEX sum to keep the calculation honest.

What most often vanishes from the cloud sum?

The regulatory cost of an external vendor in the supply chain, data egress, vector storage, and GPU-hours paid for while nobody is asking, under an always-on model.

// disclosure & biasesDisclosure and biases

I write from the perspective of someone working on AI solutions run outside the public cloud, so I have a natural pull towards stressing data control and the on-prem edge wherever compliance enters. I tried to balance that, which is why a separate section states plainly when cloud wins, and the break-even map shows that for most low-volume organizations on-prem is not cheaper. The figures in the text are indicative and based on public market rates and 2026 tariffs, not on the result of any specific deployment. For your own decision, plug in your own rates, tariff and real volume. This is not financial or investment advice.

What this does not cover

• The specific price list of model APIs and specific vendor rates, because they change every quarter and would date quickly. I give the method, not a price table.
• Hardware selection and card sizing for a specific model, because that is a separate topic driven by benchmark numbers.
• The three on-prem operating models (DIY, productized, managed) and who is responsible for what, because that is a separate note.
• The full NIS2 and AI Act regulatory mapping, which affects cost indirectly but runs on its own logic.
• Federated-model scenarios across plants in a capital group, where the TCO sum looks different.

Related notes

• Bare-metal, colo or appliance: where to place on-prem AI (CAPEX and OPEX)
• DIY, productized or managed: three on-prem AI models and who maintains them
• GPU sizing for Llama 3.1 70B inference: numbers from benchmarks
• Vendor lock-in in AI: three layers, two contractual traps
• On-prem AI in European manufacturing 2026: a complete architectural guide