vLLM vs TensorRT-LLM: Which Engine for On-Prem LLM

Why the engine is a separate decision from the GPU

Once a company has worked out how many GPUs it needs for Llama 70B, it usually assumes the hard technical call is behind it. It is not. The same 2x H100 server can serve twice as many users depending on which inference engine you run on it and how you configure it. The hardware sets the ceiling. The engine decides how close to that ceiling you actually get.

In practice, for on-prem LLM in mid-size manufacturing the choice comes down to two names: vLLM and TensorRT-LLM. A third, SGLang, enters wherever shared context matters (RAG, long system prompts). This post compares them with numbers and ends with a decision table: which engine for which scenario. Without the claim that one is always better, because it is not.

What these engines actually do

An inference server sits between the model and the user and handles three things: loading weights into VRAM, managing the request queue (batching) and returning tokens. The differences between engines lie mostly in how cleverly they batch and how aggressively they optimize for specific hardware.

vLLM is the open-source project with the broadest model support and the simplest deployment. Its signature is PagedAttention, managing the KV cache like memory pages in an operating system. You launch a model with one command, no compilation step. Updating a model means swapping the weights.

TensorRT-LLM is NVIDIA's library that compiles a model into an optimized engine for a specific GPU architecture (Hopper, Blackwell), a specific precision and a specific batch-size range. That compilation is a separate step that takes time, but in return it delivers the highest throughput the card can produce.

SGLang is an engine with aggressive prefix caching. When many requests share the same beginning (the same long system prompt, the same document in RAG context), SGLang computes that shared part once and reuses it. In pure on-prem RAG this can be decisive.

Numbers: throughput and TTFT on H100

The values below come from an independent SqueezeBits benchmark on a single H100 SXM5 80GB card, model Llama 3.3 70B Instruct in FP8, versions vLLM 0.18, TensorRT-LLM 1.2 and SGLang 0.5.9. Method: prompts of ~512 input tokens, 256 output tokens, measured at rising concurrency.

Aggregate throughput (output tokens per second), higher is better:

Concurrency	vLLM	TensorRT-LLM	SGLang
1 request	120 tok/s	130 tok/s	125 tok/s
10 requests	650 tok/s	710 tok/s	680 tok/s
50 requests	1,850 tok/s	2,100 tok/s	1,920 tok/s
100 requests	2,400 tok/s	2,780 tok/s	2,460 tok/s

Time to first token, median (p50) and p95, lower is better:

Concurrency	vLLM p50	vLLM p95	TRT-LLM p50	TRT-LLM p95	SGLang p50	SGLang p95
1 request	45 ms	68 ms	38 ms	55 ms	42 ms	61 ms
10 requests	120 ms	195 ms	105 ms	170 ms	112 ms	178 ms
50 requests	380 ms	720 ms	340 ms	620 ms	360 ms	680 ms
100 requests	740 ms	1,450 ms	680 ms	1,280 ms	710 ms	1,380 ms

The pattern is clear: TensorRT-LLM leads at every concurrency level once the engine is compiled. The gap is smallest at a single request (~8% over vLLM) and largest at 50 concurrent (~13%). TTFT points the same way: TRT-LLM has lower latency, and the gap widens in the p95 tail, exactly where users feel the stutter most.

The other side is worth knowing too. An older SqueezeBits benchmark on A100 (Llama 3 8B) showed that under a very tight latency limit with a forced small batch (batch 4) it was vLLM that came out faster, 230 tok/s against 197 tok/s for TRT-LLM. In other words: TensorRT-LLM's edge materializes under load and batching, not on single, latency-critical requests to a small model.

What the numbers mean operationally

A dozen percent of throughput sounds like a detail until you convert it into users. If a vLLM server handles 30 concurrent sessions at your SLO, that ~13% edge is realistically 3 to 4 extra sessions from the same hardware. Across two servers that can be the difference between buying a third card and not.

But there is a condition: this edge is free only when the model is stable. Every change of model, precision or target batch-size range in TensorRT-LLM means recompiling the engine. vLLM has no such step. So the numbers in the table are not the whole story. The second half is the cost of running it.

Cold start and maintenance cost

This is where most comparisons stop, and where the on-prem decision is really settled.

Engine	Cold start (first time)	Subsequent reloads	Compilation step
vLLM	~62 s	~62 s	none
TensorRT-LLM	~28 min (compilation)	~90 s	yes, per model version
SGLang	~58 s	~58 s	none

TensorRT-LLM's 28-minute compilation is not a flaw, it is a deliberate tradeoff: you build the engine once per model version, then serve it for months. The problem appears when your team changes models often, tests quantization variants or updates versions every two weeks. Then 28 minutes per iteration plus the need to version engine artifacts turns into real DevOps overhead that vLLM simply does not have.

VRAM usage is practically the same across all three engines (71 to 79 GB on an 80GB H100 for this model in FP8), so memory is not a selection criterion here. The criterion is how many hands you have to maintain it and how stable your model is.

H200 and long context: when memory changes the math

The numbers above are one H100 card. Choosing between H100 and H200 brings in a variable that a short-prompt throughput table does not show: memory capacity and bandwidth. H200 has 141 GB of VRAM against 80 GB on H100, which on long RAG context (15k to 30k input tokens, typical for technical documentation) lets you hold a larger KV cache and higher batch before memory becomes the bottleneck.

Here the engine choice starts to blend with the hardware choice. On long, shared context SGLang with prefix caching can erase TensorRT-LLM's throughput edge, because it does not recompute the same prefix per request. Rule of thumb: the more unique your prompts, the more raw throughput matters (TRT-LLM); the more they share common context, the more prefix caching pays off (SGLang).

Decision table: which engine for which on-prem

On-prem scenario	Recommendation	Why
One stable model, maximum throughput under load	TensorRT-LLM	Highest tok/s and lowest TTFT, compile once
Frequent model changes, small IT team, fast iteration	vLLM	Zero compilation, broadest model support, simplest deploy
RAG with a shared system prompt or long context	SGLang	Prefix caching reuses the shared part, lower latency
First deployment, not sure yet what you serve	vLLM	Safe default, migrate off it once the model stabilizes
Single-request latency critical, small model	vLLM or TRT-LLM	With a forced small batch the gap blurs, test both

Default recommendation

For most mid-size firms entering on-prem AI, a sensible start is vLLM: zero compilation, the broadest model support and the least overhead for a team still learning to operate its own LLM. Once the model stabilizes and traffic grows to where a dozen percent of throughput starts to translate into saving a card, migrating to TensorRT-LLM becomes a countable decision. If you know from the start that your main workload is RAG over a shared document base, SGLang is worth testing in parallel with vLLM from day one.

The worst decision is choosing an engine by a vendor's blog benchmark instead of by your own traffic profile. Measure your real distribution of prompt lengths, concurrency and model-change frequency first, then read the tables. The numbers above are a starting point for the conversation, not a verdict.

What this post does not cover

• Multi-node and tensor parallelism beyond one server. All numbers above are a single card or one node. Scaling to 8+ GPUs has its own bottlenecks (interconnect, pipeline parallelism) and deserves a separate post.
• Exact kernel-flag and CUDA-version tuning. Swings of ±10 to 15% between engine and driver versions are normal. Treat the tables as an order of magnitude, not a guarantee.
• Non-NVIDIA alternatives (AMD MI300X, Intel Gaudi). Support in these engines is maturing, but for mixed inference in mid-size production it is still immature.
• Model quality. The engine does not change what the model can do, only how fast it serves it. Model and quantization choice is a separate conversation.

// disclosure & biasesDisclosure and biases

I work on an on-prem AI platform (CortexMine) for European manufacturers. The numbers in this post come from the public SqueezeBits benchmark and public engine-selection guidance, not from our internal tests. Where sources differed (A100 vs H100, small vs large model), I flagged it in the text, because the conclusions are not universal.

A bias I am aware of: I write from the view that "on-prem is a real choice", so I naturally focus on maintenance and control criteria, not only raw throughput. A company serving one model for a year without changes has a different calculus than a team iterating every two weeks, and I try to keep those apart.

Related notes

• Sizing: GPU sizing for Llama 3.1 70B inference: numbers from benchmarks — how many cards to buy before you pick the engine to run on them.
• Pillar: On-prem AI in European manufacturing 2026: architecture guide — the wider architectural context.