The technical layer of an on-prem AI deployment: how to build a RAG that doesn't hallucinate, how to size GPUs for the model and load, and how to measure retrieval quality and inference throughput. Specifics, benchmarks, and reference architectures instead of marketing diagrams.
This cluster is the engineering side of on-prem AI. We break the RAG pipeline into the parts that actually decide quality: chunking, embeddings, retrieval, and reranking, with a clear answer on when a reranker (cross-encoder) improves relevance and when it just eats GPU budget. We publish GPU sizing in numbers (e.g. Llama 70B: throughput, VRAM, quantization, vLLM/TensorRT-LLM) and reference architectures mapped to security requirements. It's material for CIOs and architects who must make hardware and design decisions but can't find neutral benchmarks in Polish, because they're almost exclusively in English. We link every technical article to its regulatory consequence (logging, isolation, oversight) and to the TCO calculator, so an architectural decision has a price tag.
GPU sizing tells you how many cards to buy. The serving engine decides how much you get out of them. vLLM vs TensorRT-LLM (and SGLang) on Llama 70B: throughput, TTFT, compilation cost and a decision table for on-prem.
Peak throughput from a benchmark does not tell you what a million tokens costs. The real number is the card's hourly rate divided by tokens per hour, then by utilization. We price a million tokens on H100 and H200, compare with API and show the utilization above which your own GPU wins.
The europe-west4 outage exposed something worse than downtime: you cannot audit the dependencies your AI runs on. What you can actually verify on-prem, and what a hyperscaler will never show you.
Four telemetry layers for an on-prem LLM: infrastructure, serving, quality and audit. What to log, what not to store, and how observability feeds the audit trail NIS2 expects.
A reranker sharpens top-k ordering when queries are long and the corpus is dense. The numbers, the VRAM and latency cost, and five setups where it backfires.
How to build RAG outside the public cloud: the pipeline layers, the most common retrieval failures, the data boundary inside the prompt, and the questions an auditor will ask. A technical note for architects and CISOs.
How many GPUs does it really take to run Llama 3.1 70B in-house? Concrete configs (A100, H100, H200), the impact of quantization (FP16 → FP8 → INT4), tokens/s, TTFT, and cost per 1M tokens. No marketing — numbers from vLLM and TensorRT-LLM benchmarks.
No, it helps with large, noisy corpora; with small, well-chunked sets it can be an unnecessary GPU cost and latency.
For development Ollama is convenient; in production under load vLLM delivers many times the throughput (batching, multi-GPU).
It depends on model size and quantization, see the GPU sizing article; roughly, an FP16 model needs ~2× its parameter count in GB, plus headroom for context.
Want to apply this to your case: architecture, compliance, and cost?
→ Book 30 min