Monitoring and observability for on-prem LLMs: what to log and how
Monitoring and observability for on-prem LLMs: what to log and how
Reading time: about 8 minutes. Cluster: architecture. Author: Fryderyk.
Answer first
Observability for an on-prem LLM is not a single GPU-usage dashboard. It is four telemetry layers, each answering a different question: infrastructure (is the hardware coping), serving (does the model respond fast enough), quality (are the answers good and is retrieval on target), and audit (who asked what, and when). You build the first three layers for yourself, to keep the system running and to be able to tune it. The fourth layer is also a regulatory requirement: NIS2 expects an event log that lets you reconstruct what happened in the system and detect an incident. The trick is to log the whole path of a request without mindlessly storing the full text of prompts and answers, because that in itself creates a new pool of sensitive data. This note shows what to log in each layer, what not to store, and how observability ties into the audit trail NIS2 expects.
Table of contents
- Why LLM observability serves two needs at once
- Four telemetry layers: what to log
- Prompts and completions: what to keep, what to drop
- Table: signal, layer, why
- How it wires into the NIS2 audit trail
- Three mistakes you spot after an incident, not before
- FAQ
- Disclosure and biases
- What I do not cover here
- Related notes
Why LLM observability serves two needs at once
Monitoring a classic application answers whether the service is alive and how fast it responds. For an on-prem LLM that is still necessary, but not enough. The model can respond fast and stay stable while returning answers detached from the source data, if retrieval degrades. It can also work correctly on the technical level yet leave no trace that, after an incident, would let you say who asked a specific question and which documents entered the context.
Hence two needs in one system. The first is engineering: you want to see bottlenecks, quality regressions and the cost of running the system, so you can tune the deployment. The second is regulatory: you want a log that holds up in front of an auditor and feeds incident handling. Well-designed telemetry serves both at once, because it largely rests on the same events. Bad telemetry serves only the first, and it is the audit that reveals the second is missing.
Four telemetry layers: what to log
Infrastructure layer. GPU utilisation and temperature, accelerator memory pressure, throughput and latency of the network between zones, the state of the disks holding the indexes. This is the layer that tells you whether the hardware is the bottleneck, and lets you plan capacity before the request queue grows.
Serving layer. Time to first token, tokens per second, queue depth, requests per second, error and timeout rates. These are the metrics that reveal the model is choking under peak load, and they let you set sensible limits and alerts. Without them you learn about the problem from a user, not from a dashboard.
Quality layer. Retrieval relevance, meaning whether the right fragments land in the context, the share of answers with no grounding in the sources, the refusal rate, user feedback signals. This layer is the hardest, because LLM quality does not collapse into a single number, yet even simple retrieval metrics catch regressions after a change in chunking or the embedding model. I wrote about retrieval itself in the note on on-prem RAG; here it is enough to remember that quality has to be measured over time, not eyeballed once a quarter.
Audit layer. Who issued the request, when, from which identity and through which application, which documents entered the context, and which model version and system prompt served the request. This is the layer an engineer will not build unaided, because the system does not need it to run. The auditor and the incident-response team do.
Prompts and completions: what to keep, what to drop
The biggest trap in LLM observability is the reflex to log the full text of every prompt and every answer. It looks like the richest possible record, but in practice it creates a new, dense pool of sensitive data, often more sensitive than the sources themselves, because it captures what people actually ask. That log becomes a target in its own right and falls under the same protection requirements as the data on-prem was meant to keep inside the organisation.
A sensible default is to log event metadata always and content only conditionally. You always record the request identifier, the identity, a timestamp, the model and prompt version, the identifiers of the retrieved documents, and the execution metrics. Prompt and answer content you store in a truncated or pseudonymised form, turning on the full record selectively: on a sample for quality evaluation, or for the duration of diagnosing a specific problem, with short retention and separate access control. That way you keep the full event path for audit and incident response without multiplying copies of sensitive data for no reason.
Table: signal, layer, why
| Signal | Layer | Why you collect it |
|---|---|---|
| GPU utilisation and memory | Infrastructure | Capacity planning, spotting a hardware bottleneck |
| Time to first token, tokens per second | Serving | Meeting the SLA, alerting on peak load |
| Retrieval relevance, share of ungrounded answers | Quality | Catching regression after a model or chunking change |
| Identity, time, model version, document IDs | Audit | Event log for NIS2 and incident response |
Print this table and check which line your current stack actually collects. Usually the quality layer and half the audit layer are missing.
How it wires into the NIS2 audit trail
Article 21 of the Polish Act on the national cybersecurity system requires measures proportionate to the risk in, among other areas, incident handling and the assessment of the effectiveness of safeguards. No provision tells you to log a specific field, because the list is technology-neutral. But incident handling is empty if, after an event, you cannot answer who asked the model what and which data went into the answer. The audit layer is exactly that evidence.
The practical consequence is that the LLM event log should flow into the same log-collection system as the rest of the critical infrastructure, with the same retention policy and protection against modification. A log that can be quietly overwritten is not evidence. It also helps if the boundaries traffic crosses are logged consistently with the network isolation model I described separately, because pairing a zone diagram with a connection log is an artefact an auditor understands at once. I gathered the full list of documents you produce for an auditor anyway in the note on audit readiness, and observability is what feeds those documents with data.
Three mistakes you spot after an incident, not before
First: monitoring stops at GPU and uptime. The system looks healthy on the dashboard while, for weeks, it has been returning worse answers, because no one measures retrieval quality. The regression is caught only by a user.
Second: full content logging with no retention policy and no access control. Instead of evidence you get a second pool of sensitive data that becomes a risk in itself and a subject for the auditor to ask why you keep it longer than needed.
Third: LLM logs live on a separate island, outside central log collection. During an incident you have to stitch them together by hand, and some have already rotated out. A log you cannot correlate with the rest of the infrastructure defends itself poorly.
FAQ
Does NIS2 require logging prompt content?
No. The act does not name specific fields. It requires measures proportionate to the risk in incident handling, which comes down to being able to reconstruct an event. In most deployments event metadata is enough, and you store full content selectively and briefly.
Is monitoring GPU and uptime enough?
No. That covers only the infrastructure and serving layers. Without the quality layer you will not notice answer regression, and without the audit layer you will not handle an incident or defend yourself in front of an auditor.
How long should LLM logs be kept?
Retention should follow your policy and risk analysis, consistent with the rest of the critical infrastructure, not a tool's defaults. Event metadata is usually kept longer than any full content, which is limited to a short window.
How does LLM observability differ from classic APM?
APM covers infrastructure and serving well, but it does not measure answer quality and does not automatically capture which documents entered the context. Those two layers have to be added deliberately.
// disclosure & biasesDisclosure and biases
I write from the perspective of someone working on AI solutions run outside the public cloud, so I emphasise data control and the audit trail. I have tried to separate the regulatory requirement from technical preference. This text is not legal advice, nor a monitoring-system design for a specific organisation. The split into four layers is a working simplification, useful for the conversation, not the only correct model.
What I do not cover here
I do not name specific observability tools or logging stacks, because the choice depends on what you already run. I skip the details of LLM quality evaluation metrics, as they deserve a separate note. I do not go into alert configuration for a given vendor, nor observability of the training and fine-tuning stage, because I focus on production inference. I also do not cover the legal details of retention, which are worth confirming with your compliance team.
Related notes
Building CortexMine, an on-prem AI platform for European manufacturers under NIS2. Where this bias could affect conclusions, it is flagged inline.
Want to apply this to your case: architecture, compliance, and cost?
→ Book 30 minReranking in on-prem RAG: when it lifts relevance and when it just burns GPU
A reranker sharpens top-k ordering when queries are long and the corpus is dense. The numbers, the VRAM and latency cost, and five setups where it backfires.
On-prem RAG: architecture, chunking, retrieval and what actually drives quality
How to build RAG outside the public cloud: the pipeline layers, the most common retrieval failures, the data boundary inside the prompt, and the questions an auditor will ask. A technical note for architects and CISOs.
GPU sizing for Llama 3.1 70B inference: numbers from benchmarks
How many GPUs does it really take to run Llama 3.1 70B in-house? Concrete configs (A100, H100, H200), the impact of quantization (FP16 → FP8 → INT4), tokens/s, TTFT, and cost per 1M tokens. No marketing — numbers from vLLM and TensorRT-LLM benchmarks.