You deploy your Large Language Model (LLM) to production. The dashboard shows a steady stream of requests per second. Everything looks green. Then, a user complains that the chatbot feels "frozen" for ten seconds before spitting out an answer. You check the average latency. It’s fine. So what went wrong?
The problem is that traditional web server monitoring fails spectacularly when applied to LLMs. In standard HTTP services, work per request is relatively uniform. In LLM inference, it varies wildly. One request might generate five tokens; another might generate two thousand. If you only watch requests per second, you miss the fact that your GPU memory just exploded or your queue backed up because three users asked for long summaries simultaneously.
This is why modern LLM Observability is not optional infrastructure-it is the difference between a sluggish product and a profitable one. It requires tracking specific signals: token throughput, queue dynamics, and tail latency. Let’s break down exactly what to measure and why averages lie.
Why Averages Hide the Pain
If you rely on mean latency, you are lying to yourself. LogicMonitor’s AI Observability guide puts it bluntly: "Nobody cares about your average if the 99th percentile is terrible." This is especially true for LLMs due to the heavy-tailed distribution of output lengths.
Imagine a queue where 99% of requests take 100ms, but 1% take 5 seconds. Your average might look acceptable, but that 1% represents real users who closed their tabs in frustration. Research from Glean quantifies this pain precisely: for every additional input token, the P95 Time-to-First-Token (TTFT) increases by approximately 0.24ms. That sounds small until you realize a complex prompt with 2,000 context tokens adds nearly half a second just to start thinking.
Tail latency defines the actual user experience. When the P99 spikes, it usually means your batching strategy failed, your GPU ran out of KV cache, or a single massive request blocked the entire batch. Monitoring these percentiles (P50, P95, P99) reveals saturation points that averages completely mask.
Token Metrics: The Real Currency of Inference
In traditional servers, CPU cycles are the bottleneck. In LLMs, tokens are the currency. You need to track three distinct token flows to understand system health:
- Prompt Tokens: Input size. Drives prefill time and memory usage.
- Completion Tokens: Output size. Drives decode time and total generation duration.
- Total Token Throughput: The aggregate flow across all concurrent requests.
Here is the trap: variable work per request means "requests per second" (RPS) can remain stable while token throughput collapses. If your traffic shifts from short Q&A interactions to long-form document summarization, RPS stays flat, but your GPU load doubles. Tools like vLLM and Hugging Face TGI explicitly export telemetry for this reason.
You should be looking at histograms, not just counters. Track `tgi_request_generated_tokens` or `gen_ai.client.token.usage`. More importantly, track time per output token. If this metric rises, your model is struggling to keep up with the streaming rate, making responses feel fragmented even if they eventually complete.
Time-to-First-Token (TTFT): The Perception Threshold
Users judge responsiveness based on how fast the first word appears. This is Time-to-First-Token (TTFT): the initial latency before the first token streams back to the client. Kong’s AI Observability Guide notes that when TTFT jumps from milliseconds to several seconds, the interface feels broken.
What causes high TTFT? Usually, it’s prefill processing. Large contexts require significant compute to populate the Key-Value (KV) cache. If your TTFT is high, check your input token distribution. Are users pasting entire PDFs into the prompt window? Industry guidance suggests aiming for sub-50ms TTFT for seamless chat experiences, though this depends heavily on your specific use case.
| Metric Name | Standard Export Format | User Experience Impact | Common Cause of Degradation |
|---|---|---|---|
| Time-to-First-Token (TTFT) | `vllm:time_to_first_token_seconds` | Perceived freeze / lag | Large input context, cold start, queue wait |
| Inter-Token Latency | `gen_ai.server.time_per_output_token` | Stuttering text stream | GPU contention, low batch efficiency |
| Total Generation Time | `tgi_request_duration` | Overall slowness | Long output length, slow decode speed |
| Queue Wait Time | BentoML/TGI Queue Metrics | Unpredictable delays | Insufficient replicas, bad batching policy |
Queuing Dynamics: Where Requests Go to Die
LLM inference is a classic queuing problem. According to arXiv research on queueing theory, the heavy tail of output token lengths significantly extends average queuing delay. One user asking for a 4,000-word essay blocks resources that could have served fifty quick questions.
You must monitor queue depth and batch size as first-class indicators. BentoML identifies queue wait time as a critical signal revealing delays caused by waiting for an available replica. If your queue grows, you aren’t just slow; you are dropping users. The arXiv analysis shows that enforcing a maximum output token limit on a small fraction of requests can drastically reduce queueing delays. It’s a trade-off: cap the output length slightly to save the majority of users from waiting behind a monster request.
Be careful with "impatient users." If the queue is too deep, people abandon the session. Your observed load might drop not because you’re efficient, but because you’re losing customers. Monitor timeout rates and abandonment signals alongside raw queue length.
Implementing the Stack: From Prometheus to Alerts
How do you actually see this data? Most modern serving engines like vLLM and TGI expose Prometheus-compatible endpoints. You don’t need custom code to get basic visibility. Ensure you are scraping metrics like `tgi_request_mean_time_per_token_duration` and `vllm:request_success_total`.
However, raw metrics aren’t enough. You need correlation. When tail latency spikes, can you instantly filter logs to see which prompts were involved? Was it a specific user? A specific model version? LangChain and other observability platforms help bridge this gap by attaching metadata (user ID, feature flag, model config) to each trace.
Set alerts on P95 TTFT and Inter-Token Latency, not just error rates. An error rate of 0% with a P95 latency of 3 seconds is still a failing service. Also, track cost per interaction. Different models have vastly different price profiles. A sudden shift in traffic mix toward expensive models can blow your budget without triggering any technical alarms.
Practical Checklist for Production Readiness
Before you scale your next LLM deployment, verify these items:
- Instrument Token Counts: Are you logging prompt vs. completion tokens separately?
- Track Percentiles: Do your dashboards show P50, P95, and P99 for TTFT and Total Duration?
- Monitor Queue Depth: Can you see how many requests are waiting for GPU slots right now?
- Define SLOs: Have you set concrete thresholds for acceptable latency (e.g., "P95 TTFT < 200ms")?
- Correlate Cost: Are you tagging metrics by model tier to track spend anomalies?
Observability isn’t just about debugging crashes. It’s about understanding the physics of your system. Tokens flow, queues build, and tails stretch. If you don’t measure them, you’re flying blind.
Why is Time-to-First-Token (TTFT) more important than total latency?
TTFT determines perceived responsiveness. Users tolerate slower overall generation if the response starts quickly, creating a sense of immediacy. High TTFT makes the application feel frozen or broken, leading to higher bounce rates, whereas slight delays in inter-token streaming are often less noticeable.
Can I use Requests Per Second (RPS) to capacity plan my LLM cluster?
No, RPS is unreliable for LLMs because work per request varies drastically. A single request generating 2,000 tokens consumes significantly more GPU memory and time than a request generating 50 tokens. You must capacity plan based on token throughput (tokens/sec) and memory utilization (KV cache size).
What is the relationship between input tokens and tail latency?
Input tokens primarily affect Time-to-First-Token (TTFT). Research indicates that P95 TTFT increases by ~0.24ms per additional input token. Therefore, large context windows directly contribute to tail latency spikes, especially under high concurrency when prefill operations compete for compute resources.
How does queueing theory apply to LLM inference?
LLM inference behaves like an M/G/1 queue with heavy-tailed service times. Long output generations act as "head-of-line blocking," delaying subsequent requests. Managing this requires strategies like continuous batching, setting max output limits, or using priority queues to prevent long-running jobs from starving shorter ones.
Which tools natively support these metrics?
vLLM and Hugging Face Text Generation Inference (TGI) both expose Prometheus-compatible metrics specifically for token counts, TTFT, and inter-token latency. OpenTelemetry also has semantic conventions for GenAI (e.g., `gen_ai.server.time_to_first_token`) to standardize collection across different stacks.
