Latency Budgets for Interactive LLM Apps: A Practical Guide

Imagine waiting three seconds just to see the first word of an AI response. For a chatbot, that feels like an eternity. In interactive applications, users don't just want accuracy; they want speed. This is where latency budgets come in. They define the maximum acceptable delay between a user's input and the model's output. If you miss this budget, your app feels sluggish, no matter how smart the model is.

Setting these budgets isn't just about picking a fast server. It involves understanding how Large Language Models (LLMs) actually work under the hood. The two main phases of inference-prefill and decode-have very different performance characteristics. Ignoring this distinction leads to costly infrastructure mistakes and frustrated users.

Key Takeaways

• Time to First Token (TTFT) is the most critical metric for perceived responsiveness. Aim for under 500ms for real-time feel.
• The decode phase is memory-bound, meaning it's limited by how fast data moves from GPU memory, not raw compute power.
• Speculative decoding can cut latency by 2-4x by using a smaller model to predict tokens ahead of time.
• Batching requests improves throughput but increases individual request latency, creating a direct tradeoff.
• Model size dictates hardware needs: a 109B parameter model requires at least three H100 GPUs, impacting both cost and latency ceilings.

Understanding the Two Phases of LLM Inference

To manage latency, you first need to understand where the time goes. LLM inference happens in two distinct stages: prefill and decode. These stages behave differently and require different optimization strategies.

Prefill Phase is the initial step where the model processes the entire input prompt to build a key-value cache. This phase is compute-bound. Because it processes all input tokens in parallel, it benefits heavily from high-performance GPUs with strong arithmetic capabilities. If your prompts are long-say, 10,000 tokens for a Retrieval-Augmented Generation (RAG) system-the prefill phase dominates the total latency. You can't easily amortize this cost through batching without increasing the wait time for every user in the batch.

Decode Phase is the sequential generation of output tokens, one by one, relying on the previously built cache. Unlike prefill, decode is memory-bound. Each new token requires reading the entire growing Key-Value (KV) cache from GPU memory. As the context grows, the bandwidth required to fetch this data becomes the bottleneck. This means adding more compute cores doesn't help much here; you need faster memory bandwidth.

This asymmetry shapes your latency budget. If your application uses short prompts and long outputs, optimize for memory bandwidth. If it uses long prompts and short answers, optimize for compute density. Most interactive apps fall somewhere in the middle, requiring a balanced approach.

Defining Your Latency Metrics: TTFT vs. Tokens Per Second

You can't manage what you don't measure. In LLM applications, two metrics define the user experience: Time to First Token (TTFT) and Inter-Token Latency (ITL), often expressed as tokens per second (TPS).

Time to First Token (TTFT) is the latency experienced during the prefill phase until the first output token is returned to the user. This is the "loading screen" moment. Users perceive immediate feedback as responsive. Even if the full answer takes ten seconds, if the first word appears instantly, the app feels alive. Industry standards suggest keeping TTFT under 500 milliseconds for a seamless interactive feel. Exceeding one second starts to feel laggy; exceeding three seconds feels broken.

Tokens Per Second (TPS) is the speed at which subsequent tokens are generated during the decode phase. This determines how quickly the rest of the response streams in. For typing-speed readability, you generally want at least 20 TPS. Below 10 TPS, users may start skimming or abandoning the interaction. Note that TPS is constrained by memory bandwidth, while TTFT is constrained by compute power. Optimizing one doesn't automatically optimize the other.

The Batching Tradeoff: Throughput vs. Responsiveness

In production environments, you rarely process one request at a time. You batch multiple requests together to maximize GPU utilization. However, batching introduces a significant latency penalty for individual users.

When you increase the batch size, the GPU spends more time processing other users' tokens before getting back to yours. For example, with a Qwen 2.5 7B model, the latency for a single request might be 126 milliseconds at a batch size of 8. But if you drop to a batch size of 1, the latency jumps to 976 milliseconds because the GPU isn't fully utilized. Wait, that seems counterintuitive? Actually, at batch size 1, the GPU is idle for large portions of the cycle. At higher batches, the *average* latency per token drops, but the *wait time* for a specific user to get their next token increases because they have to share the bandwidth with others.

This creates a classic engineering dilemma:

• Low Batch Size: Lower latency for individual users, but higher cost per token due to low GPU utilization.
• High Batch Size: Higher throughput and lower cost per token, but increased Tail Latency (P99) for individual users.

For interactive apps, you must cap your max batch size. If your latency budget for TTFT is strict, aggressive batching will blow it out. You need to monitor P99 latency, not just averages, to ensure your worst-case users aren't suffering.

Advanced Techniques: Speculative Decoding and Quantization

If standard inference isn't fast enough, you can look to architectural optimizations. Two of the most effective techniques are speculative decoding and quantization.

Speculative Decoding is a technique where a smaller, faster 'draft' model predicts multiple tokens, which a larger 'target' model then verifies in parallel. This trades extra compute for reduced latency. Instead of generating tokens one by one sequentially, the target model checks a block of predicted tokens at once. If the predictions are correct, you save significant time. Documented implementations show latency reductions of 2x to 4x. This is particularly valuable for reasoning-intensive workloads where the decode phase is long. The downside? You need to deploy two models, increasing memory footprint and complexity.

Quantization is the process of reducing the precision of model weights (e.g., from FP16 to INT8 or FP4) to reduce memory usage and increase bandwidth efficiency. Smaller weights mean less data to move from memory to compute units. This directly helps the memory-bound decode phase. For instance, the GPT OSS 20B model uses MXFP4 quantization, allowing it to run on less memory than its BF16 counterpart. While there is a slight risk of accuracy degradation, modern quantization methods (like AWQ or GPTQ) often maintain near-original quality for many tasks. If your latency budget is tight, quantization is usually the first lever to pull.

Hardware Constraints and Cost Implications

Your latency budget is ultimately bounded by your hardware. You can't squeeze infinite speed out of a single consumer-grade GPU. The choice of model size dictates your infrastructure requirements, which in turn dictate your latency ceiling.

Consider the following hardware realities for popular model sizes:

• 8B Parameter Models: Can fit on a single NVIDIA H100 (80GB). Ideal for high-throughput, low-latency edge cases or simple chatbots. Accuracy may suffer compared to larger models.
• 109B Parameter Models: Require at least three H100s (240GB total VRAM). This setup allows for better quality but introduces communication overhead between GPUs, slightly impacting latency. Monthly costs for serving 10k requests/day can reach $15,000.
• 400B+ Parameter Models: Need at least 10 H100s. These are typically reserved for complex reasoning tasks where latency is less critical than accuracy. Costs scale linearly with hardware.

Mixture of Experts (MoE) architectures offer a middle ground. An MoE model like GPT OSS 20B has 20B total parameters but only activates 3.6B per forward pass. This sparse activation allows for faster inference and lower memory usage compared to a dense 20B model, provided your routing logic is efficient. However, MoE models introduce routing overhead, which can negate some latency benefits if not implemented carefully.

Building Your Latency Budget Checklist

Before deploying an interactive LLM app, run through this checklist to ensure your budget is realistic:

1. Define User Expectations: Is this a real-time voice assistant (sub-200ms TTFT) or a text-based chat (under 1s TTFT)?
2. Analyze Input/Output Ratios: Are prompts long (RAG) or short (chat)? Long prompts hurt TTFT; long outputs hurt TPS.
3. Select Model Size: Choose the smallest model that meets your accuracy threshold. Smaller = Faster + Cheaper.
4. Choose Hardware: Match GPU memory bandwidth to your decode phase needs. Don't overpay for compute if you're memory-bound.
5. Apply Optimizations: Implement quantization first. Add speculative decoding if TTFT is still too high.
6. Set Batching Limits: Cap max_num_seqs to protect P99 latency. Monitor continuously.
7. Implement Caching: Cache frequent queries or intermediate RAG results to bypass inference entirely for repeat questions.

Remember, latency isn't just a technical metric; it's a product feature. A fast, slightly less accurate model often provides a better user experience than a slow, perfect one. Measure, iterate, and keep your users engaged.