Infrastructure Requirements for Serving Large Language Models in Production

Why Serving Large Language Models in Production Is Nothing Like Running a Website

Running a website? You spin up a few servers, maybe use a CDN, and you’re good. Serving a large language model (LLM) in production? That’s like trying to power a small city just to answer one question.

Models like Qwen3 235B need 600 GB of VRAM just to load. That’s not a typo. A single GPU can’t handle it. You need eight or more high-end GPUs, all talking to each other over ultra-fast networks. And that’s just the start.

If you’re thinking of deploying an LLM for customer chat, internal document summarization, or real-time code generation, you need to understand what’s really under the hood. It’s not about choosing the right framework. It’s about building a custom infrastructure from the ground up - and it’s expensive, complex, and unforgiving.

Hardware: GPUs, Memory, and Why You Can’t Just Use Any Server

Forget your old cloud VMs. LLMs demand specialized hardware. The baseline? NVIDIA H100 GPUs. They’re not the cheapest, but they’re the only ones that can keep up with modern models. An H100 delivers 3.35 TB/s of memory bandwidth - more than double what the older A100 offers. That’s not a luxury. It’s a requirement.

For a 7B-parameter model, you might get away with one or two GPUs. But once you hit 70B or more, you’re looking at 8+ GPUs in a single machine. And even then, you’re barely scratching the surface. Qwen3 235B, one of the largest open models, needs 600 GB of VRAM to run at full precision. That’s over 10 H100s - each with 80 GB of memory - just to load the weights.

And memory isn’t just about VRAM. You need fast system RAM too - 1 TB or more - to handle input/output buffers, caching, and intermediate computations. Then there’s storage. Model files can be 100+ GB each. You need NVMe SSDs, not regular SATA drives. AWS charges $0.084 per GB/month for NVMe storage. For a 500 GB model, that’s $42 a month just to keep it on disk.

Network speed matters too. If you’re distributing a model across multiple servers, you need 100+ Gbps connections. Otherwise, your GPUs sit idle waiting for data. This isn’t cloud computing - it’s high-performance computing in disguise.

Software Stack: Containerization, Orchestration, and the Hidden Complexity

Running a model on a GPU isn’t enough. You need to package it, deploy it, and scale it - without breaking anything.

Containerization is the first step. But Docker images for LLMs aren’t your typical 200 MB app. They’re 10+ GB. You’re packing model weights, CUDA drivers, Python libraries, and custom inference code into a single image. Get one version wrong - say, CUDA 12.1 instead of 12.4 - and your model won’t load. That’s why teams pin every dependency: base OS, GPU driver, PyTorch version, and even the exact commit hash of their inference code.

Then you need orchestration. Kubernetes is the default, but it’s not plug-and-play. You need to configure GPU resource requests, memory limits, and affinity rules so your pods land on the right machines. Horizontal Pod Autoscalers (HPA) help, but LLMs don’t scale like web servers. You can’t spin up a new pod for every request - it takes 30 seconds just to load the model into memory. Instead, you batch requests. Five prompts in one call. Ten in another. That’s how you get efficiency.

And you need tools like vLLM or Text Generation Inference. These aren’t just libraries. They’re optimized inference engines that handle attention caching, continuous batching, and memory pooling. Without them, your throughput drops by 70%. You’re not just deploying a model - you’re deploying a whole new software stack.

Costs: Cloud vs. Self-Hosted vs. API - The Real Trade-Offs

There are three ways to serve LLMs: cloud platforms, self-hosted clusters, or third-party APIs. Each has trade-offs.

Cloud services like AWS SageMaker or Google Vertex AI are easy. Start a g5.xlarge instance? $12/hour. But scale that to 10 H100s running 24/7? That’s $8,640 a month. Enterprise deployments often hit $100,000+ monthly. You get managed scaling, monitoring, and security - but you’re locked into their pricing model.

Self-hosted? You pay upfront. A single NVIDIA H100 server costs $30,000-$50,000. A full 8-GPU cluster? $500,000+. But once it’s running, your cost per inference drops by 40-60%. You control everything. You can optimize batching, use quantization, and avoid cloud markups. The catch? You need a team of MLOps engineers. Setup takes 3-6 months. One misconfigured network policy, and your model is unreachable.

Third-party APIs like OpenAI or Anthropic? Simple. Pay $0.005 per 1K tokens for GPT-3.5-turbo. No hardware. No maintenance. But you lose control. You can’t fine-tune. You can’t cache. You’re at their mercy for uptime, latency, and pricing changes. And if you’re processing millions of tokens daily, that adds up fast.

Here’s the truth: most enterprises don’t pick one. They pick hybrid. 68% use a mix of cloud, on-prem, and edge. Why? To balance cost, control, and compliance. Sensitive data stays on-site. High-volume, low-sensitivity tasks go to the cloud.

Optimization: Quantization, Batching, and the 40% Cost Cut

Without optimization, LLM serving is a money pit. The good news? You can cut costs by 40-60% without killing performance.

Quantization is the biggest lever. Converting a model from 16-bit to 8-bit cuts memory use in half. Go to 4-bit? You’re down to 25% of the original size. Qwen3 235B goes from 600 GB to 150 GB. That means you can run it on four H100s instead of eight. Accuracy drops? Maybe 1-3%. For most applications - chatbots, summarization, tagging - that’s acceptable.

Batching is next. Instead of processing one prompt at a time, group 16, 32, or even 64 prompts together. Modern engines like vLLM can do this dynamically, filling empty slots as new requests arrive. This boosts throughput by 3-5x. One server that handled 20 requests per minute now handles 100.

And don’t forget caching. If 10 users ask the same question - “What’s our Q4 revenue?” - you don’t need to run inference 10 times. Store the answer in Redis or Memcached. For repetitive queries, this can reduce load by 30%.

Neptune.ai’s 2024 study found that teams using these techniques cut monthly infrastructure costs by 25-40%. That’s not a nice-to-have. That’s survival.

Architecture: What You Need Beyond the GPU

LLM serving isn’t just about inference. It’s about the whole pipeline.

Most serious applications use Retrieval-Augmented Generation (RAG). That means pulling real-time data from a vector database - like Pinecone or Weaviate - before generating a response. So now you’re not just serving a model. You’re serving a system: query → retrieval → prompt engineering → inference → output.

That requires orchestration tools. LangChain and LlamaIndex aren’t optional anymore. They’re standard. Adoption jumped from 15% in 2024 to 62% in 2025. Why? Because manually wiring together prompts, retrievers, and memory is a nightmare. These frameworks handle it for you.

Security is another layer. Model weights are intellectual property. If someone steals them, you lose your competitive edge. That means encrypted storage, role-based access, network segmentation, and audit logs. Tools like Trivy scan containers for vulnerabilities before deployment. You don’t want a known CVE in your inference stack.

And monitoring? You need it. Not just CPU/GPU usage. You need latency per request, token throughput, error rates, and memory fragmentation. Set up alerts. If response time spikes above 1.2 seconds, auto-scale or switch to a backup instance. Aim for 99.9% uptime. That’s not a goal - it’s the baseline for enterprise apps.

Real-World Challenges: What No One Tells You

Here’s what actually breaks in production:

• GPU memory allocation - 78% of teams struggle with this. You think you have enough VRAM. Then your model crashes because of a tiny extra tensor. You need to profile every input size.
• Latency spikes - 65% of teams report inconsistent response times. It’s not the model. It’s the queue. If you don’t batch well, you get 200ms one second, then 2 seconds the next.
• Driver and version hell - CUDA updates break things. NVIDIA driver 550 works. 555 doesn’t. You need a locked-down environment.
• Underutilization - Most companies run GPUs at 35-45% utilization. That’s waste. You need dynamic scaling, not static clusters.

Best practices? Test everything in a sandbox first. Try quantization. Test batching. Measure latency with real user traffic. Don’t deploy on Friday. Don’t skip health checks. And never trust a cloud provider’s default settings.

The Future: What’s Coming in 2026

NVIDIA’s Blackwell GPUs, launched in March 2025, are 4x faster than H100s for LLMs. That’s a game-changer. But they’re expensive. And they’re not the end.

By 2026, Gartner predicts 50% of enterprise LLMs will use 4-bit quantization. 70% will use dynamic scaling. That’s because the cost of raw compute is unsustainable. Trillion-parameter models are coming. Without architectural innovation, they’ll be impossible to serve.

Specialized AI chips from startups are also emerging. Companies like Cerebras and Graphcore are building hardware designed only for inference. They’re not as powerful as NVIDIA yet - but they’re cheaper and more efficient.

And the biggest shift? Infrastructure is becoming a product. Platforms like Qwak and Northflank aren’t just tools - they’re managed services that handle scaling, monitoring, and security for you. You still pay, but you stop hiring MLOps engineers just to keep the lights on.

Next Steps: Where to Start

Don’t try to build everything at once. Start small:

1. Pick one use case - say, internal FAQ summarization.
2. Use a 7B model like Mistral 7B. It’s fast, cheap, and easy to optimize.
3. Deploy it on a single H100 using vLLM and Docker.
4. Measure latency, cost, and throughput.
5. Apply 4-bit quantization. See how much you save.
6. Add batching. See how much throughput improves.
7. Only then, consider scaling to larger models or hybrid setups.

LLM infrastructure isn’t a sprint. It’s a long-term engineering investment. Get the basics right, and the rest follows. Skip them, and you’ll burn cash - and your team’s sanity.