You’ve got the GPUs. You’ve got the model architecture. But when you try to feed a petabyte of text into your training loop, the pipeline chokes. It’s not always a compute problem; often, it’s a plumbing problem. If your storage can’t keep up with your accelerators, those expensive H100s sit idle, burning cash while waiting for bytes. Efficient sharding and data loading are the unsung heroes that make large language model (LLM) training at scale actually possible.
Why Standard Loading Fails at Scale
Think about the last time you trained a small model on a laptop. You loaded the whole dataset into RAM, shuffled it, and fed batches to the CPU/GPU. Simple. Now multiply that by a thousand. A petabyte is roughly one million gigabytes. No single server has enough RAM to hold that, let alone shuffle it efficiently. If you try to stream raw files directly from object storage like S3 or GCS for every batch, network latency becomes your enemy. The GPU finishes a forward pass in milliseconds, but fetching the next batch takes seconds. That gap? That’s wasted money.
The solution isn't just "more bandwidth." It's restructuring how data is stored and accessed. This is where sharding comes in. Instead of millions of tiny JSONL or Parquet files, you serialize data into large, sequential shards-think .tar, .tgz, or custom binary formats. These shards act as contiguous blocks of memory-mapped data. When a worker node needs a batch, it doesn't open ten thousand files; it reads a slice from one large file. This reduces metadata overhead and maximizes sequential read throughput, which is what modern storage systems are best at.
The Memory Math: Why We Need Distributed States
Before we dive deeper into storage, let’s look at why the compute side demands this structure. Training an LLM isn't just about storing weights. During backpropagation, you need three things in memory simultaneously: parameters, gradients, and optimizer states (like Adam’s momentum). For a 70-billion parameter model, this requirement scales roughly as $Memory \approx 3-5 \times Model Size$. Even with half-precision (bfloat16), fitting a 70B model on a single 80GB A100/H100 is impossible without tricks.
This is where Sharded Data Parallelism (SDP) a technique that distributes optimizer states, gradients, and parameters across multiple GPUs to reduce per-device memory usage enters the chat. SDP modifies standard data parallelism. Instead of replicating the full optimizer state on every GPU, it shards these states across the group. Each GPU holds only a fraction of the optimizer state. When a gradient update is needed, all-reduce operations gather the necessary pieces. This frees up massive amounts of VRAM, allowing you to use larger batch sizes or longer sequence lengths-both critical for LLM performance.
Tiered Storage Architecture
You can’t train on cold storage. And you can’t afford to put a petabyte of hot data on NVMe SSDs forever. The winning strategy is tiered. Most production pipelines follow a three-layer hierarchy:
- Cold/Warm Layer (Object Storage): AWS S3, Google Cloud Storage, or Azure Blob. This is your source of truth. It’s cheap, durable, and scalable. Raw data lives here.
- Hot/Caching Layer (Distributed FS or Cache): Systems like Lustre, GPFS, CephFS, or specialized caches like AIStore. This layer sits closer to the compute cluster. It stores the active shards currently being used for training epochs.
- In-Memory Buffer: The local RAM of the compute nodes. Prefetchers load shards here before the GPU needs them.
The flow is predictable: Data moves from Object Storage to the Distributed File System (DFS) or cache, then to local RAM, then to GPU VRAM. By pre-fetching shards into the DFS or local cache, you hide the latency of object storage. Tools like AIStore an open-source distributed storage system designed specifically for AI/ML workloads, offering high-performance access to unstructured data excel here because they understand ML-specific access patterns, such as global shuffling of shard names to eliminate bias.
Data Loading Frameworks and Prefetching
Having the right storage layout is half the battle. The other half is the loader. PyTorch’s standard `DataLoader` combined with `DistributedSampler` works, but it struggles at petabyte scale due to Python’s Global Interpreter Lock (GIL) and inefficient file handling. Enter specialized frameworks:
- WebDataset: Designed for streaming tarballs. It avoids opening/closing files repeatedly, making it ideal for image/text pairs stored in tar archives.
- NVIDIA DALI: Offloads preprocessing (decoding, augmentation) to the GPU or dedicated CPUs, keeping the main training thread free.
- TF.data: TensorFlow’s native pipeline, which supports automatic prefetching and caching within the graph execution engine.
A key feature of efficient loaders is prefetching. While the GPU computes batch $N$, the CPU should already be decoding batch $N+1$ and $N+2$. If your loader is synchronous, you’re bottlenecked. Asynchronous loaders decouple I/O from computation, ensuring the GPU never waits for data unless absolutely necessary.
Comparison of Sharding Strategies
Not all sharding is created equal. Here’s how different approaches stack up against common pain points:
| Strategy | Best Use Case | Pros | Cons |
|---|---|---|---|
| File-Level Sharding | Small to medium datasets (<10TB) | Simple implementation; easy debugging | High metadata overhead; slow random access |
| Tarball Sharding (WebDataset) | Large multimodal datasets | Sequential I/O optimized; low file count | Harder to modify individual samples; requires repacking |
| Binary Shard Files | Petabyte-scale text corpora | Fastest read speed; memory-mappable | Complex serialization/deserialization logic |
| Database Indexing (Iceberg/Delta) | Data lakes requiring ACID transactions | Schema evolution; time travel; query flexibility | Higher latency for raw bulk reads compared to flat files |
Practical Implementation Tips
If you’re setting up a new pipeline, don’t over-engineer day one. Start with a baseline: split your data into 1-10GB shards. This size is large enough to amortize file open costs but small enough to fit comfortably in local disk caches or RAM buffers.
Watch out for the "small file problem." If your tokenizer outputs millions of tiny JSON objects, packing them into tarballs or binary shards is non-negotiable. Also, consider shard-level shuffling. Randomly accessing samples within a shard is fine, but randomly accessing shards across the entire dataset ensures better convergence than strictly sequential epoch processing. However, ensure your shuffle buffer is large enough. A small buffer leads to correlated batches, which hurts model generalization.
Finally, monitor your GPU utilization. If it’s below 90% during steady-state training, your data pipeline is likely the bottleneck. Profile your I/O wait times. Are you hitting network limits? Disk seek times? Or is the CPU struggling to decode tokens? Adjust your prefetch factor accordingly. Sometimes, simply increasing the number of worker processes in your DataLoader yields more benefit than buying faster GPUs.
Frequently Asked Questions
What is the ideal shard size for LLM training?
There is no single magic number, but 1GB to 10GB is a common sweet spot. Smaller shards increase metadata overhead and file open/close latency. Larger shards may exceed local cache capacities or cause long tail latencies if a single shard fails to load. The goal is to maximize sequential read throughput while keeping individual shard retrieval times under a few hundred milliseconds.
How does sharded data parallelism differ from tensor parallelism?
Tensor parallelism splits the model itself (layers/matrices) across GPUs, requiring frequent communication between GPUs during each layer. Sharded data parallelism (SDP) keeps the model replicated but splits the optimizer states and gradients across GPUs. SDP generally requires less inter-GPU communication than tensor parallelism, making it more scalable across nodes, while tensor parallelism is better suited for fitting very large models within a single node’s high-bandwidth NVLink domain.
Do I need a distributed file system like Lustre for petabyte-scale training?
Not necessarily, but it helps significantly. Modern cloud-native solutions like AWS FSx for Lustre or Google Parallelstore provide high-throughput POSIX-compatible interfaces. Alternatively, caching layers like AIStore or Alluxio can sit on top of object storage, providing similar benefits without managing a separate cluster. If your object storage provider offers high-performance endpoints (e.g., S3 Express One Zone), you might skip the traditional DFS entirely.
How do I handle data shuffling without loading everything into RAM?
Use two levels of shuffling. First, shuffle the order of shards globally at the start of each epoch. Second, use a local shuffle buffer within the data loader to randomize samples within a limited window of the current shard. This approximates true random sampling without needing to index or sort the entire petabyte dataset in memory.
Can I use Apache Iceberg or Delta Lake for LLM training data?
Yes, especially if you need schema evolution or ACID compliance during data curation. However, for pure training throughput, reading directly from Parquet/ORC files via a specialized loader is often faster than querying through a table format’s metadata layer. Many teams use Iceberg for data management and export stable snapshots to flat shard files for the actual training runs.