What Happens to Your Data Pipeline at Scale

At 8 GPUs the data loader is fine. At 64 it's the bottleneck. How data loading, preprocessing, and shuffling change as you scale up training jobs.

Single-node data loading works until it doesn't

On one machine with a few GPUs, PyTorch's DataLoader with a handful of workers is fine. Data sits on local NVMe, preprocessing keeps up, GPU utilization looks healthy.

• CPU cores and disk bandwidth are sized for the GPU count on that machine
• Everything changes when you go multi-node - the ratio breaks

More GPUs means more demand on the same pipeline

Scale from 8 GPUs to 64. Each GPU still needs the same batch throughput. That's 8x the data loading demand.

• If data lives on network storage (S3, GCS, NFS), bandwidth becomes the constraint
• If preprocessing is CPU-heavy (image decoding, tokenization, augmentation), CPU cores become the constraint
• GPU utilization drops not because the model is slow, but because GPUs are waiting for data

Shuffling gets expensive

Random access into a large dataset is fast on local disk, slow over a network.

• True random shuffling across terabytes of data on cloud storage is impractical
• Common workaround: shuffle shards, then shuffle within shards. Not perfectly random but close enough
• WebDataset and Mosaic StreamingDataset were built to handle this - sequential reads with shard-level shuffling

Preprocessing needs to move off the training node

At scale, you can't afford to decode images or tokenize text on the same machines running the GPUs.

• Pre-process and write to an optimized format (WebDataset tars, Mosaic MDS, pre-tokenized shards)
• Trade storage cost for training speed - store the preprocessed version, not the raw data
• Data preparation becomes its own pipeline with its own compute budget

Caching and locality matter more than you think

• First epoch is always slower than subsequent epochs if data is being pulled from remote storage
• Local SSD caching on training nodes helps - but only if the dataset fits
• At petabyte scale, you're designing a distributed data serving system, not just calling DataLoader