HOW SHOULD I SCALE?
What it does: Each GPU gets a copy of the model. Data is split across GPUs. Gradients are averaged via all-reduce.
DDP: Full model on each GPU. Simple but memory-inefficient at scale.
FSDP: Shards model states across GPUs. 3-4x memory savings vs DDP. PyTorch's recommended approach.
When to use: Your model fits on one GPU but you want faster training. Or when using FSDP, when the model is too large for one GPU.
What it does: Splits individual layers across GPUs. Each GPU computes a slice of the matrix multiplication.
Best for: Single-GPU OOM situations where the model is too wide for one GPU's memory.
Requirement: Fast interconnect (NVLink) between the GPUs in each tensor-parallel group.
What it does: Splits layers across GPUs in sequence. GPU 0 processes layer 0-7, GPU 1 processes layer 8-15, etc.
Trade-off: Reduces per-GPU memory but introduces "bubble" time where GPUs wait for activations/grads.
Best for: Very large models where tensor parallelism alone is insufficient.
2D: Tensor parallelism within a node + data parallelism across nodes. The most common setup for 70B+ models.
3D: Tensor + pipeline + data parallelism. Used for the largest models (400B+ parameters).
Guideline: Start with data parallelism (FSDP). Add tensor parallelism if a single GPU can't hold a layer. Add pipeline parallelism if you have many GPUs per node but the model still doesn't fit.