Distributed Training

What it does: Each GPU gets a copy of the model. Data is split across GPUs. Gradients are averaged via all-reduce.

DDP: Full model on each GPU. Simple but memory-inefficient at scale. If one GPU OOMs alone, DDP changes nothing.

FSDP: Shards weights, grads, and optimizer states across GPUs (~÷N memory each). 3-4x memory savings vs DDP for full training. PyTorch's recommended approach.

When to use: Model fits one GPU and you want speed (DDP); model exceeds one GPU (FSDP/ZeRO-3). Start FULL_SHARD + per-block wrapping; HYBRID_SHARD across slow inter-node links.

Failure modes: missing distributed sampler (all ranks train on the same data); all-gather storms from wrapping per-linear instead of per-block; checkpoint loads with mismatched world size or FSDP version.

What it does: Splits individual layers across GPUs. Each GPU computes a slice of the matrix multiplication.

Best for: Single-GPU OOM situations where the model is too wide for one GPU's memory — especially inference (70B on 2×24GB).

Requirement: Fast interconnect (NVLink) between the GPUs in each tensor-parallel group. TP over PCIe-only links is usually slower than fewer GPUs.

Rule: TP group stays inside one NVLink domain (typically ≤8 GPUs); kv_heads must divide evenly by the TP degree.

What it does: Splits layers across GPUs in sequence. GPU 0 processes layer 0-7, GPU 1 processes layer 8-15, etc.

Trade-off: Reduces per-GPU memory but introduces "bubble" time where GPUs wait for activations/grads. Bubble fraction ≈ (stages−1) ÷ micro-batches — keep micro-batches ≥ 4× stages.

Best for: Very large models where tensor parallelism alone is insufficient, or pipeline stages across nodes with TP inside nodes.

2D: Tensor parallelism within a node + data parallelism across nodes. The most common setup for 70B+ models.

3D: Tensor + pipeline + data parallelism. Used for the largest models (400B+ parameters).

Expert parallelism: for MoE models — experts live on different GPUs, tokens route via all-to-all. Memory plans use total params, latency plans use active params.

Guideline: Start with data parallelism (FSDP). Add tensor parallelism if a single GPU can't hold a layer. Add pipeline parallelism if you have many GPUs per node but the model still doesn't fit. Measure all-to-all bandwidth before big MoE runs.