Reference — LLM Toolkit

KEY FORMULAS

Memory — Weights

weights = params × bytes_per_param

BF16 = 2 bytes, FP32 = 4 bytes

Memory — Full training (mixed precision, AdamW)

total ≈ P×(w + 4 + g + 8) + acts + overhead

P = params; w/g = stored bytes (2 for BF16); +4 FP32 master copy; +8 Adam m+v (FP32). So BF16 full-FT ≈ 16 bytes/param before activations. LoRA trains ~1% of params: gradients+optimizer apply to adapters only.

Memory — Optimizer (AdamW, FP32 states)

optimizer = params × 8 bytes (m + v in FP32)

Plus a 4-byte/param FP32 master weight copy when training in BF16/FP16. Pure-FP32 training needs no separate master copy.

Memory — Activations (rough lower bound)

acts ≈ batch × seq × hidden × layers × bytes

Scale proxy only. FlashAttention, packing, and per-layer saved tensors change it. Checkpointing trades ~20–30% extra compute for large activation savings.

Perplexity

PPL = exp(cross_entropy_loss)

Lower is better. PPL = 20 means the model is as surprised as if guessing uniformly among 20 tokens. Only comparable on the same test set and tokenizer.

MFU

MFU = actual_FLOPs / peak_FLOPs

Target: 40-60% for training. Measures hardware efficiency.

KV Cache Memory (per request batch)

KV = 2 × seq × layers × kv_heads × head_dim × bytes × batch

bytes = KV precision (often BF16/FP16 = 2). GQA reduces kv_heads vs query heads. Allocated per request; grows linearly with context length.

Effective batch size

eff_batch_tokens = per_device × grad_accum × num_gpus × seq

Steps per epoch = dataset_tokens ÷ eff_batch_tokens. Gradient accumulation trades step time for memory, not gradient math.

USEFUL COMMANDS

nvidia-smi                    # GPU status
nvidia-smi dmon -s u          # Real-time utilization
nvidia-smi pmon -g all        # Per-process memory
nccl-tests/build/all_reduce_perf -b 8 -f 2 -g 1  # NCCL benchmark

with torch.profiler.profile() as prof:
    # training step
print(prof.key_averages().table(sort_by="cuda_time_total"))

# Single node, 4 GPUs
torchrun --nproc_per_node=4 train.py

# Multi-node
torchrun --nnodes=2 --nproc_per_node=8 \
  --node_rank=$RANK --master_addr=$ADDR \
  train.py

import torch
print(f"Allocated: {torch.cuda.memory_allocated()/1e9:.1f}GB")
print(f"Cached:    {torch.cuda.memory_reserved()/1e9:.1f}GB")
torch.cuda.max_memory_allocated()  # Peak memory

PRECISION REFERENCE

Format	Bits	Max Value	Training Use	Inference Use
FP32	32	~3.4×10³⁸	Master weights, accumulation, reductions	Rarely
TF32	19 (internal)	~3.4×10³⁸	Ampere+ matmul (stored as FP32)	-
BF16	16	~3.4×10³⁸	Preferred	Common
FP16	16	~6.55×10⁴	Use with care	Common
FP8	8	~448 (E4M3)	Experimental	Hopper+ inference
INT8	8	127	-	Widespread
INT4	4	7	-	Aggressive quant

Verification

Specifications should be verified against current vendor documentation. Hardware capabilities change with driver and framework updates.