Training loss should decrease monotonically. Validation loss should decrease then plateau. If validation loss starts increasing while training loss continues decreasing, you are overfitting.
What to watch: A gap of more than 0.1-0.2 between train and val loss is a warning sign. The gap should narrow with more data or stronger regularization.
Contamination occurs when test data appears in the training set. This inflates benchmark scores artificially. With web-scale pretraining, some contamination is nearly unavoidable.
How to detect: Exact n-gram matching, fuzzy matching, and checking known benchmark sources against your training data. Tools like lm-evaluation-harness include contamination detection.
Good benchmarks are: (1) hard for humans to game, (2) aligned with the capability you want to measure, (3) large enough for statistical significance, and (4) resistant to contamination.
Common pitfalls: Multiple-choice formats can be gamed by memorization. Few-shot prompts can leak into training. Always report zero-shot and few-shot results separately.
Numerical benchmarks miss a lot. Systematically evaluate outputs by reading them. Check for: coherence, factual accuracy, instruction following, safety, and style consistency.
Technique: Create a diverse set of 50-100 evaluation prompts spanning your target capabilities. Have humans rate outputs on a 1-5 scale across multiple dimensions.