Home/Data

Dataset Engineering

Data is the most important factor in LLM quality. Learn how to build, clean, and manage datasets.

DATASET PIPELINE

1. Collect

Web crawling, API access, public datasets (The Pile, C4, Dolma). Track sources and licenses.

2. Clean

Remove duplicates, filter low-quality text, deduplicate, detect language, remove PII.

3. Format

Convert to standard format (JSONL), tokenize, pack sequences, create train/val splits.

KEY CONCEPTS

Contamination occurs when evaluation data appears in your training set. With web-scale pretraining, some contamination is nearly inevitable.

Mitigation: Use dedicated clean test sets. Remove known benchmark sources from training data. Report contamination estimates.

Exact deduplication removes identical documents. Near-duplicate detection (MinHash + LSH) removes substantially similar documents.

Why it matters: Duplicate data causes overfitting, inflated benchmark scores, and poor generalization.

Different data sources contribute differently. Web text, books, code, and papers each teach different capabilities.

Strategy: Start with proportional mixing by source quality. Adjust based on which capabilities you want to emphasize.