Home/Data

Dataset Engineering

Data is the most important factor in LLM quality. Learn how to build, clean, and manage datasets.

DATASET PIPELINE

1. Collect

Web crawling, API access, public datasets (The Pile, C4, Dolma). Track sources and licenses per document — gate releases on license metadata, not memory.

2. Clean

Remove duplicates, filter low-quality text, deduplicate (exact then near), detect language, remove PII. Report tokens after filtering — raw bytes lie.

3. Format

Convert to standard format (JSONL), tokenize, pack sequences, create train/val/test splits before tuning anything — dedup across splits.

KEY CONCEPTS

Contamination occurs when evaluation data appears in your training set. With web-scale pretraining, some contamination is nearly inevitable.

Mitigation: Use dedicated clean test sets. Remove known benchmark sources from training data. Report contamination estimates.

Exact deduplication removes identical documents. Near-duplicate detection (MinHash + LSH) removes substantially similar documents.

Why it matters: Duplicate data causes overfitting, inflated benchmark scores, and poor generalization.

Different data sources contribute differently. Web text, books, code, and papers each teach different capabilities.

Strategy: Start with proportional mixing by source quality. Adjust based on which capabilities you want to emphasize. Cap any single domain around ~40% without a measured reason, and track domain shares before/after every filter stage.

Next actions
Open Data Planner →Mixing, splits & versioning →Licensing basics →Budget tokens →