Pre-training is the foundational stage of building a large model, where it ingests massive, diverse datasets to learn general patterns of language, code, or per
Pre-training is the foundation phase: a model learns from trillions of tokens of broad data via self-supervised next-token prediction, absorbing language, world knowledge, and reasoning patterns into its weights over weeks of massive GPU computation. Post-training (instruction tuning, RLHF) then shapes that raw capability into helpful, safe behavior.
Pre-training sets the capability ceiling everything else refines, and its scale (compute, data, cost) is why frontier models come from a handful of labs while everyone else builds on top. Understanding the pre-train/post-train split explains model behavior: knowledge and reasoning come from pre-training; persona and compliance from post-training.
Pre-training builds general capability from massive unlabeled data at enormous cost; fine-tuning cheaply adapts that capability to tasks with small curated sets. Builders almost always start from someone's pre-trained model.
Weights freeze when pre-training ends: later events simply aren't in them. Retrieval and tools provide current facts; expecting post-cutoff knowledge from the model itself invites hallucination.
Extending pre-training on domain corpora (medical literature, codebases) before instruction tuning: deeper than fine-tuning for vocabulary-rich specialties, at correspondingly higher data and compute cost.