Self-supervised learning trains models on unlabeled data by deriving the supervision from the data itself: predicting masked words, the next token, or missing i
Training models on unlabeled data by deriving the supervision from the data itself: predicting masked words, the next token, or missing image patches. No human labeling is required, which is what makes web-scale pre-training possible.
It is the foundation of the foundation-model era: nearly everything a large language model knows was learned self-supervised from raw text, with labeled data reserved for the much smaller post-training phase.