Mixture of Experts is a model architecture in which many specialized subnetworks ('experts') exist but only a few activate for each token, selected by a learned
An architecture in which many specialized subnetworks ('experts') exist but only a few activate for each token, selected by a learned router. A model can hold enormous total capacity while spending only a fraction of it per inference.
Because it bends the cost curve: big-model quality at smaller-model serving cost. The trade-offs are memory footprint, since all experts must be loaded, and greater training complexity.