Quantization stores a model's weights (and sometimes activations) in lower-precision numbers, 8-bit or 4-bit instead of 16/32-bit, shrinking memory footprint an
Storing a model's weights, and sometimes activations, in lower-precision numbers, 8-bit or 4-bit instead of 16 or 32-bit, which shrinks memory footprint and accelerates inference with modest quality loss.
Usually only modestly, and it varies by task: some tasks lose nothing, others degrade noticeably. The craft is measuring the quality trade per task. In 2026, 4-bit quantization routinely puts large models on single GPUs and capable models on phones.