Attention is the neural mechanism that lets each token in a sequence weigh its relevance to every other token and blend their information accordingly: how 'it'
Attention is the neural mechanism that lets each token in a sequence weigh its relevance to every other token and blend their information, how 'it' finds its referent three paragraphs back. Self-attention stacked in layers is the engine of the transformer architecture.
Cost. Attention scales quadratically with sequence length, so long contexts are expensive to process. That cost is why long context windows are pricey and why efficient-attention variants and alternative architectures like state space models remain active research.