Tokens are the fundamental units of data, such as words, subwords, or characters, that an AI model processes, generated via algorithms like Byte Pair Encoding (
Tokenizers split text into subword units: common words may be one token while rare words split into several; a rough English heuristic is one token ≈ 4 characters or ¾ of a word. Models read and generate token sequences, and every operational property follows: context limits are token counts, latency scales with tokens generated, and API pricing is per million tokens (with separate input and output rates).
Tokens are the unit of cost and capacity in AI systems. Teams that instrument token usage catch runaway costs, fit context budgets intelligently (what to retrieve, what to summarize), and price their own AI features sustainably. Token economics is now a standard line in AI product P&Ls.
Generating tokens is sequential and compute-heavy, while reading input parallelizes, so output tokens typically cost several times more. Cost-aware designs minimize generated length (structured, concise outputs) even when inputs are large.
Roughly 1.3 tokens per English word: a one-page memo ≈ 600–800 tokens, a 10-page report ≈ 6–8k. Code and non-Latin scripts tokenize less efficiently, often doubling counts for the same visual length.
Providers cache the processed state of repeated prompt prefixes (like long system prompts or shared documents) and charge cached input at a steep discount: one of the highest-leverage cost optimizations in production AI.