Tokenization is the process by which AI language and multimodal models break down text input into discrete units called tokens before processing it. Rather than working with raw character strings or whole words, models operate on these tokens, which may correspond to words, parts of words, punctuation marks, or other linguistic units depending on the tokenization scheme used. Understanding tokenization helps explain some of the ways AI models interpret and sometimes misinterpret prompts.

The specific tokenization scheme a model uses determines how it segments language. Common approaches include word-level tokenization, subword tokenization (where rare words are split into smaller components), and byte-pair encoding, which balances vocabulary size with the ability to handle novel words by breaking them into familiar subword units. Each token is converted into a numerical vector that the model processes. The total length of a prompt in tokens determines how much of the model's context window it occupies, which is why token limits are a practical constraint in working with AI systems. Prompt tokens also interact with each other in the model's attention mechanism, meaning the order and proximity of concepts in a prompt can influence how strongly they are associated in the model's interpretation.

For practical prompting, tokenization awareness means understanding that very unusual compound words or creative spellings may be tokenized in ways that reduce a model's ability to process them as intended, and that prompt length in tokens affects both processing cost and the model's ability to attend to all parts of a long, detailed prompt equally. For most creative generation work, tokenization operates invisibly in the background, but it becomes relevant when troubleshooting why specific words or unusual constructions are not being interpreted as expected.