Tokenization
Tokenization is the process of splitting text into discrete units called tokens, which a language model consumes and produces. Most modern systems use subword schemes such as byte-pair encoding or SentencePiece, where common words map to single tokens and rare strings split into pieces. Token counts drive context limits, latency, and pricing, so they directly affect prompt design and cost.
Definition
Tokenization is the process of splitting text into discrete units called tokens, which a language model consumes and produces. Most modern systems use subword schemes such as byte-pair encoding or SentencePiece, where common words map to single tokens and rare strings split into pieces. Token counts drive context limits, latency, and pricing, so they directly affect prompt design and cost.
Examples
- The word "tokenization" splitting into "token" and "ization" under a BPE tokenizer.
- Counting tokens before sending a 12-page contract to stay within the context window.
- Comparing token cost between English and Japanese to estimate per-request pricing.
Related terms
No credit card required
Frequently asked
- What is Tokenization?
- Tokenization is the process of splitting text into discrete units called tokens, which a language model consumes and produces. Most modern systems use subword schemes such as byte-pair encoding or SentencePiece, where common words map to single tokens and rare strings split into pieces. Token counts drive context limits, latency, and pricing, so they directly affect prompt design and cost.
- How is Tokenization used in practice?
- Tokenization is commonly used in scenarios such as: The word "tokenization" splitting into "token" and "ization" under a BPE tokenizer.; Counting tokens before sending a 12-page contract to stay within the context window.; Comparing token cost between English and Japanese to estimate per-request pricing..
- How does yno.ai support Tokenization?
- yno.ai supports Tokenization as part of its agentic AI platform — combining multiple frontier models with extensible tool use.
- What other concepts are related to Tokenization?
- Tokenization relates to Large Language Model, Context Window, Embedding. Each links to its own glossary page below.