Token

What is a Token

A token is the basic unit a model actually reads and writes, roughly a common word or a fragment of a longer word, not quite the same thing as a word or a character. The context window's size limit, an agent's usage-based cost, and even how a Skill's file size gets measured in this glossary are all expressed in tokens rather than raw character counts.

As a rough rule of thumb, English text runs close to four characters per token on average, which is the estimate used for the Skill entry's hero stat in this glossary, since an exact tokenizer was not available in this environment. Common short words are often a single token each, while a rare or technical word can split into two or three.

A token is not interchangeable with a word for planning purposes, code and unusual identifiers tend to tokenize less efficiently than plain prose, sometimes splitting a single variable name into several tokens. This is one reason why the same amount of visible text can cost meaningfully different amounts of a context budget depending on what kind of content it is.