Tokenization

Before a model processes text, it breaks it into tokens, small units (words, subwords, or symbols). For example, "marketing" might become [mark, et, ing]. The model predicts one token at a time. Token count affects cost and length limits: longer prompts = more tokens = higher usage cost.