ai-tokenizer

1 min read Original article ↗

A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

GitHub NPM

Tokens 0

Characters 0

Encoding -

Encode Time -

Input Text

Tokens

Select a model and enter text to see tokens

Note: Token counts are approximations based on the best available encoding for each model. Accuracy varies by model and context (typically 95-100%). See accuracy table for details.