Token-space font compiler

The listed presets use tokenizer-specific compiler paths. These include DeepSeek, OpenAI, Kimi, Qwen, GLM, Llama 3, Trinity and Laguna; Gemma and Gemini have experimental space-marker/byte-fallback support. ctok uses a separate minimum-cost backend. Raw ByteLevel BPE is also accepted with use_regex: false and add_prefix_space: false. Uploaded pipelines must match a supported profile; arbitrary normalizers, WordPiece and Unigram are not generally supported. Acceptance does not imply exact tokenization for every input.

The output is a single TTF with the tokenizer built into its shaping rules. No tokenizer script is needed to display it. It uses your font’s default variation and character coverage. Missing characters may use missing-glyph outlines or system fallback unless you enable fallback fonts, which embeds selected fallback outlines in the same token font. Coverage still depends on the selected sources. Noto Emoji is the default emoji fallback; Mutant Standard is an optional color source with separate license requirements. The precompiled DeepSeek + Inter preview already includes Noto Emoji.

Within a supported shaping run, each emitted token occupies 3 em. Extra token spacing defaults to zero and is capped at 0.5 em. The generated CSS leaves word spacing and line height alone. Browser tab stops, hard line breaks and whitespace processing do not behave like ordinary token cells.

N.B. Browser shaping runs, line breaks, normalization and complex scripts can change the result. Supported horizontal pair kerning is retained, but contextual shaping is not universally preserved. This is a token font, not an exact whole-message token counter.

The latest whitespace audit compared 17,904 BPE checks across six available font binaries and 2,355 ctok cases against local tokenizer references. Browser checks covered 60 widths and 20 wrapping samples. These are targeted tests, not a guarantee for all text; failures and coverage/capacity limits remain in the reports.

• DeepSeek whitespace: U+001C–U+001F controls are incorrectly classified as whitespace. For example, two spaces followed by U+001C merge into one space token. The directed audit found 59 ID mismatches in 2,984 checks. Ordinary isolated whitespace runs and CRLF cases passed; the control-separator bug remains unfixed.
• GLM-5.3 and Llama 3: the experimental presets use font-only regex boundaries, ranked BPE and whole-piece shortcuts. The tested Gutenberg passages and covered prose, code, whitespace and punctuation fuzz cases pass. Other pieces have a 64-byte / 32-round limit and display [BPE limit] if exceeded. The new directed whitespace audit reached that limit in 353 GLM and 354 Llama checks out of 2,984 each; the other checks passed. Browser shaping, source glyph coverage and formatting boundaries remain limitations.
• Trinity Large Thinking and Laguna M.1: experimental presets preserve the original preprocessing pipeline and ordered merges. The compiler validates those profiles instead of reducing them to raw BPE. Both passed all 2,984 directed whitespace checks. Browser shaping, source glyph coverage and formatting boundaries remain limitations; special tokens and automatic message framing are excluded.
• Hugging Face links and uploaded tokenizers: accepting a file does not establish compatibility with every tokenizer pipeline. Unsupported pre-tokenizers, normalizers and model types may be rejected; arbitrary uploads have not received the preset-specific audits.
• Claude / ctok: ctok is an unofficial approximation, not a verified implementation of Claude’s tokenizer. The full ctok vocabularies now fit, but the shipped minimum-cost backend has a 256-step search limit per connected component and displays [limit] on overflow. Long wrapped text can still render slowly. Invisible characters such as zero-width space, word joiner, BOM and soft hyphen can suppress word boundaries and add a token. v3/v4.7 also have quote-framing and leading-space normalization errors. Of 785 directed cases per version, v3 had 29 count/width and 45 semantic-only failures; v4.7 had 25 and 49; v4.8 had 16 and 8. Ordinary whitespace and runs through 257 characters passed. Equal counts alone can hide incorrect normalization. These failures remain unfixed; passing ctok comparisons does not establish agreement with Claude, and this audit made no Claude calls.
• Normalization and Unicode: a renderer can normalize text before the font sees it, changing token boundaries. General combining sequences remain limited. Qwen’s NFC support handles at most eight consecutive nonstarters after decomposition and displays [NFC limit] beyond that. Browser and native compiler Unicode versions can also differ for newly assigned characters.
• Rendering and font coverage: fonts cannot tokenize across separate shaping runs. Formatting, links, script changes, bidirectional text, fallback fonts and hard breaks can split those runs. Missing source-font glyphs and fallback may defeat equal token widths; complex-script shaping, ligatures and mark positioning are not universally preserved. Single-run tests do not guarantee identical results in every browser or app.
• Browser whitespace and wrapping: a leading space can form a separate shaping run under pre-wrap or break-spaces. At 24px, Hello world measured 216px rather than the whole-run 144px. Of 60 width comparisons, 43 matched, 15 differed because of tab/newline layout, and two exposed that leading-space boundary. Tested trailing/repeated spaces, NBSP, narrow NBSP and ideographic space matched whole-run widths. The earlier duplicated-token wrap example did not repeat in the checked 180px/320px screenshots, although wrapping can change tokenization. This does not establish correctness in every browser.
• Emoji and invisible characters: the current default Noto font passed 5,250 supported emoji checks for token identity, spacing and drawing. Missing artwork, private-use characters and emoji/text run boundaries remain limitations. Spaces and control characters in uploaded emoji sources are now excluded from emoji joining, fixing false emoji classification and glyph-count inflation. Non-emoji split-byte text and complex scripts remain limited.
• Message counts: special tokens, chat templates, multimodal content and automatic message overhead are not universally represented. Empty text cannot draw a framing token. These fonts visualize ordinary text, not exact API billing counts.

See the whitespace audit, ctok reproductions, and detailed discrepancy tracker for test results, repaired issues and remaining limitations.