Settings

Theme

Can we reconstruct a tokenizer using only two oracles from the chat API?

old.reddit.com

1 points by bayes-song 11 days ago · 1 comment

Reader

bayes-songOP 11 days ago

From the chat APIs alone, we can extract two useful oracles: Token length oracle: Given any string s, return len(tokenize(s)).

Prefix token oracle: Given a string s and integer n, return the string decoded from the first n tokens of tokenize(s).

The first oracle (token count) is straightforward. The second oracle's main purpose is to resolve ambiguities in merge order. For example, without it we couldn’t distinguish between the segmentations (a, bc) and (ab, c). By leveraging the model’s ability to repeat text in a controlled way, we can construct this second oracle from the regular chat interface.

With only these two oracles, is it possible to reconstruct an equivalent tokenizer — one that produces the exact same token boundaries/segmentation on any input (up to token ID remapping)?

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection