Show HN: Skeletoken, a Python package for editing model tokenizers

1 points by stephantul a month ago · 0 comments · 1 min read

Hello!

I just released a new version of Skeletoken, a package for editing tokenizers. New in this version is the ability to automatically adapt a model to an edited tokenizer.

For example, you can a new token to your tokenizer, and then ask skeletoken to add new token indices in the correct embedding tables.

Let me know what you think!

No comments yet.