GitHub - jamierpond/tokencount: counts tokens! like wc, but for tokens!

1 min read Original article ↗

Skip to content

Navigation Menu

tc

Token counter CLI. Like wc but for LLM tokens.

git diff | tc        # before pasting into chat
tc *.py              # check project size

Install

uv tool install git+https://github.com/jamierpond/tokencount

Usage

tc file.txt                    # single file
tc *.py                        # multiple files
cat file.txt | tc              # stdin
tc -e cl100k_base file.txt     # different encoding
tc --json file.txt             # JSON output

Output

   475  counts.py
   878  test_main.py
 1,565  main.py

  386 lines, 11.36 KB

  2,918 tokens (o200k_base)
  Somewhere between The Cat in the Hat and The Waste Land
  139% of The Cat in the Hat, 69% of The Waste Land

2918

Sorted by token count. Total on stdout for piping. Warns if over 1M tokens.

Encodings

  • o200k_base (default) - GPT-4o, Claude
  • cl100k_base - GPT-4, GPT-3.5-turbo