GitHub - 1amageek/swift-gigatoken: 670 MB/s Swift tokenizer inspired by Marcel Roed’s gigatoken. Zero-copy, NEON/CRC32, WebAssembly, and Embedded Swift.

Byte-level BPE tokenization with a Pure Swift portable data path for Native, WebAssembly, and Embedded Swift, plus compile-time-gated native kernels where the target contract guarantees the required instructions.

On an Apple M4 Max, the single-threaded warm r50k_base path reaches 988.76 MB/s while producing exactly the same token IDs as the reference implementation.

Acknowledgements

This project is an independent Swift implementation inspired by gigatoken, created by Marcel Roed. The original project established the gigatoken name and demonstrated high-throughput language-model tokenization. swift-gigatoken exists because of that work, and references to this implementation's lineage should credit the original project and author.

Performance

swift-gigatoken is designed for tokenizer hot paths where copying, allocation, branching, and cache misses are measurable costs.

That result is 1.014x the throughput of the original Rust implementation on the same Apple M4 Max, model, input bytes, and single-threaded benchmark path. The strict interleaved gate rejects concurrent compiler activity and more than 5% paired common-mode drift instead of treating a busy host as an implementation regression. Every process in the accepted run preserved the full token identity.

These are measured results on one machine rather than a universal throughput guarantee. See the complete benchmark methodology for toolchain versions, pinned revisions, input hashes, and reproduction steps.

General tokenizer API comparison

The same 16 MiB input was also encoded through the public APIs of two widely used general-purpose tokenizer packages. Every implementation produced the same 4,929,342-token count and the same 9bb0d84c9a7a327d checksum over all token IDs.

This is an end-to-end comparison of public encode APIs, including each API's required output materialization. It is not a claim that the underlying tiktoken or Hugging Face native kernels alone are slower by those factors. Python input decoding, model loading, garbage collection, and checksum calculation are outside the timed warm encode region. Hugging Face parallelism is disabled so all four paths remain single-threaded. The absolute numbers differ from the stricter interleaved Swift/Rust gate above because this four-implementation comparison was recorded in a separate low-load run.

Why it is fast

Features

• Foundation-free GigaTokenCore with typed failures and no silent fallback.
• Native Apple ARM64 acceleration with a Pure Swift portable implementation.
• WebAssembly and Embedded Swift build and smoke-test coverage.
• Caller-owned, reusable storage for allocation-sensitive applications.
• Contiguous UInt32 token access for MLX Swift, Core AI, and other runtimes.
• Exact r50k_base parity against pinned reference fixtures.
• Exact cold/warm parity on a pinned 110,126-byte corpus that crosses multiple 256-pretoken batches and grows both cache paths.

The current compatibility baseline is r50k_base. The package separates the Foundation-free tokenizer core from host model loading and benchmarking.

import Foundation
import GigaToken

var tokenizer = try GigaTokenizer(r50kModelAt: modelURL)
let tokens = try tokenizer.encodeOrdinary("Hello, world!")
let text = try tokenizer.decode(tokens)

Build

The package baseline pins one Swift 6.4 development snapshot and matching standard and Embedded Wasm SDKs:

source Scripts/swift-toolchain.sh
verify_swift64_environment
swift64 build
swift64 build --swift-sdk "$SWIFT_WASM_SDK" --target GigaTokenCore
swift64 build --swift-sdk "$SWIFT_EMBEDDED_WASM_SDK" --target GigaTokenCore
Scripts/verify-portability.sh

GigaTokenCore accepts rank-ordered token bytes directly, so Wasm and Embedded Swift applications can embed model tables without Foundation, JSON, Base64, filesystem APIs, or Rust. The GigaToken host target parses the standard .tiktoken text format when Foundation is available.

The tokenizer's public API, storage ownership, BPE logic, and portable hash are Pure Swift. On ARM64 targets whose platform contract guarantees CRC32, VectorKernelsNative contributes only the two-instruction cache-hash primitive. On macOS, release codegen validation requires those intrinsics to inline into the Swift hot loop. Other ARM64 targets use the same two-instruction kernel behind one C ABI call. Targets without CRC32 use the Pure Swift pair hash.

Low-level data path

The performance API borrows the input storage and writes directly into a move-only, caller-owned token allocation:

import GigaTokenCore

var encoder = BPEEncoder(model: model)
var output = TokenBuffer(minimumCapacity: input.count / 3)

try input.withUnsafeBufferPointer { bytes throws(TokenizerError) in
  try encoder.encodeOrdinary(bytes, appendingTo: &output)
}
output.withUnsafeBufferPointer { tokens in
  consume(tokens)
}

This path does not materialize input slices or an intermediate token array. String, [TokenID], bytes(for:), and decode(_:) are convenience/output boundaries and intentionally materialize owned results. Use withTokenBytes(for:_:), borrowed buffer overloads, and TokenBuffer when the caller needs scoped views and reusable storage.

For MLX Swift, Core AI, or another tensor runtime, depend on TokenEncoding and consume the contiguous UInt32 IDs without an intermediate array:

output.withUnsafeRawTokenIDs { tokenIDs in
  constructRuntimeInput(tokenIDs)
}

The pointer is valid only for the closure. Runtime-specific adapters and device transfer are intentionally outside this package; the borrowed buffer and ownership rules are the stable connection contract.

The r50k test fixture is OpenAI's public r50k_base.tiktoken model with SHA-256 306cd27f03c1a714eca7108e03d66b7dc042abe8c258b44c199a7ed9838dd930.

See Documentation/Architecture.md for target boundaries and correctness requirements, and Documentation/Benchmark.md for the reproducible Rust and general-tokenizer comparisons.