Open-source server and client for the Xet protocol, Hugging Face's chunk-deduplicated storage for large, versioned files. Models and datasets are common examples; any large file that changes over time benefits.
The goal: the official tools (hf, huggingface_hub, hf_xet, git-xet,
git-lfs) work against your own infrastructure unmodified. Data stays in your
bucket, identity stays in your login system, and only changed chunks move.
A hosted version will be available at openxet.com.
Status: the storage server and the Hub API work.
openxet serveis a Xet CAS that stockhf_xet(1.7.0 and newer) uploads to and downloads from directly, with global dedup, verified uploads, scoped tokens and signed, range-bound download URLs. On top of it, stockhfandhuggingface_hub(2.2.0 and newer) create repositories, upload, download and list them, andgit cloneandgit pushwork over HTTPS on the same history, with large files through Git LFS andgit-xet. Reconstruction answers a 100k-term file within 50 ms at p99 on an idle node, gated in CI. It runs on local disk with a SQLite index; S3, GCS, Azure and Postgres come next, in M3. There is no release yet.
Try it
- Quickstart: hf and git: start a server, upload
a folder with
hf upload, download it, then clone, change and push it withgit. - Quickstart: storage only: the storage
layer on its own, with
hf_xetand theopenxetcommand line.
CI runs both quickstarts as written on every change.
Documentation
- Quickstarts: hf and git and storage only
- Guides: Git hosting and Git LFS for any Git host
- Server configuration, command line, exit codes, and the routes: Hub, Git, Git LFS, CAS and the OpenAPI document
- Security tests: each security requirement and the test that enforces it
- Benchmarks: the latency, fetch and thin-base gates and the benchmarks, with their last results
- Changelog
- Design, architecture decisions and the competitive matrix
Project
- Contributing and
AGENTS.mdfor agents - Governance
- Code of Conduct
- Security policy