GitHub - schapman1974/briskdb: Ordinary SQLite files, one sharded database—parallel writes, PostgreSQL, HTTP, Rust, Python, and shard-safe IDs.

SQLite files. One sharded database.

BriskDB turns ordinary SQLite files into one database with parallel writes, PostgreSQL compatibility, HTTP access, and embedded Rust/Python APIs. It keeps SQLite's proven storage engine and tooling; BriskDB adds the routing layer, shard-safe IDs, cross-shard indexes, protocols, and operational guardrails.

Try it without a compiler · Download an alpha · Open the data browser · Follow MongoDB and MySQL

Why developers might care

• No SQLite fork. Each shard is an ordinary SQLite WAL database that normal tools can inspect.
• No central write lock. Writes to different shards use different WALs and can progress in parallel.
• No central ID write per row. Native range and hi/lo allocation provide collision-free generated IDs across shards and processes.
• Safe cross-shard pruning. Global uniqueness is authoritative; asynchronous indexes use verification, watermarks, Bloom filters, and min/max summaries so an optimization cannot silently hide a row.
• One engine everywhere. PostgreSQL, HTTP, Rust, and Python share routing, limits, cancellation, errors, and storage behavior.
• Operations are visible. The separate loopback administration listener serves /health, /metrics, admin JSON, and the browser without exposing those handlers on the data listener.

One engine, many ways in

flowchart LR
    subgraph Clients
        WEB[Browser + HTTP]
        PG[PostgreSQL clients]
        MONGO[MongoDB clients · planned]
        MYSQL[MySQL clients · planned]
        RUST[Rust embedding]
        PY[Python embedding]
    end

    WEB --> ENGINE
    PG --> ENGINE
    MONGO -.-> ENGINE
    MYSQL -.-> ENGINE
    RUST --> ENGINE
    PY --> ENGINE

    ENGINE[Protocol-neutral Rust engine] --> ROUTER[4,096 virtual buckets]
    ROUTER --> S0[(SQLite WAL · shard 0)]
    ROUTER --> S1[(SQLite WAL · shard 1)]
    ROUTER --> S2[(SQLite WAL · shard 2)]
    ROUTER --> SN[(SQLite WAL · shard N)]

The protocol adapters do not own database semantics. Routing, limits, cancellation, values, sessions, and execution live in the shared Rust engine, leaving room for more protocols and storage adapters later.

Browse the whole logical database

BriskDB serves a responsive, read-only data browser at /admin on its separate administration listener. It uses the same bounded engine paths as other clients, combines sharded rows into one logical view, reads global tables once, and preserves large integer values.

For the current local alpha:

http://127.0.0.1:7655/admin
username: admin
password: admin

The temporary credentials are a development convenience—not a security boundary—which is why both HTTP listeners currently require loopback addresses.

The unusual part: shard-safe generated IDs

BriskDB has two generated-ID designs for sharded tables:

• native_range_v1 gives every shard a non-overlapping positive 64-bit range. SQLite's own INTEGER PRIMARY KEY AUTOINCREMENT performs the actual allocation locally, with no central write for each inserted row.
• hilo_v1 durably leases blocks of 4,096 IDs from the manifest, then allocates in memory and hash-routes each ID. Crashes may leave gaps, but an ID is never reused.

Both policies are versioned in the manifest. Generated-key execution is still experimental and opt-in; the exact contract lives in Generated keys.

What works now

Where BriskDB fits

These projects solve different problems. This table is a compass, not a benchmark scoreboard.

Choose BriskDB when you want one local service or embedded engine to spread write contention across inspectable SQLite files while speaking familiar database protocols. Choose the others when a single SQLite file, replicated high availability, managed edge sync, or a mature multi-node PostgreSQL cluster is the real requirement.

Try it in 30 seconds

Install the published native wheel—no clone and no Rust compiler:

python -m pip install --only-binary=:all: briskdb
curl -fsSLO https://raw.githubusercontent.com/schapman1974/briskdb/main/examples/launch_demo.py
python launch_demo.py

The demo makes 32 routed writes from four Python threads, proves that all four ordinary SQLite shard files received rows, reads every row back, checks HTTP health, and starts the PostgreSQL listener. It uses a temporary directory and cleans up after itself. The GIF renderer executes this exact scenario, and CI tests it against every published wheel target.

To run the standalone service, download the matching macOS/Linux ARM64 or x86-64 archive from the latest GitHub release, then:

./briskdb --data-dir ./briskdb-data --shards 4

Open the data browser or inspect the service on the separate administration listener:

curl http://127.0.0.1:7655/health
curl http://127.0.0.1:7655/v1/ready
curl http://127.0.0.1:7655/v1/admin/catalog
curl http://127.0.0.1:7655/metrics

Enable the PostgreSQL listener explicitly. Simple and parameterized text/binary prepared queries share the same bounded engine path:

./briskdb --data-dir ./briskdb-data --postgres-listen 127.0.0.1:5433
psql -h 127.0.0.1 -p 5433 -d default

That local development form is unauthenticated and therefore loopback-only. The PostgreSQL quickstart shows the four settings for TLS plus SCRAM-SHA-256; secure mode is required for any remote bind.

Registered tables can also be queried over HTTP:

curl -X POST http://127.0.0.1:7654/v1/query \
  -H 'content-type: application/json' \
  -d '{"sql":"SELECT id, name FROM widgets WHERE id = ?1","params":["widget-1"]}'

The versioned HTTP contract defines requests, response shapes, value encodings, request IDs, eligible-write idempotency, bounded row streaming, limits, and errors. GET /v1 reports the supported API version and configured result ceilings. The checked OpenAPI 3.1 artifact describes the exact v1 machine surface from the same route and DTO definitions and is included in Cargo, native archive, and Debian distributions. The HTTP listener contract defines the strict route split, configuration, and shared startup/shutdown behavior. The data listener defaults to 127.0.0.1:7654; administration defaults to 127.0.0.1:7655 and can be disabled with --admin-listen disabled.

Have an existing SQLite database? Use the offline SQLite importer. Linux releases also include .deb packages with a hardened systemd service.

Embedding in Rust starts with BriskDb::open() or the validated builder. The embedded Rust guide includes a complete listener-free example. Choose a shard count when creating data; later opens detect it from the manifest and reject explicit mismatches. Use default-features = false with the embedded feature for SQL-only embedding. Select documents and explicitly enable DocumentSupport to submit native BSON commands through BriskDb or an owned BriskSession. Both forms use the same document engine and routing plans as future protocol adapters; see the crate feature map.

Python runs the same engine directly in-process. It starts no listener by default, but Database.serve() can attach data HTTP, administration HTTP, and PostgreSQL listeners (remote PostgreSQL requires its TLS/SCRAM arguments):

with briskdb.open("./data", shards=4) as db:
    with db.serve(postgres="127.0.0.1:0") as server:
        print(server.data_address, server.admin_address, server.postgres_address)

The wheel also exposes the current protocol-neutral document slice through sync and asyncio sessions. Install PyMongo for its BSON value classes, then enable document commands on the database handle:

python -m pip install briskdb pymongo

from bson import ObjectId
import briskdb

with briskdb.open("./data", shards=4, documents=True) as db:
    with db.session() as session:
        session.create_collection("app", "notes")
        note_id = ObjectId()
        session.insert_one("app", "notes", {"_id": note_id, "body": "hello"})
        print(session.find("app", "notes", {"_id": note_id})["documents"])

PyMongo remains optional and is loaded only when a document method runs, so a SQL-only installation has no BSON dependency. The current slice supports collection/index metadata, one explicit-ID insert, empty or exact-_id find/count, and exact-_id deletion. Secondary index declarations remain pending_build.

See the Python quickstart for sync and asyncio write/read examples and the Python value contract for BSON, datetime, UUID, and error behavior. Tagged releases publish compiler-free cp39-abi3 wheels for the supported platform matrix; repository checkouts can still be installed from source with Rust 1.85+. Independently spawned Python, Rust, and server processes can share a ready local data directory; read the multi-process contract before deploying that pattern.

Still just inspectable files

briskdb-data/
├── .briskdb-process.lock
├── .briskdb-startup.lock
├── manifest.sqlite
├── global-indexes/
│   └── global.sqlite
└── shards/
    ├── 0000.sqlite
    ├── 0001.sqlite
    ├── 0002.sqlite
    └── 0003.sqlite

The manifest versions routing, catalogs, migrations, generated-ID ownership, and integrity metadata. Application rows stay in ordinary SQLite files.

Where this is going

• MongoDB: complete the matcher, write, index, cursor, and aggregation semantics behind the native embedded command facade, then add a Rust Mongo listener and differential TinyMongo parity.
• More wire protocols: broader PostgreSQL client compatibility and a MySQL listener, all sharing the same engine behavior.
• Serverless storage: atomic snapshots, object-store adapters, and fenced single-writer operation beyond today's embedded warm-handler pattern.
• Future storage adapters: SQLite is the first backend, while the engine boundaries are being kept reusable for other durable backends.

Follow the roadmap or browse the open issues.

Follow the build

Star BriskDB if you want to follow any of these bets:

• a native MongoDB wire protocol with large-app TinyMongo parity;
• MySQL compatibility over the same protocol-neutral Rust engine;
• serverless snapshots and object-store-backed lifecycle;
• more storage backends without giving up the ordinary SQLite option; or
• honest benchmark and failure evidence as the alpha becomes a real release.

If you try it, an issue with your client, workload, or missing SQL shape is even more valuable than a star. Start with the alpha releases, then tell us what broke or what surprised you.

Honest alpha boundaries

• PostgreSQL has TLS and single-identity SCRAM-SHA-256 authentication, but no roles or authorization yet. The separated HTTP data and administration listeners remain loopback-only development surfaces.
• No general atomic transaction across multiple shard files.
• Global ordering/pagination and general aggregate pushdown are still limited.
• HTTP offers bounded row streaming but no retained cursor, continuation token, cross-shard snapshot, or new global ORDER BY/OFFSET/LIMIT semantics.
• The supported backup today is a stopped-server copy of the complete data directory after every server and embedder exits. Passive checkpoints now report shards, manifest, and global-index storage through both Rust and the administration API, but are not an online snapshot. The backup capability endpoint reports this limitation rather than copying live files; online/serverless snapshots are planned.
• Multi-process access is same-host/local-filesystem only. Schema, catalog, upgrade, and recovery work requires sole-process ownership.
• Pre-1.0 storage and public-library compatibility can change between releases.
• Python document commands currently require explicit _id values and only implement empty or exact-_id filters. Updates, aggregation, bulk writes, retained document cursors, and built secondary indexes are still planned.
• Ubuntu 24.04 x86-64 receives the full required Rust CI suite. Python wheels receive native build, audit, install, restart, corruption, and concurrency checks on Linux/macOS x86-64 and ARM64.
• Global-index operational metrics are available, but BriskDB still lacks the broader production suite for traces, slow-query logs, resource saturation, alert rules, and long-running capacity validation.

Go deeper

• Architecture
• OpenAPI v1 artifact
• HTTP listener separation
• Global uniqueness and value authority
• Global-index production gate
• Embedded Rust
• Embedded SQL
• Crate features and support tiers
• PostgreSQL quickstart
• Tested PostgreSQL clients
• Mongo compatibility parity contract
• BSON value and codec contract
• Document catalog, storage, and TinyMongo import
• Protocol-neutral document engine
• SQL compatibility
• Generated keys
• Storage format
• Sharing one data directory between processes
• Debian and systemd installation
• Pre-1.0 compatibility policy
• Contributing

BriskDB is available under the MIT License.