GitHub - superfly/litevfs: LiteFS VFS SQLite extension for serverless environments

2 min read Original article ↗

LiteVFS - LiteFS VFS implementation for serverless environments (WIP)

LiteVFS is a Virtual Filesystem extension for SQLite that uses LiteFS Cloud as a backing store.

SQLite CLI

To test with SQLite CLI:

  1. Build the extension:
  1. Provide LITEFS_CLOUD_TOKEN env variable and load the extension
$ LITEFS_CLOUD_TOKEN=<your token> sqlite3
sqlite> .load target/release/liblitevfs.so
  1. Open the database
sqlite> .open file:db1?vfs=litevfs

That's it. It should work now. The database is stored under tmp in a random directory.

To enable debug logging, run sqlite3 binary like this:

The following environment variable are handled by LiteVFS:

  • LITEFS_CLOUD_TOKEN - LiteFS Cloud token (mandatory)
  • LITEFS_CLOUD_CLUSTER - LiteFS Cloud cluster (optional for cluster-scoped tokens, mandatory otherwise)
  • LITEFS_CLOUD_HOST - LiteFS Cloud host (optional, defaults to https://litefs.fly.io)
  • LITEVFS_CACHE_DIR - cache directory for databases (optional, random directory under /tmp if not specified)
  • LITEVFS_LOG_FILE - log into the given file instead of stderr

The same shared library can be loaded from any language using their SQLite bindings.

Modifying the database through LiteVFS

In order to modify a database, the LiteVFS instance must host a write lease to the database. A write lease can be obtained via a pragma statement:

sqlite> pragma litevfs_acquire_lease;
sqlite> <updates here>
sqlite> pragma litevfs_release_lease;

Only one LiteVFS instance can hold a write lease for speficic database at a time.

Limitations

  • Databases with journal_mode=wal cannot be modified via LiteVFS (but can be read)
  • Databases with auto-vacuum cannon be opened via LiteVFS at all
  • VACUUM is not supported

Building LiteVFS for browsers

The build process uses Emscripten target, thus, Emscripten SDK needs to be installed and configured on the system. Refer to Emscripted docs on how to do this. Alternatively, devenv.nix file in this repo includes all the required dependencies.

To build simply do:

The command will build LiteVFS with Emscripten, download SQLite3 sources, build it with Emscripten and link it with LiteVFS. At this point you should have target/sqlite3-wasm/sqlite3.{js,wasm} files.

Note that since LiteVFS uses synchronous Emscripten's FETCH API, SQLite3 can only be used from a Worker thread, not from the main browser UI thread.