investigation.md

SpacetimeDB 2.0 benchmark

Here is the source video

Cool Demo but...huh?

I know that sqlite is famously faster than fopen.
This demo showcase was hilarious and cool, but something is off.
I can see how not having tcp between the DB and the app is fast, but over 10x faster than in process sqlite (which can run in memory?) I dont know...

I'd like to preface all this with saying I did this over an hour or so and just out of curiosity and I think spacetimedb is super cool and worth using for it's intended use case. This is simply an exercise in light investigation.

This is an interesting benchmark and I'm not sure these things matter but they're worth pointing out and seeing if they are the reason for such a massive difference:

The first thing is if we're benchmarking spacetime we do something completely different. https://github.com/clockworklabs/SpacetimeDB/blob/master/templates/keynote-2/src/demo.ts#L357

the spacetimedb metric uses a rust client and everything else uses the same in process nodejs script: spacetimedb: https://github.com/clockworklabs/SpacetimeDB/blob/master/templates/keynote-2/src/demo.ts#L318 everything else: https://github.com/clockworklabs/SpacetimeDB/blob/master/templates/keynote-2/src/demo.ts#L292

Here is the main: https://github.com/clockworklabs/SpacetimeDB/blob/master/templates/keynote-2/spacetimedb-rust-client/src/main.rs

Rust Binary Runtime

First of all seeding is handled from the rust binary here This is handled by the same rust client with a different param (seed) as below. It is set up wtih the name "local", which is a default built in alias/name in spacetimedb cli

10018-> % spacetime server list
WARNING: This command is UNSTABLE and subject to breaking changes.

 DEFAULT  HOSTNAME                   PROTOCOL  NICKNAME
     ***  maincloud.spacetimedb.com  https     maincloud
          127.0.0.1:3000             http      local

It is called with this code:

await sh('cargo', [
    'run',
    //"--quiet",
    "--manifest-path",
    "spacetimedb-rust-client/Cargo.toml",
    "--",
    "bench",
    //"--quiet",
    '--server',
    server2,
    "--module",
    moduleName,
    "--duration",
    `${seconds}s`,
    "--connections",
    String(concurrency),
    "--alpha",
    String(alpha),
    "--tps-write-path",
    "spacetimedb-tps.tmp.log",
  ]);

• Uses a concurrency of 10
• uses an alpha of .5
• Uses a moduleName of module-1
• Runs at a server at localhost:3000
• has a "tps write path" of a local log where it will log the transactions per second

So what does this binary do?

You can have a look, but it:

1. Creates a tokio runtime with 10 connections and one thread per connection
2. connects that many websocket event loops in tokio tasks
3. Runs a warmup which is the same as the below looping for a specified duration (supposedly to get the caches warmed up locally and on spacetimedb)
4. Runs the benchmark where max_inflight_reducers (16k) per thread are run with the "transfer" backend server command

Nodejs Binary Runtime

So what does the nodejs bench do? Well unless it spawns 10 sub processes we already know rust has an advantage running on multiple threads (barring anyother bottlenecks on the current machine). I have to assume they ran this on some huge cloud node.

1. for each system do the following (lets just focus on sqlite for now)
2. create the connector for the database (RpcConnector)
3. This provides call via httpCall
4. This sends over HTTP with JSON a request to sqlite...that isn't how you use sqlite when I use it!!! I thought we were comparing in process stuff...well that's one explanation.
5. That http request then is further processed into a drizzle orm request to then finally send to sqlite
6. sqlite is by default of the benchmark configuration running with WAL, an illegal value for synchronous which makes it default to fully synchronous to disk. It also has a 64k page cache which might be small but i have no idea.

Here is the runner code which calls this connector: https://github.com/clockworklabs/SpacetimeDB/blob/6fea15f745c2fe75cb54358c61042e9cca1ea366/templates/keynote-2/src/core/runner.ts#L52

So:

• instead of websockets (basically tcp) we're doing http
• with json serialization instead of sending directly over a socket
• in a single thread (well sqlite only supports one writer anyway!), although concurrently with 10 concurrent rpc clients.

Ok so sqlite only supports one writer (common turso!) so what about postgres?

Well yeah the calling process is still nodejs and therefore using one thread which has to:

1. go down through the nodejs runner
2. make a system call per http request which is per reducer call

whereas rust has to just send over a websocket with noserialization on 10 threads. I also wouldn't be surprised if tungstenite buffered up and sent things over a websocket efficiently (more efficiently then message oriented) (but i have no idea.)

Breakdown

• Rust has a warmup and NodeJS didnt
• SpacetimeDB is only getting about 10k per thread, still better.
• The node tps counter uses spacetime_num_txns_total counter, but rust just counted directly. The issue with this is while we do block on a response from spacetimedb, the final is implicitly extended until the last batch completes, which is max inflight reducers per thread. I might be missing something but that means spacetime can get in an extra 8k transactions per thread if it's right in the middle. That doesn't feel correct, but take it qualitatively
• SpacetimeDB operates in memory entirely and does not safely write to the WAL before returning (afaik). I think spacetime still achieves isolation by serializing reducers who touch the same data.

So yeah, I'm not sure what to make of this but to me it seems as cool as spacetimedb is it is a little misleading and not durable if the server crashes (but perhaps durable enough).

I have never used spacetimedb and am not super familiar with it, so I'd love for someone to just rip this apart and educate me.

Brandon