Measured, and reproducible.
sqlflow consumed 100,000,000 messages from Kafka in about 95 seconds on a laptop: 1,057,360 a second, DuckDB SQL in the middle, a sink at the end. That run peaks at 303 MiB of container memory, and no run in the table below exceeds 344 MiB, whatever the batch size or handler. Under a steady stream the working set, the memory the process itself holds, stays flat for an hour. Every number on this page comes from one script, and the script is linked below.
- Apple M1 Pro, 10 cores, 32 GB
- Docker 20.10.13
- DuckDB v1.5.2
- Go 1.25.5
- cp-kafka 7.3.2, one partition
- sqlflow v1.1.0, commit 1839a5a
- host load 4 to 8 during the runs, one run at 12, not an idle machine
1.06M msgs/sec
100,000,000 messages in about 95 seconds, StructuredBatch, batch 5000
303 MiB
Container peak on that run; no cell in the table above 344 MiB
25 to 29 MiB
Working set over 60 minutes at 1,000 msgs/sec
Throughput
Batch size sets the rate. The handler sets the ceiling.
10,000,000 JSON messages per run, three, medians. Consumed from Kafka, aggregated with SQL in DuckDB, written to the sink. StructuredBatch declares the schema up front. InferredMemBatch infers it per batch.
| Handler | batch_size | Throughput | Peak memory | Working set |
|---|---|---|---|---|
| StructuredBatch | 500 | 240,510/s | 335 MiB | 246 MiB |
| StructuredBatch | 2000 | 745,423/s | 306 MiB | 217 MiB |
| StructuredBatch | 5000 | 1,103,849/s | 298 MiB | 211 MiB |
| InferredMemBatch | 500 | 126,579/s | 327 MiB | 241 MiB |
| InferredMemBatch | 2000 | 227,953/s | 322 MiB | 236 MiB |
| InferredMemBatch | 5000 | 278,182/s | 344 MiB | 260 MiB |
Memory is flat under a steady stream
A ten times larger batch buys four and a half times the throughput. It doesn't buy it with memory: peak falls from 335 MiB to 298. Over an hour at a steady rate the working set held between 25 to 29 MiB. Draining a backlog costs more, and the section below says how much.
Declaring your schema is worth 4x
At batch 5000, StructuredBatch moves 1,103,849/s against InferredMemBatch at 278,182/s. If you know your schema, say so in the config.
Peak memory is everything the container is charged for, including page cache and the Go runtime's lazily freed pages. It's the number to provision for. Working set is anonymous memory, comparable to RSS. It's the number that reflects the engine. A run lasts ten seconds at the fastest cell and eighty at the slowest. Run-to-run spread at the top end is about 12%: the headline cell's three runs measured 1,039,579, 1,103,849 and 1,176,472, and the table shows the middle one. Every cell is the median of three runs, taken with host load 4 to 8 during the runs, one run at 12, not an idle machine. An idle machine reads a few percent higher.
Until 2026-09-10 every run on this page was 300,000 messages, which the headline cell consumed in a third of a second. A rate measured over a third of a second is a startup measurement, and it moved ten percent run to run. The page said 927,000/s on that basis: the Go rewrite the day it landed, commit 337541e, on an idle machine. The shipped v1.0.6 ran 706,000/s on that cell, because correct offset commits (#154) and durable window state (#158) cost about a fifth of the throughput at batch 5000, and two leak fixes (#243, #247) gave it back. None of those figures compare to the table above, which is a different build and a run size thirty three times larger.
Memory under load
Flat under a stream. Flat under a backlog.
The runs above last ten seconds to eighty. Two longer runs answer the question they cannot: an hour of steady stream, and a hundred million message backlog.
A steady stream for an hour
One pipeline, StructuredBatch at batch 5000. A paced 1,000 msgs/sec for 60 minutes: 3.6 million messages, 0 errors. The working set, sampled once a minute, held between 25 to 29 MiB for the whole hour. Size the container once.
A backlog, drained as fast as it will go
A topic already holding millions of messages, consumed unpaced. The rate goes up, because startup is amortized. Memory used to go up with it, because the consumer read ahead without a bound. It no longer does. This is the morning after an outage, and the hundred million row is the headline run.
| Handler | Backlog | Throughput | Container peak | Unbounded |
|---|---|---|---|---|
| StructuredBatch | 3,000,000 | 1,177,636/s | 285 MiB | 1,037 MiB |
| StructuredBatch | 10,000,000 | 1,006,210/s | 293 MiB | 2,860 MiB |
| StructuredBatch | 100,000,000 | 1,057,360/s | 303 MiB | not run |
| InferredMemBatch | 3,000,000 | 274,611/s | 339 MiB | 1,534 MiB |
The last column is the same build with the read-ahead set back to the hundred fetches sqlflow used to hold, measured the same day on the same machine. Peak memory rises 18 MiB between a three million message replay and a hundred million one. Before, it rose 1,823 MiB between three million and ten, and the hundred million row has no unbounded figure because it would have needed about 28 GB on an 8 GB VM. What changed is pull request 252: the depth of the queue between the consumer and the pipeline is now a setting, fetch.prefetch, and it defaults to two fetches. Bounding it cost no throughput. Every depth measured beat the unbounded one, by 2.4% to 8.0%.
Before the rewrite
Six to nine times, measured the same day.
sqlflow 1.0 replaced the Python runtime with Go. The two were run back to back in the identical setup: one docker network against the same broker, the same StructuredBatch config to a noop sink, 300,000 messages, a fresh topic and consumer group per run, both built native for arm64. The Python runtime is retired; every release since 1.0 is Go.
| batch_size | Python | Go | Speedup | Python memory | Go memory |
|---|---|---|---|---|---|
| 500 | 38,379/s | 307,083/s | 8.0x | 181 MiB | 172 MiB |
| 2000 | 95,334/s | 583,749/s | 6.1x | 180 MiB | 160 MiB |
| 5000 | 101,517/s | 946,223/s | 9.3x | 180 MiB | 169 MiB |
The gap widens with batch size
Python plateaus around 100,000 messages a second past batch 2000. Go keeps scaling. That's why the multiple is largest at the top end, not constant across the table.
Memory is not the difference
Working set is comparable between the two, between 160 and 181 MiB either way. The footprint belongs to DuckDB and Arrow, which both runtimes embedded. The rewrite bought throughput, not memory.
The Go figures in this table are the day-one build on 300,000 message runs, quoted as measured that day. They do not compare to the table at the top of the page, which is a different build and a different run size. A controlled comparison only means something if both halves come from the same sitting, and the Python engine is retired, so this one cannot be re-run.
A network-bound sink
When the wire is the bottleneck, batch size is the only knob.
The same engine, with the SQL inserting into MotherDuck over the internet instead of a local table. One million rows loaded and verified server side: exactly plus one million, zero errors.
| batch_size | Throughput | Bottleneck |
|---|---|---|
| 1,000 | ~7k/s | network round-trips |
| 5,000 | ~18k/s | network round-trips |
| 25,000 | ~38k/s | network round-trips |
Throughput scales almost linearly with batch size. CPU sits around 70% idle throughout. The bottleneck is the wire, not the engine.
Reproduce it
One script, one make target.
The script that produced every number on this page is in the repository. It builds sqlflow for linux, runs it on the same docker network as the broker, publishes the topic, consumes it under a fresh consumer group, and prints throughput and both memory figures at the end.
The target wraps scripts/benchmark-container.sh. Read it before you run it: the headline run needs about 20 GB on the broker’s disk for the hundred million message topic, and takes about an hour to publish a hundred million messages, then about 95 seconds to consume them. Every SQLFLOW_ variable in your environment is forwarded into the container, so a config templated on one can be swept from outside the script.
Run it in-network, not through the NAT.
Docker Desktop's host to container port forwarding caps Kafka fetches at roughly 10 to 15 MB/s. It understates throughput by about ten times. make benchmark-container exists so you measure the engine, not the NAT. Numbers from make benchmark in host mode are not comparable to the tables above.
Method
How the benchmarks were run.
The table publishes one topic of 10,000,000 messages and consumes all of it in every run under a fresh consumer group, so each run starts at offset zero and ends at the same offset. Every cell is the median of three, medians. The headline run publishes its own topic of 100,000,000. Reusing a topic while publishing more into it is what once made the same configuration read anywhere between 165k and 305k msgs/sec, depending on how many benchmarks had run before it. Consuming a fixed topic from the start is not that, and the backlog section above is the read-ahead effect measured on purpose.
Memory is sampled every 100 ms from memory.current and the anon line of memory.stat in the container's cgroup. Kernels 5.19 and later use memory.peak directly. Sampling is accurate for this workload because batch memory is a sustained plateau rather than a spike.
The pipeline under measurement is JSON to Arrow, zero-copy, into DuckDB over ADBC, aggregated with SQL, then written to the sink, with manual offset commits after each sink flush.
sqlflow v1.1.0, commit 1839a5a, DuckDB v1.5.2, measured 2026-09-10, host load 4 to 8 during the runs, one run at 12, not an idle machine. The previous figures were 300,000 message runs at commit 337541e, measured 2026-09-01 on an idle machine, and at commit 46a1bd1 on 2026-09-09.