The Same Matching Engine Measured 130,000/s and 106/s

Bitsten is a cryptocurrency exchange. Its matching engine is the service that takes buy and sell orders for a trading pair, decides which of them trade against each other, and records the result. This post is about measuring that engine, and about how easy it is to publish a matching engine performance number that is true and useless at the same time.

One vendor reports millions of operations per second. Another production system reports a few hundred commands per second. Both can be telling the truth, because they are timing different things and calling both “throughput”. We ran into this inside a single codebase. Here are four measurements of the same engine, taken on the same afternoon on the same machine:

Top row to bottom row is a spread of roughly 1,200x. Publish only the first number and you have a marketing asset. Publish only the last and you have understated the engine by three orders of magnitude. Neither one tells an operator what to buy or a developer what to fix.

So every result in this post comes with a boundary statement. “Pure domain” is the deterministic trade calculation and nothing else. “Conductor” adds validation, ordering, and the durable state commit. “Confirmed pipeline” adds event publication with broker confirms. A fifth boundary, “client round trip”, would add the API gateway and the response path. We have not measured that one carefully yet, so it is not in the table.

The numbers only make sense if you know what the pieces are, so here is the path one order takes. There are two services and three pieces of infrastructure: RabbitMQ carries commands in and events out, Redis (or Valkey) holds the order books, and a service called the conductor sits between them. A second service, the conductor-outbox, moves committed events from Redis to RabbitMQ.

Trading pairs are assigned to partitions with a fixed function, pairId % partitionCount, in libs/shared-lib/src/matching/contracts.ts. Each partition has one RabbitMQ queue, matching:commands.pN, and one conductor process that consumes it. The queue is a quorum queue with single active consumer turned on, so even if two conductor processes are running for the same partition, only one of them receives messages. That is the entire concurrency control for a partition. There is no lock.

@RabbitRPC({
  exchange: MATCHING_COMMAND_EXCHANGE,
  routingKey: matchingCommandQueue(ASSIGNED_PARTITION),
  queue: matchingCommandQueue(ASSIGNED_PARTITION),
  queueOptions: {
    durable: true,
    arguments: {
      'x-queue-type': 'quorum',
      'x-single-active-consumer': true,
    },
  },
})
processCommand(payload: unknown): Promise<ProcessedCommandResult> {
  return this.service.processCommand(payload);
}

Inside the conductor, processCommand in conductor.service.ts chains every command onto a promise tail so command N+1 cannot start before command N has settled. Then processCommandSerially does the work, in this order:

1. Parse the command and check its partition field against the conductor’s own assigned partition. A command for the wrong partition throws.
2. Look up matching:{matching-pN}:processed-command:<commandId> in Redis. If it exists, this is a broker redelivery. Return the stored result, after a durability wait, without touching the book.
3. Open a Redis MULTI. Nothing is written to Redis until step 7.
4. For a place-order command, check accepted-order:<orderId>. If that key exists, the same order was already accepted under a different command ID (a client retry with a fresh ID). Count it as a duplicate order and skip it. Otherwise stage a SET for that marker.
5. Read candidate makers from the opposite side of the book. The book is a sorted set per side, keyed book:<pairId>:ask:rates and bid:rates, with price as the score and a zero-padded timestamp in the member string so ties come back in time order. The read is ZRANGEBYSCORE (or ZREVRANGEBYSCORE for bids) with LIMIT 0 64, followed by a pipelined HGETALL for each candidate’s order hash.
6. For each candidate, call calculateDeal in matching.domain.ts. It is pure decimal.js arithmetic: executed volume is the smaller of the two remaining volumes, rounded in the book’s favour, and it throws if maker and taker volume changes ever diverge. Each match stages an HSET or a ZREM+DEL for the maker, and an XADD of a deal.executed.v1 event onto matching:{matching-pN}:events. If the taker is left with volume and it is a limit order, stage a ZADD so it rests, plus an order.rested.v1 event.
7. Stage SET processed-command:<commandId> with the result, then EXEC. Every key in the transaction carries the same {matching-pN} hash tag, so this stays a single-slot transaction if the datastore is ever clustered.
8. Call WAITAOF 1 1 2000. The command only counts as done when the primary and one replica have both fsynced the append-only file. If that does not happen inside two seconds, the command fails and the broker will redeliver it.
9. Return the result. The RabbitMQ RPC layer acknowledges the message.

async execTransaction(transaction: MatchingTransaction): Promise<void> {
  const results = await transaction.exec();
  if (!results) throw new Error('Matching transaction returned no results');
  for (const [index, [error]] of results.entries()) {
    if (error) throw new Error(`Matching transaction command ${index} failed: ${error.message}`);
  }
  await this.ensureDurability();   // WAITAOF localAofFsyncs replicaAofFsyncs timeoutMs
}

That is the conductor boundary. Everything the exchange considers authoritative has happened by the end of step 8. But nobody downstream knows yet: the ledger, the market data feed, and the user’s WebSocket all learn about trades from RabbitMQ, and the events are still sitting in a Redis stream.

Moving them is the outbox’s job. conductor-outbox.service.ts is a separate process, one per partition, and it runs a loop:

1. Take a lease. SET matching:{matching-pN}:outbox-lease <consumerId> PX 10000 NX on a control connection, renewed every 3.3 seconds with a compare-and-expire Lua script. If the lease is held by someone else, sleep a second and try again. This is what keeps one publisher per partition, which is what keeps events in order.
2. Run XAUTOCLAIM to pick up entries a dead consumer left pending for more than 30 seconds.
3. If the consumer group still has pending entries, wait. Do not read new ones ahead of unconfirmed old ones.
4. Otherwise XREADGROUP GROUP matching-outbox-v1 <consumerId> COUNT 100 BLOCK 1000 STREAMS matching:{matching-pN}:events >.
5. Decode each entry. An entry whose envelope does not parse goes to events:dead-letter in one MULTI with its XACK and XDEL, and the loop moves on.
6. Publish the batch to RabbitMQ with publisher confirms, then XACK and XDEL the batch. Publish failures retry with exponential backoff and jitter, capped at 30 seconds, forever. They are never dead-lettered, because a broker outage is not a bad message.

Each event has a deterministic ID derived from the command ID, its ordinal within the command, and its type, so a republished event is recognisable as a duplicate. It goes out as a persistent, mandatory message with the event ID as messageId and the stream entry ID in an x-partition-offset header. Consumers keep an inbox keyed by event ID. Delivery is at-least-once and nobody pretends otherwise.

With the pieces named, the four rows of the table are: the arithmetic in step 6; the Redis transaction shape from step 7 in isolation; steps 1 to 9; and steps 1 to 9 plus the outbox loop draining to zero.

Matching cost depends entirely on what the orders look like, so the workload matters more than the hardware. scripts/conductor-performance.js publishes N commands to matching:commands.p0, alternating a limit bid and a limit ask on pair 1 at price 100, volume 1. Every bid rests (one order.rested.v1 event). Every ask matches it (two order.filled.v1 events and one deal.executed.v1). Two commands, four events, and the book is empty again.

We chose this because it is the worst case for a single-writer engine. Every command touches the same pair, so there is no relief from partition parallelism. Every second command does a real match with a state change on both sides. Nothing rests at the end to make the numbers look better. The opposite kind of benchmark posts limit orders at unique prices and never matches anything, and what it measures is your hash map.

It also means these results do not generalise upwards. Real traffic has many pairs, and the design runs pairs in different partitions in parallel, so mixed traffic should do better on aggregate throughput. We have not run that test, so we are not claiming it.

The script waits for the conductor’s /metrics counter to reach the target, fails the run if matching_command_failures_total moved, and with BENCH_WAIT_OUTBOX=1 it also waits for XLEN on the events stream to hit zero. At the end it reports the remaining book depth. Because every pair of commands fully matches, any depth other than zero means a command did not do what it should have, and you can see that without computing a digest.

• Publish the command mix and the fill-count distribution, not only “orders per second”.
• Describe the book: number of markets, resting orders, occupied price levels, orders per level, and how concentrated the book is near the touch.
• Warm up before timing. matching-domain-performance.js runs 10,000 throwaway iterations first because the first few thousand calls through a JIT look nothing like the steady state.
• Use deterministic IDs and timestamps so two implementations get byte-identical input.
• Run long enough to cross garbage collection and AOF rewrite cycles. A three-second sample measures the gaps between them.
• Check final state after every run. The section on invariants below says what we check.

The gap between the conductor number and the full pipeline number is the one to understand, and it is plain once you look at the outbox code as it was at the time. The original publishAndAcknowledge handled one stream entry at a time: publish, await the confirm, XACK, XDEL, next entry. That preserves partition order. It also means one RabbitMQ round trip per event, serialised.

We measured that ceiling directly. scripts/outbox-drain-performance.js waits for a backlog on the stream, then times how long the outbox takes to drain it, polling XLEN every 10ms. With a 10,000-event backlog and nothing else running, one worker confirmed 517 events per second.

The workload produces two events per command, so 517 events/s is about 258 commands/s of headroom on its own. Then the conductor and outbox started competing for the same Redis instance and the same RabbitMQ broker, and the combined figure dropped to 106.

You can see the contention in the broker timings that conductor-performance.js reports. Publishing and confirming a 5,000-command burst took 296 to 303ms when the outbox was idle. In the concurrent run the same burst took 1,088ms, 3.6x slower, while the producer’s own publish() call latency stayed low:

p50   0.029 ms
p95   0.342 ms
p99   2.590 ms
max  12.296 ms

That distribution is what a healthy producer in front of a saturated consumer looks like. Publishing is cheap; the work behind it is not. It is also why measuring only the client-visible submit latency would have told us the system was fine.

The fix follows from the measurement: keep sending in stream order on one channel, but do not wait for each confirm before sending the next. publishBatchAndAcknowledge in outbox-delivery.ts now takes the whole XREADGROUP batch (up to 100 entries), fires every publish() without awaiting, waits for all of them with Promise.allSettled, and only then advances the stream as one MULTI:

const confirmations = deliveries.map(({ entry, event }) =>
  publishEvent(rabbit, entry, event),
);
const confirmed = await Promise.allSettled(confirmations);
if (confirmed.some((r) => r.status === 'rejected' || !r.value)) {
  throw new Error('RabbitMQ did not confirm publication batch');
}

const transaction = dataClient.multi();
transaction.xack(streamKey, consumerGroup, ...streamIds);
transaction.xdel(streamKey, ...streamIds);
transaction.hdel(attemptsKey, ...eventIds);
await transaction.exec();

Ordering is preserved because the sends still leave in stream order on a single channel. If any confirm fails, the whole batch is retried, which can republish events that were already confirmed. That is fine, because consumers already dedup on event ID. allSettled rather than all matters here: on a failure you still want every outstanding publish to settle before you retry or close the channel, or confirm callbacks fire against a channel you already tore down.

This change is in the code and has not been through a fresh full-pipeline run yet. I am not quoting a number for it until it has one.

Dragonfly is a multithreaded Redis-compatible datastore, and multithreaded is the headline. We migrated to it, then benchmarked it against plain Redis 7 on the transaction shape the conductor issues. scripts/redis-command-performance.js runs MULTI; HSET; ZADD; XADD; EXEC in a loop, sequentially per worker, with N workers on N connections. Sequential per worker on purpose, because that is what one ordered partition does.

Dragonfly came in 30 to 31% slower at both connection counts. That is not a criticism of Dragonfly. A transaction-heavy, small-payload workload where every key in a transaction shares one hash tag is close to the worst case for a design whose advantage is spreading work across cores. The conductor builds its keys that way on purpose, so the shape that does not shard is the shape we have.

The end-to-end conductor runs against Dragonfly came in between 164 and 344 commands/s, with zero failures and the expected event count. That spread is too wide to say anything about the datastore. At that scale, local RabbitMQ and Docker Desktop scheduling dominate. End-to-end numbers are the wrong instrument for comparing one component.

The architectural bet is that matching scales horizontally through independent cells: one writer, one storage group, one outbox per partition. scripts/redis-multicell-performance.js tests that with the same synthetic transaction. It spreads 16 logical partitions across the Redis URLs you give it, one connection per partition, and with BENCH_WAITAOF=1 it issues WAITAOF 1 1 2000 after every transaction, which is what the conductor does.

Both cells shared one Docker Desktop CPU and disk budget. Adding a second cell added processes. It did not add CPU, disk bandwidth, or a failure domain, which are the three things that would have made it faster. The result does not show that partitioning fails. It shows that the test did not test partitioning.

This is an easy way for a scaling benchmark to lie, and we did it by accident. The claim (“independent failure domains give independent capacity”) and the experiment have to match. If the experiment does not provide independent resources, all you have measured is contention you introduced yourself.

The real test needs cells on separate hosts, or at least enforced CPU and storage budgets. It should route independent market traces through the production code path and show at least 1.5x sustainable throughput going from one cell to two, with p99 latency reported per partition so an idle market cannot hide a hot one, zero ownership conflicts, zero sequence gaps, and matching final-state digests. Until that run exists, the multi-cell design is a hypothesis.

Memory-only state, an asynchronous replica, local append-only persistence, and local-plus-replica fsync are four different products that get described with the same word. In the table above, the only difference between 25,621 tx/s and 3,254 tx/s is the durability policy.

That is what a real recovery point objective costs on this hardware. The conductor issues WAITAOF 1 1 after EXEC and before it acknowledges the command to RabbitMQ, so a confirmed command has been fsynced on both the primary and a replica. If the datastore cannot satisfy that within MATCHING_DURABILITY_TIMEOUT_MS (default 2,000), the command fails instead of succeeding with weaker semantics than the caller was promised.

We had changed this once without noticing. Replacing KeyDB’s every-second AOF with Dragonfly’s five-minute snapshot moved the worst-case unreplicated loss window from one second to five minutes. That arrived as part of a datastore migration, not as a durability decision. It is now a named policy, MATCHING_DURABILITY=aof-replicated or memory, and the process refuses to boot in the replicated mode against a datastore that cannot honour it.

• Put the exact policy next to every throughput number. “Persistent” is not a policy.
• Measure AOF acknowledgement latency on its own and in combination with the command.
• Record durability timeouts and replica availability during the run. A fast run with an absent replica is measuring memory mode.
• If the service fails closed when durability is unavailable, exercise that path under load, not just in a unit test.
• Disk settings, filesystem, host cache, storage class, and virtualisation all move this number. A laptop’s memory-backed filesystem cannot support a production recovery-point claim.

A benchmark that only checks the clock will happily reward dropped work, duplicate execution, broken FIFO, negative remainders, and unconfirmed events. Every run here reports its invariants alongside the throughput: zero command failures, zero outbox retries, zero dead letters, the exact expected event count from the conductor’s matching_events_created_total counter, and zero remaining book depth.

Some of the invariants live in the engine rather than the harness. calculateDeal throws MatchingInvariantError if a volume goes negative or if the maker and taker volume changes disagree after rounding, and execTransaction throws on any per-command error inside the EXEC result. A benchmark run that trips either of those shows up as a command failure and the script aborts.

• Compute a deterministic digest of the final books and the ordered event stream, and compare it across implementations.
• Assert executed buy quantity equals executed sell quantity, no order fills beyond its accepted quantity, every live order appears in exactly one side and price level, empty levels are removed, and FIFO holds within a level.
• Count accepted, rejected, duplicate, rested, cancelled, and filled commands. Duplicates caught by dedup are a success metric.
• For replay tests, run the trace uninterrupted and with injected crashes, then compare. Kill before EXEC; after EXEC but before the broker ack; after publication but before XACK.
• A correct idempotent design reaches the same authoritative state in all three cases. At-least-once delivery may repeat an event ID. Prove the consumer applies it once.

Zero reported failures means nothing if the harness never checked. Say what you asserted, not just that nothing went wrong.

Partway through, we reset the matching keyspace in Redis while the outbox was still running. conductor-performance.js does this at startup to clear stale state (SCAN MATCH matching:*, then DEL), and it took the outbox’s consumer group with it.

The worker retried the resulting NOGROUP error forever, and its /metrics endpoint started returning HTTP 500 because XPENDING failed too. So the component was stuck and unable to say that it was stuck. In production that is a silent event-delivery outage with a broken health signal on top.

Two changes came out of it. The outbox loop now catches NOGROUP, calls ensureConsumerGroup (an XGROUP CREATE ... MKSTREAM that tolerates BUSYGROUP), and continues. And pendingCount returns zero while the group is missing instead of throwing, so the metrics endpoint stays up. We discarded the affected run and reran it after rebuilding.

A benchmark is fault injection with a stopwatch attached. Sustained load plus a hostile hand on the infrastructure finds recovery gaps that unit tests cannot, because unit tests do not delete things out from under a running process. Budget time in every performance run to break something on purpose.

A single end-to-end timer says the request is slow. It does not say which capacity to buy or which code to change. Instrument the stages separately: queue wait, validation, candidate lookup, match calculation, EXEC, WAITAOF, event stream lag, broker confirm, consumer application. The conductor already exposes matching_last_command_duration_ms per command; the next step is splitting that into its parts.

Correlate every command and event with partition, pair, command ID, event ID, and stream entry ID. The event envelope carries all of those. Keep detailed logging out of the hottest loop, or you will be profiling your logger.

Then change one thing at a time and re-run the same trace: batch candidate reads, cut redundant serialisation, reuse connections, widen the publisher-confirm window, isolate disk. Compare percentiles and invariants, not just the headline.

The five scripts and the run log they produced are published as a public gist, copied from the backend repository with one local Docker credential replaced. Four of them run against any Redis-compatible datastore or any RabbitMQ; the pure-domain one imports the private calculateDeal and is there to show the harness rather than to run standalone. Here is the environment, in full, so the numbers above can be placed:

Host        Apple M3 Max, 14 logical CPUs, 36 GiB RAM, macOS arm64
Engine      Docker Desktop 28.1.1
Runtime     Node.js 20.20.2 (containers)
Broker      RabbitMQ 3.9 Management
Datastore   Redis 7 Alpine / Dragonfly 1.39.0 / Valkey + AOF
Workload    single pair, alternating equal-price bid/ask, 2 events per command
Consumers   durable benchmark sink queues bound; no downstream business consumers

node scripts/matching-domain-performance.js 100000    # pure domain: calculateDeal in a loop
node scripts/conductor-performance.js 1000            # conductor: publish N commands, wait on /metrics
BENCH_WAIT_OUTBOX=1 node scripts/conductor-performance.js 5000   # plus wait for XLEN == 0
node scripts/outbox-drain-performance.js              # time the outbox draining an existing backlog

BENCH_REDIS_URL=redis://localhost:6379 pnpm perf:redis -- 20000 16
BENCH_REDIS_URLS=redis://127.0.0.1:6580,redis://127.0.0.1:6581 \
  BENCH_WAITAOF=1 pnpm perf:redis:cells -- 10000 16

And what this does not measure, stated plainly: no network latency, no downstream consumer or database work, one hot pair, no resource limits, no multi-pair parallelism, no client round trip. Docker Desktop loopback differs from deployment conditions in every one of those.

The operational conclusion was more useful than any single figure. Confirmed event delivery was the bottleneck. The matching arithmetic had three orders of magnitude of headroom. Durability costs roughly 8x and is worth paying. Independent cells are the intended scale unit and a shared-host test cannot demonstrate that. The task we had been about to start, “make matching parallel”, was the wrong one. “Reduce round trips and open a confirm window” was the right one, and only measuring boundary by boundary made that visible.

Pair every throughput figure with latency percentiles, queue growth, error counts, and invariant results at the same offered load.