YCSB performance series: YDB, CockroachDB, and YugabyteDB

It’s a challenge to implement a distributed database with strong consistency ensuring high speed and scalability. YDB excels in these aspects, and our customers can attest to this through their own experiences. Unfortunately, we have never presented any performance numbers to a wider audience. We recognize the value of this information and we are preparing more benchmark results to share.

YDB is a distributed SQL database. While there are many complex benchmarks like TPC-C that cover distributed transactions, we believe that good performance in distributed transactions is, to some extent, derived from basic key-value performance. For this reason, in this post, we show the results of the popular YCSB benchmark, with CockroachDB and YugabyteDB — trusted and well-known distributed SQL databases — serving as reference points. YDB outperforms them in many YCSB workloads.

Below is a detailed description of our setup and the reasoning behind it. To spark your interest, we provide a brief summary of the results for a 3-node cluster with a 2-billion-row dataset (2 TiB of data). The 3-node cluster has 1.5 TiB of RAM, and the total cache size is just 240 GiB, meaning that you can’t fit all the data into RAM. Here we exclude CockroachDB results for workloads A, B and F, with a full rationale provided further down in the post. The graph above shows the throughput in kilo operations per second (higher is better), while 99% latency is below 50 ms. In the results section, we will explore the details.

TL;DR feel free to skip to the results section.

YCSB introduction

YCSB is a popular key-value benchmark. We start with simpler key-value loads because achieving fast distributed transactions is unlikely if you cannot handle regular key-value loads efficiently. Another reason is that the key-value load ratio is typically high in many real-world scenarios.

Almost all databases compete in the YCSB arena and use it as a baseline. Database developers have fun by mapping themselves on it. That is why the official YCSB repository supports more than 20 databases and has countless forks, which further extend this number. Even many NewSQL databases use YCSB as a baseline measurement despite the fact that their nature is not solely about raw key-value accesses. For instance, in 2019, CockroachDB published a post titled “Yugabyte vs. CockroachDB: Unpacking Competitive Benchmark Claims.” In that study, CockroachDB significantly outperformed YugabyteDB. However, in May 2020, YugabyteDB prepared a counter-study, split into two posts (1, 2), with the revealing title “Bringing Truth to Competitive Benchmark Claims — YugabyteDB vs CockroachDB.” Then, in 2021, ScyllaDB showcased its comparison with CockroachDB, despite the two databases belonging to different classes. With their sense of humor, they titled it “An Apples and Oranges Comparison.” Later, in March, ScyllaDB published benchmarking best practices, and, as expected, YCSB was included in the list of “trusted open-source frameworks.” We simply cannot ignore this trend and want to participate in the YCSB competition.

Why did we choose CockroachDB and YugabyteDB? Both are distributed SQL databases. To a large extent, the modern history of distributed SQL began with Google Spanner. However, Google Spanner had an interesting peculiarity: it required specialized hardware, including atomic and GPS clocks. CockroachDB was created by ex-Google employees with the goal of achieving Spanner-like performance on commodity hardware. YugabyteDB was founded by ex-Facebook employees who had worked on Cassandra and HBase. Both CockroachDB and YugabyteDB are included in various lists of the top distributed SQL databases.

Competition helps all participating parties to improve: when you win, you push others to enhance their offerings, and when you lose, you identify weak areas to strengthen. That is why we want to compete with the best. At YDB, we believe that we have valuable insights to share with others, and we understand there are lessons for us to learn as well. We aim to create a collaborative competition so that everyone can benefit from it. We have shared all results and issues discovered with CockroachDB and YugabyteDB prior to publishing this post. We hope that our findings will help these systems become even better. Also, we plan to prepare a guide with instructions on how to reproduce our results.

The key concept of YCSB is workload. It defines a key distribution and a ratio of read/update/insert/scan operations. The length of the data row is fixed at 1 KiB by default. Here is a list of default core YCSB workloads with a short description:

· A (update heavy workload): 50% reads and 50% updates

· B (read mostly workload): 95% reads and 5% updates

· C (read only)

· D (read latest workload): 95% reads, 5% inserts

· F (read-modify-write): 50% reads and 50% read-update operations

· E (short ranges): 95% scans and 5% inserts.

All workloads except D have Zipfian distribution. The Zipfian distribution is a type of statistical distribution where a few items occur very frequently, and many others occur rarely, similar to the Pareto principle where 20% of the causes are responsible for 80% of the effects. Workload D uses the latest distribution.

Test setup

Our performance stand is a bare metal cluster with 8 machines. Each machine:

· 128 cores: 2 32-cores Intel Xeon Gold 6338 CPU @ 2.00GHz with hyper-threading turned on

· 4xNVMe Intel-SSDPE2KE032T8

· 512 GB RAM

· 50 Gb network

· Transparent hugepages turned on

· Ubuntu 20.04.3 LTS

We run all essential performance benchmarks on bare metal. This approach helps to achieve excellent reproducibility of results, which is crucial for precise optimization evaluation. Additionally, there are other important practical reasons:

1. Scalability is a key characteristic of modern distributed databases. It’s easy to be fast on a small cluster with just a few nodes. Achieving speed on a cluster consisting of tens or hundreds of nodes is a more challenging task. We have YDB installations with thousands of servers. While we cannot use thousands of servers in our test setup, we can evaluate scaling with eight 128-core servers, which is significant computational power. Also using 128-core machines is good to test vertical scalability.

2. YDB supports different topologies (replication models). Most common are:

· mirror-3-dc: we write 3 replicas similarly to CockroachDB and YugabyteDB

· block 4+2: we use erasure coding to store the data on the disks. This mode is recommended for clusters within a single availability zone. With block 4+2, you need half the disk space compared to mirror-3-dc. For block 4+2, you need at least eight servers.

Unfortunately, neither CockroachDB nor YugabyteDB support block 4+2, while this feature might be advantageous for single DC setups. We compare YDB block 4+2 setup against YDB mirror-3-dc setup.

We test the following versions:

· YDB trunk (main branch as of March 24), which is at this moment is nearly equal to YDB 23–1 (created around a month earlier) with only a few features set to be merged soon.

· CockroachDB 22.2.7

· YugabyteDB 2.17.1.0-b439

YDB installation consists of storage (also known as a static node) and compute (also known as a dynamic node or dynnode) daemons. On each server, we start a single YDB storage and four dynnodes, each within a 32-core taskset. We set the shared cache size to 20 GB for each dynnode (recall that each machine hosts 4 dynnodes). The trunk version of YDB supports the auto-configuration of its subcomponents like the actor system and gRPC. This enhancement will be backported to the 23–1 version shortly. Nonetheless, we have included these subcomponents in our provided configurations: the static node configuration and the dynamic node configuration.

As far as we know, CockroachDB does not separate its compute from storage. We believe this is a drawback because it prevents the use of diskless nodes for compute power. Moreover, as we understand, the number of CockroachDB instances that you can run on a single node is limited to the number of disks. On a multicore server, you must run multiple instances to scale, but consider the case when you have just one or two disks. Fortunately, we have four disks and run four CockroachDB instances, each with its own disk and each within a 32-core taskset.

We encountered trouble when running many YCSB clients: CockroachDB started to fail requests and print errors. We were advised here to lower gc.ttlseconds to 600 seconds. This change reduces the MVCC “window” and may interfere with manual backups, but without it, we cannot achieve a real high load. Another configuration we set is cache size (20 GB) and SQL memory size (30 GB), each per CockroachDB instance. See the discussion about setup and the issues here.

In their hardware requirements, YugabyteDB states that they have tested their system on machines ranging from 2 to 64 cores. Nevertheless, they claim that “YugabyteDB performance improves with more aggregate CPU in the cluster. You can achieve this by using larger nodes or adding more nodes to a cluster”. We were unable to utilize all 128 cores when running a single tserver; we collected the data and YugabyteDB confirmed that they found multiple bottlenecks. We failed to run multiple tservers per machine due to this issue. As a result, we used three machines running both master and tserver and five machines running only tserver, i.e. only single tserver per machine.

To test YDB and Yugabyte, we use a fork of the original YCSB implementation, which includes the YDB and YugabyteDB CQL drivers. Please note that we merged YugabyteDB drivers from their own YCSB fork. As we ran multiple YCSB instances in parallel we applied this patch to ensure that all instances used the same seed and had the same zipfian distribution. You can find the instructions for running YCSB on YDB here and for running YCSB on YugabyteDB here.

The YCSB paper describes its abstract interface for database drivers in terms of the following operations:

· read()— read a single record from the database, and return either a specified set of fields or all fields.

· insert()— insert a single record into the database.

· update()— update a single record in the database, adding or replacing the specified fields

· delete()— delete a single record in the database

· scan()— execute a range scan, reading a specified number of records starting at a given record key

This description leaves it up to the implementation to use either UPSERT or INSERT in insert() and either UPDATE or UPSERT in update(). In particular, here is a quote from the original paper, which describes read-modify-write workload F and compares it to update heavy workload A: “We also ran a “read-modify-write” workload, which reflects the frequently used pattern of reading a record, modifying it, and then writing the changes back to the database. This workload is similar to workload A (50/50 read/write ratio) except that the updates are “read-modify-write” rather than blind writes”.

This choice is performance-critical, as UPSERT is a blind write that is much faster. By default, YDB uses INSERT in insert() and UPSERT in update(). We find this combination reasonable based on some high-performance applications built on top of YDB. When properly used, this combination can be consistent and provide much better throughput/latency in non-uniform distributions such as Zipfian or hot spot. The Yugabyte CQL driver uses UPSERT in both cases.

To measure CockroachDB, we use their own YCSB implementation. It uses INSERT and UPDATE, and we haven’t found a way to change it to the INSERT/UPSERT or UPSERT/UPSERT combination. The combination of INSERT/UPDATE incurs a significant performance penalty. Especially it affects read-modify-write workload F because it makes it read-read-modify-write.

Overall, the following workloads use exactly the same underlying operations:

· Workload C: the same for all DBs

· Workloads A, B, and F: the same in YDB and Yugabyte

· Workloads D and E: the same in YDB and CockroachDB

Comparing Cockroach’s A, B, and F workloads with those of others is somewhat like comparing apples to oranges. We provide these numbers just to demonstrate the performance impact of operation choices made by users. YDB and YugabyteDB show similar results when switched to the INSERT/UPDATE ‘backend’. We will present these results in a separate post, while here we focus on the fastest and most scalable solution that application developers can use.

Please note that we follow the recommended order of loading and running workloads provided by YCSB. We load the initial YCSB data, then run workloads A, B, C, F, and D. After that, we load the data again (because workload D changes the dataset size) and run workload E. This allows us to compare workload C results on databases with many row modifications done in workloads A and B. It is crucial because read performance on “clean” and “dirty” states can differ dramatically.

Our goal is to achieve the highest throughput while keeping the latency below 50 ms. Since the cluster is so large, a single YCSB instance cannot generate an appropriate rate of requests, so we run multiple YCSB instances on different machines. This setup can be used for all workloads except D and E. Due to their specific features, we can start only one YCSB instance to run these workloads. This means that if we increase the number of machines in the cluster, it’s impossible to increase the number of clients to generate a higher load. Therefore, we measure D and E in a 3-machine setup and exclude in an 8-machine setup.

Results

We start from 300M-row dataset, which is approximately 300–400 GiB of data. The graph shows throughput measured in kilo operations per second and 99% latency of the operations. As you can see, YDB significantly outperforms its counterparts in the read-only workload C, which is the same in all YCSB implementations. We surpass Yugabyte in workloads B and F, and lose in workload A (recall all have the same implementation). Additionally, we outperform Yugabyte in read latest workload D, even though they have a head start due to UPSERT vs. INSERT. We outdo CockroachDB in workload D, while reasonably losing in workload E with short ranges.

A 300M-row dataset is not too much for a cluster with 1.5 TiB of RAM. Even with NVMe, reading from disk versus cache can penalize read throughput. So, we increased our dataset from 300M to 2B rows (about 2–3 TiB of data). At the beginning of the post, we show the results without CockroachDB’s A, B and F (please recall that they’re UPDATEs, not UPSERTs like in YugabyteDB and YDB). Now the full results:

Originally we wanted to measure only read-only workload C, which is mostly affected by the insufficiency of RAM for caching. We ran workloads A and B just to have row modifications in DB. Surprisingly we found that YugabyteDB had a very strange and significant performance degradation in write-intensive workloads A and F. We used the exact same configuration, and double-checked with the YugabyteDB team, as in previous measurements. There are many IDLE resources; the database simply doesn’t utilize them. We have collected all the data requested by the YugabyteDB team to help them to triage this issue. As of the time of publication, the issue is still under investigation. You can refer to this github issue for updates.

Due to this degradation, YDB now outperforms in all workloads except workload E. We found a performance issue in our scan implementation. The fix should be available soon and we plan to remeasure this workload.

Now we continue with a horizontal scaling test. We increased the number of machines from 3 to 8. Additionally, we run YDB block 4+2. This time the dataset is again 300M rows. Please remember that it’s impossible to increase the load generated by workloads D and E and we exclude them in this setup.

The results are similar, except that now Yugabyte slightly outperforms YDB in workload F. YDB mirror-3-DC wins in workload C by a wide margin, demonstrating fantastically low latency. YDB block 4+2 is slightly slower than YDB mirror-3-DC because it uses more CPU to perform erasure coding (recall that it consumes half the disk space compared to mirror-3-dc); that’s a tradeoff between disk space and CPU. However, it still significantly outperforms CockroachDB and Yugabyte in workload C, as well as in a few other workloads.

To gauge scalability, we took the following approach: we calculated requests per core based on the results for a 3-node cluster and a 300M row data load. We then multiplied this by the number of cores in an 8-node cluster to estimate throughput in the case of ideal linear scaling. Yugabyte displayed suboptimal scaling in workload C: both YDB and CockroachDB demonstrated superior scalability, achieving 0.81 and 0.79 respectively of the calculated ideal scaling, while YugabyteDB only managed 0.65.

Conclusion

All three databases exhibited decent performance, but none dominated across all workloads and dataset sizes. There were scenarios in which either YugabyteDB or CockroachDB outperformed YDB, and others where YDB won by a significant margin.

We recommend choosing YDB for read-mostly or read-only workloads, as it excels in these scenarios according to our YCSB results. At the same time, you shouldn’t worry if your workload changes at some point: YDB still maintains performance comparable to YugabyteDB and CockroachDB.

If you have a single datacenter setup (single availability zone), you should seriously consider choosing YDB block 4+2. It offers excellent performance and requires half the disk space compared to mirror-3-dc. Even nowadays, SSD/NVMe disks can be quite expensive, and YDB block 4+2 can significantly reduce spending.

We believe that YugabyteDB and CockroachDB will benefit from having us as fellow competitors, just as we gain from competing with them. We would like to express our gratitude to the CockroachDB and YugabyteDB developers for their assistance with configuration and result analysis. A special note of appreciation goes to Ivan Prisyazhnyy, a ScyllaDB expert and the author of one of the performance posts we cited. He thoroughly reviewed our paper draft and provided insightful feedback.

Update: We have released the next post on performance, this time focusing on TPC-C. You can check it out here.