YDB v23.1 Release Webinar Recap

Eight YDB engineers gathered to share what they’ve developed for the recent YDB v23.1 release. The features they covered belong to two main categories: functional and performance. Let’s start with the former.

Initial scan for Change Data Capture (CDC)

Change Data Capture is the mechanism that allows to create a feed of changes to data stored inside a database and consume it elsewhere. In YDB it is implemented on top of YDB topics, which is a first-class-citizen representing a persistent queue similar to Apache Kafka®. Normally, CDC only sends changes that occur after it has been configured, but there are also use cases where you need to have all pre-existing data. Here’s where the new feature of v23.1 comes in: initial scan for CDC. With this option, before sending the changes, CDC first sends the current state at the moment of creation in the same format as the changes that will follow. This feature allows users to avoid manual workarounds that were needed before to achieve the same result.

Audit log improvements

The second major feature covered was the audit log, which is often perceived as table stakes by any security team. It is an invaluable resource for auditors to identify unauthorized activities, manage system-to-system interactions, and support incident investigations. In this release, logging of changes to YDB schema ​objects was added: databases, directories, tables, and topics. Additionally, it logs changes in the number of partitions, backup and restore operations, ​as well as modifications to access and more.

Performance enhancements

Automatic actor system pools configuration

Actor system is a fundamental C++ parallel computing framework on top of which YDB is built. Actors are lightweight entities that talk with each other by message passing, both locally or via a network. YDB nodes execute the actor’s code in several thread pools. Thread pool configuration was quite complex, but now it has an automatic mode with dynamic thread pool size allocation based on the system load and available CPU cores.

Improved data transfer formats between execution stages

YDB’s query execution consists of multiple stages, and we’ve reviewed all data passing between them and switched to the most efficient computation-friendly formats where possible. Additionally, we got rid of excessive data copying and timers along the way. These changes gave up to 30% CPU resource savings on release for some of our production workloads.

Computation graph pattern cache

If we zoom into the actual computation of query result, there are three main steps:

1. Query compilation​
2. Building computation graph pattern​
3. Execution of computation graph

In the OLTP scenario, the first two stages can easily be more expensive than the third. The apparent solution is to cache them. YDB had a query compilation cache for a while now, and in version 23.1 there is a new cache for the second stage — computation graph patterns. On a cache hit such a pattern is cloned, enriched with temporary values, and then becomes ready for execution.​

Secondary indices improvements

Secondary indices are another critical feature for database management system performance. In this YDB release, we’ve introduced improvements for two scenarios involving secondary indices.

YDB secondary indices have names and, at the time of writing, need to be explicitly used in a query by name. Thus in a production environment, it might be beneficial to change an index under a specific name, for example, to introduce more columns covered by the index. To accommodate this scenario, YDB now supports atomic secondary index replacement.

YDB allows for two types of queries: data queries for OLTP workloads and scan queries for OLAP workloads. Initially, only data queries were able to use secondary indices, and now scan queries can do so as well. However, this feature is still not production ready and not enabled by default in v23.1, but you can still experiment with it.

Improved predicate pushdown for table reads

On the input-output level, databases always try to do their best to read from disk the smallest possible amount of data necessary to produce the requested query result. In this release, we eliminated yet another case that was doing an excessive full table scan when a query used OR to read multiple table ranges by the primary key.

What’s next?

If you want to learn more about the features mentioned above, watch the video version of this webinar (there are also time codes in the description) or follow the links from the release notes. Also, the slides are published on GitHub.

If you’re running an older version of YDB, we recommend upgrading at your earliest convenience by downloading the latest release and following the upgrade instructions.