ShannonBase Rapid Engine: Parallel Load and Async Column Read Technical Overview

Background

During initial data loading, AP analytical scenarios often require importing massive datasets.
If the traditional serial loading approach is used, loading tens or even hundreds of millions of rows from InnoDB into Rapid can take an excessively long time, severely delaying the start of analytical tasks.

In addition, analytical workloads often operate on wide tables (tables with a large number of columns).
In a columnar storage layout, if a traditional serial parsing approach is used, the system must combine data from each column sequentially during query execution to produce the final results — a process that can be time-consuming.
To address this, the Rapid engine introduces coroutine-based parallel parsing, accelerating column data parsing and assembly to significantly reduce query rendering time for wide tables.

Feature Introduction

To address these bottlenecks, two key system variables were introduced:

rapid_parallel_load_max — Parallel Load Row Threshold

During secondary_load, if the number of rows exceeds this value, the system automatically enables multi-threaded/coroutine-based parallel loading.

Fully utilize multi-core CPUs to parse and write data concurrently, drastically reducing total load time.

rapid_async_column_threshold — Async Column Read Threshold

During row scans, if the number of columns exceeds this value, the column parsing tasks are split into batches and executed concurrently in a coroutine pool.

Reduce per-row latency in wide-table scans and increase query throughput.

Performance Benefits

• Bulk Loading
  When the row count exceeds rapid_parallel_load_max, parallel loading reduces the loading time for tens of millions of rows from minutes to seconds, achieving a 3–5× speedup.
• Wide Table Scans
  When the column count exceeds rapid_async_column_threshold, batch-based coroutine column parsing reduces per-row scan latency by over 40%.
• CPU Utilization
  Parallel and coroutine-based execution keeps multi-core CPUs fully utilized during load and scan phases, reducing idle time.

Core Implementation

System Variable Control

• Registered via MySQL plugin variable mechanism, adjustable at runtime (some variables can only be changed before data is loaded).
• Validation checks ensure no active load/scan tasks are running before changes, preserving consistency.

Parallel Loading (rapid_parallel_load_max)

• During secondary_load, when row count exceeds the threshold, data is partitioned and imported into Rapid’s in-memory columnar store in parallel.

Coroutine-based Column Parsing (rapid_async_column_threshold)

• In DataTable::next_async(), columns are batched according to max_batch_sz and processed concurrently in a Boost.Asio coroutine pool.
• shared_promise is used to aggregate batch results and detect errors early.

Adaptive Batch Size

• SimpleRatioAdjuster dynamically adjusts batch size to balance coroutine scheduling overhead and CPU cache efficiency.