ArrowMetal: Apache Arrow compute on the Apple silicon GPU via Metal

Apache Arrow compute on the Apple silicon GPU, inside DataFusion, Polars and DuckDB.

The CPU and the GPU of an Apple silicon Mac share one memory: the GPU scans an Arrow column where it already is, with no copy across a bus, and while it runs the CPU is free for the rest of your application. So every figure here gives the CPU time of a call beside its wall time.

Apple M4 Max. The engine's own time ÷ its time with ArrowMetal's default; for Polars, the faster of its in-memory and streaming engines. Each engine's second row is its lowest speed-up (best of the runs) among the cases its default takes. On the first run after 500 ms of idle, against the engine's own first run, DataFusion's group-bys are 1.24x to 2.17x and four of DuckDB's thirteen rewritten pairs are behind, down to 0.38x.

DataFusion. A physical optimizer rule for DataFusion 55.1; the SQL is unchanged. A Rust crate used by crates.io; it loads libArrowMetalC.dylib (install):

datafusion-arrowmetal = "0.4.1"   # beside datafusion = "=55.1.0"

Full sorts 6.9x to 28.8x faster than DataFusion alone (250,000 to 50,000,000 rows), ten group-by series 2.31x to 4.27x; 13,632 query pairs, 0 mismatches. Left to DataFusion: top-k 0.14x to 0.41x, filters 0.31x to 0.79x. rule.report() gives each node, TAKEN or LEFT, and why. DataFusion in full

Polars. MetalEngine runs the subtrees measured ahead (full sorts, inner, left and anti joins, unique, group-bys judged by their estimated number of groups); Polars runs the rest. 1.52x to 11.54x on all 75 of 220 case-size pairs the default took; 15,376 generated cases, 0 different. Left to Polars: whole-frame aggregates 0.09x to 0.24x, filters 0.14x to 0.42x. Polars in full

pip install "arrowmetal[polars]"   # macOS 14 or later on Apple silicon; pins the tested Polars range
python -m arrowmetal.bench         # your Mac, 30 s or less, every answer checked against pyarrow

import polars as pl, arrowmetal as am
lf = pl.scan_parquet("orders.parquet")
engine = am.MetalEngine()                             # the measured defaults
df = lf.sort("amount", descending=True).collect(engine=engine)  # the frame lf.collect() returns
print(engine.last_report)                             # Metal or Polars, per node, and why

DuckDB. An optimizer extension; the SQL is unchanged. Built from a checkout with ./duckdb-extension/build_rewrite.sh (DuckDB 1.5.6), then LOADed, or opened by am.duckdb_connect(). Rewritten aggregates 1.08x to 5.32x faster than DuckDB's own operators; 33,376 generated queries, 0 different. Left to DuckDB: a single sum 0.32x, VARCHAR keys 0.22x. DuckDB in full

339 benchmark rows, 77 where the CPU is ahead, each with its cause (matrix, to improve); 307 of Arrow v25's 307 compute function names, 283 on the GPU (list).

Details

ArrowMetal runs Apache Arrow compute on the Apple silicon GPU: Arrow arrays that live in Metal shared memory, GPU kernels that keep Arrow's semantics (validity bitmaps, packed booleans, the C Data and C Device Data Interfaces), and one C ABI reachable from Swift, Python, Rust, Go, TypeScript, R and C.

Contents: The engines in full (DataFusion · Polars · DuckDB) · The one-row picture · Measure it on your Mac · Release · The pitch · What was measured · Why this exists · Benchmarks · From Python · Bindings · Quick start (Swift) · Reading Arrow files · What is implemented · Design notes · Citing · License

The engines in full

ArrowMetal runs inside three query engines. Each one plans the query as it always does; ArrowMetal takes the parts of the plan measured faster on the Apple GPU, hands back what the engine would have returned, and leaves every other node to the engine. Every figure below is from a run recorded in this repository, named beside it, on an Apple M4 Max (16 CPU cores, 64 GB); each speed-up is the engine's own time divided by the time with ArrowMetal, with the CPU time of the call next to its wall time.

DataFusion in full

A physical optimizer rule for DataFusion 55.1, the Rust crate datafusion-arrowmetal in datafusion/, on crates.io. It loads libArrowMetalC.dylib, built from the repository root by swift build -c release --product ArrowMetalC (DataFusion):

# Cargo.toml
[dependencies]
datafusion = { version = "=55.1.0", default-features = false, features = ["sql"] }
datafusion-arrowmetal = "0.4.1"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

use std::sync::Arc;
use datafusion::{datasource::MemTable, prelude::*};
use datafusion_arrowmetal::{session_context, ArrowMetalConfig, ArrowMetalRule};

let rule = ArrowMetalRule::new(ArrowMetalConfig::default());
let ctx = session_context(SessionConfig::new(), rule.clone());
let t = MemTable::try_new(batch.schema(), vec![vec![batch]])?;
ctx.register_table("t", Arc::new(t))?;
ctx.sql("SELECT * FROM t ORDER BY amount DESC").await?.show().await?;   // the SQL as written
println!("{}", rule.report());                                          // TAKEN / LEFT, and why

Registered on a SessionContext, the rule replaces DataFusion's full sorts and the hash aggregates of measured shapes with MetalExec, which runs them on the GPU and hands DataFusion the RecordBatches it expects. The default takes full sorts (ORDER BY without LIMIT) from 250,000 rows, over DataFusion's own Parquet reader too, and, over a MemTable of 50,000,000 rows or more, count(*) and DISTINCT over two int32 keys and MIN/MAX of an integer column over one int64 key or two integer keys, at the numbers of groups where each was measured ahead, estimated from a sample of the keys when the query runs (DataFusion).

• Full sorts of 250,000 to 50,000,000 rows: 6.9x to 28.8x faster than DataFusion alone; at 50,000,000 rows, 96.2 to 125.8 CPU-ms against DataFusion's 6,104.1 to 8,310.5 (datafusion/results/datafusion_sort_warm_2026-09-29.csv). DataFusion's own sort of 50,000,000 rows spends 6,104.1 CPU-ms in 1,581.22 ms, about four of the sixteen cores on average, which is part of why the figure is large; Polars sorts the same three columns by the same int64 key in 380.46 ms on this machine, and ArrowMetal under Polars in 53.03 ms, 7.17x (Benchmarks/results/polars_engine_bench_2026-10-02.csv).
• The ten group-by series the default runs on the GPU, in both table layouts (20 cases), at 50,000,000 rows: 2.31x to 4.27x warm; on the first run after 500 ms of idle against DataFusion's first run after the same idle, 1.24x to 2.17x, and after 5 s, 1.11x to 1.68x (datafusion/results/datafusion_groupby_refit_check_2026-10-02.csv and the current-crate rows of datafusion/results/datafusion_groupby_default_import_2026-10-01.csv).
• 13,632 query pairs, each run with and without the rule and the answers compared: 0 mismatches (DataFusion).

To improve, left to DataFusion by the default (DataFusion alone ÷ with the rule): top-k (ORDER BY … LIMIT 100) 0.14x to 0.41x at 250,000 to 50,000,000 rows (datafusion/results/datafusion_sort_warm_2026-09-29.csv), and filters 0.31x to 0.79x at 10,000,000 and 50,000,000 rows (datafusion/results/datafusion_filter_2026-10-01.csv); the other aggregate shapes are under To improve.

Polars in full

A MetalEngine for lf.collect(engine=…):

pip install "arrowmetal[polars]"      # pins the tested Polars range

import polars as pl, arrowmetal as am
lf = pl.scan_parquet("orders.parquet")
engine = am.MetalEngine()                                         # the measured defaults
df = lf.sort("amount", descending=True).collect(engine=engine)    # the frame lf.collect() returns
print(engine.last_report)                                         # Metal or Polars, per node, and why

Polars optimises the lazy plan as it always does. MetalEngine replaces the subtrees it translates with ArrowMetal plans that run on the GPU and return Polars DataFrames; Polars' in-memory engine runs everything else. The default takes the shapes measured ahead: full sorts from 1,000,000 rows, inner, left and anti joins, unique, String sorts and unique from 5,000,000 rows, a sort over a Parquet scan, and group-bys judged by their estimated number of groups (Polars).

• Over 220 benchmarked case-size pairs (2,000,000 and 50,000,000 rows), the default took a subtree in 75, and in every one of them it is 1.52x to 11.54x faster than the faster Polars engine, in-memory or streaming (Benchmarks/results/polars_engine_bench_2026-10-02.csv).
• 15,376 generated cases against Polars: 15,096 identical, 33 within the float-summation bound, 0 different, and 247 where Polars' plan had nothing to run (Benchmarks/results/engine_conformance_2026-10-02.csv).

To improve, left to Polars by the default (Polars ÷ MetalEngine, 250,000 to 50,000,000 rows, Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv): whole-frame aggregates 0.09x to 0.24x, row-wise filters and projections 0.14x to 0.42x, a semi join against a 1,000-row table 0.21x to 0.47x, and the top 100 by a nullable Float64 key 0.75x to 1.05x, which keeps top-k with Polars. Every translatable shape, with its measured result, is in Polars engine capabilities.

DuckDB in full

An optimizer extension; the SQL is unchanged. Built once with ./duckdb-extension/build_rewrite.sh (against DuckDB 1.5.6), and loaded into a connection opened with allow_unsigned_extensions, or by am.duckdb_connect() from Python:

LOAD 'duckdb-extension/build/arrowmetal_rewrite.duckdb_extension';
SELECT region, sum(amount), count(*), max(amount) FROM sales GROUP BY region;   -- as written
SELECT * FROM arrowmetal_rewrites();     -- each decision, with its reason
SET arrowmetal_rewrite = 'off';          -- plain DuckDB; 'auto' is the default

The extension puts ARROWMETAL_AGGREGATE where DuckDB's HASH_GROUP_BY or UNGROUPED_AGGREGATE would have been; the scan, the pushed-down filters and everything above the aggregate stay DuckDB's. The default (auto) takes sum, avg, min, max and count over integer, DATE and TIMESTAMP columns, grouped by one column or none, over a table scan or read_parquet: a fused group-by of an estimated 10,000 groups or more from 10,000,000 rows, the other measured classes from 50,000,000 (DuckDB).

• Every query the default rewrote in the benchmark is 1.08x to 5.32x faster than DuckDB's own operators (Benchmarks/results/duckdb_rewrite_2026-10-04-ac.csv).
• 33,376 generated queries with the rewrite off and forced: 21,844 rewritten and identical, 11,532 left to DuckDB, 0 different (Benchmarks/results/engine_conformance_2026-10-04.csv).

To improve: on the first run after 500 ms of idle, four of the thirteen rewritten query-size pairs are behind DuckDB's first run after the same idle (same file), all at 50,000,000 rows: sum, max, avg (BIGINT), 22.38 ms against 8.55 ms (0.38x); sum, count, min, max (BIGINT, 10% NULL), 29.39 against 23.72 ms (0.81x); sum, min, max, avg (INTEGER), 12.54 against 10.34 ms (0.82x); and 1,000 INTEGER keys with five aggregates, 36.68 against 32.61 ms (0.89x). Left to DuckDB by the default, each with a query behind at 50,000,000 rows in the same file: a single sum (sum(INTEGER), 0.32x), few groups with at most two aggregates (1,000 INTEGER keys, sum, 0.59x) and VARCHAR keys (1,000 long VARCHAR keys, sum, count, 0.22x).

The one-row picture

Apple silicon has one physical memory shared by the CPU and the GPU, so an Arrow buffer placed in a Metal shared buffer is at once a valid CPU Arrow buffer and a valid GPU buffer: a column is used where it already is, with no copy across a bus in either direction. While a kernel runs, the CPU is free for the rest of the application, and every table in this repository shows the CPU time of each call next to its wall time.

One row, the same in-process data in every column:

From Benchmarks/results/full_matrix_2026-09-07-parallel.csv, Apple M4 Max (16 CPU cores, 64 GB), best of up to five calls after one warm-up, release build; the 339-row matrix that row comes from, including the 77 rows where the CPU idiom is ahead, is in The full matrix. The DuckDB row is from Benchmarks/results/duckdb_matrix_2026-09-12.csv, measured on 2026-09-12 with the matrix's generators and protocol over a native DuckDB table, the data loaded before timing (DuckDB).

Measure it on your Mac

pip install arrowmetal          # macOS 14 or later on Apple silicon; pyarrow is the only dependency
python -m arrowmetal.bench      # 30 seconds or less
python -m arrowmetal.bench --parquet data.parquet   # read, sum, filter, group-by on your own file

The second line generates 10,000,000 rows, runs sum, filter, sort and group-by sum through pyarrow (and Polars when it is installed) and through ArrowMetal, and checks every ArrowMetal answer against pyarrow's. It prints one table for your Mac, wall and CPU milliseconds per call, with a block ready to paste into a benchmark result issue or the Discord #benchmarks channel; nothing is sent.

import pyarrow as pa, arrowmetal as am
amount = am.MetalArray.from_arrow(pa.array([10.0, 20.0, 30.0, 40.0]))
region = am.MetalArray.from_arrow(pa.array([1, 2, 1, 2], pa.int32()))
print(am.group_by([region]).sum(amount).to_arrow())     # [40, 60]: GPU group-by sum, one row per region
print(amount.filter_where(">", 15).to_arrow())          # [20, 30, 40]: GPU filter, back as a pyarrow array

Release

Listed on Apache Arrow's Powered By page. Site and docs: https://arrowmetal.org · Python: pip install arrowmetal · Rust: arrowmetal = "0.4.0" · DataFusion: datafusion-arrowmetal = "0.4.1" · Release: v0.4.0

New in 0.4.0: datafusion-arrowmetal, a DataFusion optimizer rule on crates.io; the Polars engine's sort keys passed one to one with Polars' null placement and float order, and its group-by crossovers refitted; correctly rounded grouped Float64 sums and means; per-key sort options (ieee, total, nan_largest) in every binding; columns of many chunks imported without a concatenated copy; and UInt32 index arrays (CHANGELOG.md).

The pitch in one paragraph

Every array is Arrow layout in memory the GPU already shares, so there is nothing to upload. Gathers, group-by, sorts and string scans run on the GPU 3.8x to 24.2x faster than the fastest CPU idiom in the table below. That idiom is Polars' lazy engine or pyarrow's Acero, using a median of eleven of the sixteen cores and up to fifteen. While they run the CPU is free for the rest of the application: every table here reports CPU time per operation next to wall time, and the same rows cost 26 to 1,284 CPU-ms on the other side (the shift view apart, at 0.06) against 0.4 to 1.7 here. Chains of operations share one GPU round trip. The whole thing is reachable from Swift, Python, and any language with Arrow bindings through one C ABI. The full distribution over all 339 measured rows — including the 77 where the CPU idiom is ahead — is in The full matrix and Where ArrowMetal needs to improve.

What was measured

• 339 benchmark rows over 173 operations against four CPU libraries at their most parallel idiom — Polars' lazy engine, pyarrow's Acero, pandas and numpy on an idle M4 Max, with the cores each call used recorded beside it. 145 rows at or above 3x, 102 between 1x and 3x, 77 where the CPU idiom is ahead, 15 with no CPU equivalent; each of the 77 has its cause written down. The full matrix, Where ArrowMetal needs to improve
• 307 of Apache Arrow v25's 307 compute function names — 283 on the GPU, 17 on the host, 7 with a stated limitation. The table is generated from a registry the test suite executes against pyarrow.compute. Arrow functions, one by one
• 41,688 differential cases against pyarrow over 46 column types, 1,046 Swift tests against a CPU oracle and 7,188 Python cases (7,159 passed, 29 skipped), all run in release. Testing
• Seven languages on one C ABI — Swift, Python, C, Rust, Go, TypeScript and R, each binding with its own suite against that language's Arrow library. docs/README.md
• 19 findings in other projects — pyarrow, Apple Metal, the Arrow JS, R and Go libraries, and the Swift compiler — each with the evidence behind it and where its report stands. Upstream tracker

Why this exists

Apple silicon has one physical memory shared by CPU and GPU. An Arrow buffer placed in a MTLBuffer with storageModeShared is simultaneously a valid CPU Arrow buffer and a valid GPU buffer. No upload and no download: a page-aligned producer's buffers are used in place, and a buffer that is not page aligned is copied once on the way in. No Arrow project we surveyed took advantage of that:

ArrowMetal fills that hole: Arrow arrays whose buffers live in Metal shared memory, GPU kernels that honour Arrow semantics, and standard Arrow C interfaces in and out so it plugs into arrow-rs, pyarrow, DuckDB, Polars, arrow-swift or anything else that speaks the C Data Interface.

Benchmarks

Apple M4 Max (16 CPU cores), 50,000,000 rows (10,000,000 for the string row), best of up to five calls after a warm-up, release build. Full history and methodology in Benchmark method and Benchmarks/README.md. Every number here is from that one machine. If you have another Apple silicon Mac, the output of swift run -c release arrowmetal-bench 10000000 5, posted on Discord or in an issue, is new information.

Called from Python, same in-process data. Every CPU library is measured twice — its plain eager idiom, and the most parallel idiom it has for the same answer: polars-lazy through pl.LazyFrame on the in-memory or streaming engine, pyarrow-threaded through an Acero plan over 16 record batches (pandas' threaded paths, numexpr and numba, were not installed for this run; numpy's ufuncs are single-threaded). The baseline below is the fastest of all of them, named, with the cores that call used (its CPU-ms over its wall ms). Every row is taken from Benchmarks/results/full_matrix_2026-09-07-parallel.csv, the raw output behind The full matrix; the eager-only run of the same build is kept beside it as full_matrix_2026-09-07.csv. Wall time in milliseconds, with the CPU time each call consumed in parentheses. The last three rows are ones where the CPU idiom is ahead:

Swift, against all 16 CPU cores (tight typed loops over the same Arrow layout) and Accelerate. These are the Swift-level baselines of rounds 2, 3 and 7 in Benchmark method, measured 2026-09-06; the sort row predates the 2026-09-07 sort rewrite that the matrix above measures:

Takeaways, all against the fastest idiom of any CPU library. The rows at or above 3x are gathers, group-by, sorts and GPU string predicates: gathers (take, 24.2x), multi-key and high-cardinality group-by (lexsort 24.0x, sum by int32 key 3.8x at a thousand groups and 6.3x at a hundred thousand), sorts (argsort int64 7.7x, sort float64 4.1x) and GPU string predicates (contains 7.0x at 10M rows; the four GPU predicates measured at ten million rows — match_like on a pure prefix, contains, starts_with and match_substring_regex on a literal — run 1.6x to 7.0x against Polars lazy and Acero). The group-by family is 65 of 84 rows at or above 3x. A single pass that reads one column and writes one is memory bound on both sides: eleven to fifteen cores reach the same unified memory the GPU does.

Across all 339 rows the verdicts are 145 at or above 3x, 102 between 1x and 3x, 77 where the fastest CPU idiom is ahead, and 15 with no CPU equivalent. The 77 by measured cause, as Where ArrowMetal needs to improve groups them: 16 at a million rows and below, where the dispatch floor is the operation; 28 memory-bound single passes; 6 software binary64 transcendentals and calendar arithmetic; 2 regular expressions matched on the host; 4 where the CPU library returns a view and ArrowMetal materialises a column; 8 temporal extractions; 4 unique and value_counts rows against a threaded hash aggregation; 1 grouped moment; 4 string conversions with variable-length output; and 4 whole-query chains against a streaming engine. shift (lag 1) at 0.01x is the lowest ratio in the matrix outside the latency family. Every one of them is listed with its cause in Where ArrowMetal needs to improve. The crossover states the other side of the floor: from what row count the GPU path is ahead, per operation and per family, measured with a size sweep from a thousand rows to fifty million. Below the router's crossover in that document (the GPU kernel against a single-core CPU loop, not against a CPU library), seven operations (sum, min, max, compare, add/subtract/multiply, filter, group-by sum) take a byte-identical single-core CPU path on integer columns instead; ARROWMETAL_ROUTER=gpu turns that off (Design).

DuckDB on the same rows. The matrix's CPU libraries are Polars, pyarrow and pandas. DuckDB 1.5.5, measured on 2026-09-12 on the matrix's sort, group-by, filter and sum rows with the same data and protocol (DuckDB, Benchmarks/duckdb_matrix.py), is the fastest CPU engine on low-cardinality group-by and multi-key sort: against it, sum by int32 key at 1,000 groups and 50M rows is 1.8x rather than the 3.8x the matrix shows against Acero, and lexsort is 7.0x rather than 24.0x. From 100,000 groups upward, and on every sort, ArrowMetal stays 3x or more ahead of DuckDB, at 1 to 2 CPU-ms against DuckDB's 135 to 4,350. An optimizer extension also runs eligible aggregates of unchanged DuckDB SQL on the GPU with DuckDB's exact answers (DuckDB §4b). In Rust, a physical optimizer rule for DataFusion 55.1 runs the full ORDER BY sorts of unchanged SQL on the GPU with DataFusion's answers, 6.9x to 28.8x faster than DataFusion alone from 250,000 to 50M rows on an M4 Max (DataFusion).

From Python

pip install arrowmetal installs the published wheel (macOS 14 or later on Apple silicon). Two more ways, both from this checkout:

# From source (needs the Swift toolchain)
swift build -c release --product ArrowMetalC        # .build/release/libArrowMetalC.dylib
PYTHONPATH=python python -c "import arrowmetal as am; print(am.device_name())"

# From a wheel that carries the dylib (no Swift toolchain at install time)
pip install build && scripts/build_wheel.sh         # or python/build_wheel.sh, if the dylib is built
pip install python/dist/arrowmetal-0.4.0-*.whl      # macOS arm64 only, pyarrow comes with it
python -c "import arrowmetal as am; print(am.device_name())"

import pyarrow as pa, arrowmetal as am               # pyarrow is the only import dependency
col = am.array(pa.array([1, None, 3, 40]))
print(col.filter_where(">", 2).sum())              # 43, computed on the GPU
with am.batch():                                   # several kernels, one GPU round trip
    total = col.filter((col > 1) & (col < 40)).sum()

With the polars extra installed (pip install 'arrowmetal[polars]', or just pip install polars), a Polars Series crosses the same way:

import polars as pl
col = am.array(pl.Series([1, None, 3, 40]).to_arrow())
print(pl.from_arrow(col.filter_where(">", 2).to_arrow()))

A whole lazy plan can run through lf.collect(engine=am.MetalEngine()), which takes the parts it measured ahead of Polars and leaves the rest to Polars (Polars, tier 4).

from decimal import Decimal

# Group-by over arbitrary key columns — strings here, several columns if you pass several.
# A null key forms its own group, as in Arrow.
region = am.array(pa.array(["emea", "apac", "emea", None, "apac"]))
amount = am.array(pa.array([10.0, 20.0, 30.0, 40.0, 50.0]))
gb = am.group_by([region])
dict(zip(gb.keys()[0].to_pylist(), gb.sum(amount).to_arrow().to_pylist()))
# {'emea': 40.0, 'apac': 70.0, None: 40.0}

# Checked arithmetic raises exactly where the unchecked kernel wraps, naming the row.
big = am.array(pa.array([2**62, 2**62], type=pa.int64()))
(big + big).to_arrow()                             # wraps, which is Arrow's unchecked `add`
big.add_checked(big)                               # ArrowMetalError: add_checked: overflow at index 0

# Regex runs on the host behind the same API; a pattern that is really a literal
# routes back to the GPU kernel.
names = am.array(pa.array(["ArrowMetal", "polars", "pyarrow"]))
names.match_substring_regex("^p").to_arrow()       # [false, true, true]

# decimal128 arithmetic on the GPU; decimal32/64 widen into it and narrow back.
price = am.array(pa.array([Decimal("19.99"), Decimal("5.01")], type=pa.decimal128(10, 2)))
price.decimal_add(price).to_arrow()                # [39.98, 10.02]

See Python.

Bindings

Four language bindings over the same C ABI (include/arrowmetal.h) are in 0.1.0, in this repository, each exchanging columns through the Arrow C Data Interface and each with its own test suite:

C and C++ callers use the header directly. Any language with an Arrow binding can use the same header.

Quick start (Swift)

import ArrowMetal

let prices = try MetalArray<Float>([9.5, nil, 12.0, 3.25])       // nullable, lives in Metal shared memory
let mask   = try prices.compare(.gt, 5)                          // GPU, Arrow boolean bitmap out
let picked = try prices.filter(mask)                             // GPU stream compaction
print(try picked.sum(), try picked.max(), picked.nullCount)      // .float(21.5), 12.0, 0

// Multi-column batches behave like an Arrow RecordBatch.
let orders = try MetalRecordBatch(names: ["region", "amount"], columns: [.int32(region), .float32(amount)])
let hits = try orders.filter(try region.compare(.eq, 2).and(try amount.compare(.gt, 100)))
let sample = try hits.take(try MetalArray<Int32>([0, 5, 9]))
let window = try hits.slice(offset: 32, length: 1000)             // zero-copy view

Chains of operations can share one GPU round trip:

let total = try MetalContext.shared.batch {
    try amount.filter(try region.compare(.eq, 2).and(try amount.compare(.gt, 100))).sum()   // one command buffer
}

batchAsync is the same thing without the wait, so the CPU is free while the GPU scans. Return a pending array from the body (anything that reads a scalar on the CPU would sync inside it) and use the async accessors for scalars:

let ctx = MetalContext.shared
let hits = try await ctx.batchAsync {                          // records, commits, releases the thread
    try amount.filter(try region.compare(.eq, 2).and(try amount.compare(.gt, 100)))
}
print(hits.length)                                             // already resolved: no GPU round trip
let total = try await hits.sumAsync()                          // reduction, still without blocking

// Callback form, for code that is not async:
ctx.batchAsync({ try amount.filter(where: .gt, 100) }) { result in
    switch result { case .success(let hits): print(hits.length); case .failure(let e): print(e) }
}

Six end-to-end scenarios (analytics query, feature preparation, Float64 with NaN, C Data Interface interop, sliced windows, one batched command buffer) live in Sources/ArrowMetalExamples: swift run -c release arrowmetal-examples.

Interop with any Arrow implementation through the C Data Interface:

import CArrowABI

var schema = ArrowSchema(); var array = ArrowArray()
producer.export(into: &schema, &array)                            // arrow-rs, pyarrow, nanoarrow, arrow-swift ...
let imported = try importArrowArray(schema: &schema, array: &array)
// imported.zeroCopy == true when the producer's buffers were page aligned

var device = ArrowDeviceArray()
picked.exportArrowDeviceArray(into: &device)                      // device_type == ARROW_DEVICE_METAL

Requirements: macOS 14+ / iOS 17+, Swift 5.10+. Shaders compile at runtime so the Command Line Tools are enough to build and use it. Running the test suite needs Xcode (for XCTest): DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test.

Reading Arrow files

The Arrow IPC streaming and file formats are read and written directly, with no dependencies: a small FlatBuffers codec lives in Sources/ArrowMetal/IPC. Buffers land straight in Metal shared memory, so a file read is ready for the GPU with no further copy.

let batches = try ArrowIPCReader(url: url).readAll()          // .arrow or .arrows, memory mapped
try ArrowIPCWriter.write(batches, to: url)                     // file format, readable by pyarrow

let reader = try ArrowIPCReader(url: url)                      // random access via the file footer
print(reader.schema.names, reader.batchCount)
let first = try reader.batch(at: 0)

let stream = try ArrowIPCWriter.encode(batches, format: .stream)   // Data, for sockets or Flight

Int8 to UInt64, Float32/64, Bool, Utf8, LargeUtf8 and Binary are read into their MetalArray types, and dictionary-encoded columns round trip through DictionaryBatch messages. Temporal columns (date32/64, time32/64, timestamp, duration) are carried by their storage integer array while the logical type stays visible in reader.schema.

Both directions take every type this package can hold, nested children and all: decimals, float16, fixed_size_binary, the three interval units, null, list / large list / fixed-size list, struct, map, dense and sparse unions, run-end encoded columns and extension types. pyarrow reads each one back with the right type and the right values, and every file pyarrow writes for one of them reads back with pyarrow's values. The reader also decompresses LZ4_FRAME and ZSTD bodies (ZSTD through libzstd, as the Parquet reader does, with a clear error when it is not installed), and follows message order for dictionaries, so a stream may replace a dictionary part way through or extend it with a delta. The writer emits uncompressed, little-endian bodies of classic types only. The reader takes big-endian sources, the view types (materialised to the classic layouts) and the arrow.fixed_shape_tensor extension type, and refuses IPC tensor messages. Coverage summary says what each type is covered by.

What is implemented

307 of Apache Arrow v25's 307 compute function names — 283 entirely on the GPU, 17 on the host, 7 with a stated limitation, none missing. Every one of those names has a row in Arrow functions, one by one naming the Swift file behind it, the ArrowMetal call that reaches it and what it does differently, and that table is generated from a registry the test suite executes: python/tests/test_functions.py calls every runnable row and compares the answer to pyarrow.compute. Coverage summary is the same ground arranged by family, with the Arrow type matrix and the interop status.

• MetalArrowBuffer: page-aligned shared-memory buffers, zero-copy wrap of foreign page-aligned memory.
• MetalArray<T> for Int8/16/32/64, UInt8/16/32/64, Float16/32/64; MetalBooleanArray with packed bits; MetalStringArray (utf8, binary), MetalDecimalArray (decimal128/256), MetalSmallDecimalArray (decimal32/64), MetalTemporalArray (date32/64, time32/64, timestamp with timezone, duration), MetalIntervalArray (all three layouts), MetalFixedBinaryArray, MetalListArray, MetalStructArray, MetalMapArray, MetalUnionArray, dictionary, run-end-encoded and extension arrays.
• MetalRecordBatch: named equal-length columns with filter, take, slice, selecting, plus a GPU hash join over int32/int64 keys (Kernels/Join.swift).
• Element-wise: the six comparisons, wrapping and checked (overflow-raising) add/subtract/ multiply/divide/power/negate/abs/sqrt/the logarithms/the shifts, bit_wise_*, boolean and/or/not/xor/and_not and the Kleene forms, all ten Arrow round modes with ndigits, min/max element-wise, is_nan/is_finite/is_inf.
• Transcendentals: the twelve trigonometric and hyperbolic functions, their seven _checked twins, atan2, expm1, log1p, logb, hypot — float32 through Metal's library functions, float64 through a software binary64 implementation on the GPU; measured worst case 5 ulp (tan) on float64 and 4 on float32 — the per-function table is in Coverage summary.
• Aggregates: sum, min, max, mean, product, variance, stddev, quantile, median, mode, count_distinct, first/last, index, min_max, skew, kurtosis, tdigest.
• Grouped aggregates: all 24 hash_* names over arbitrary key columns — integers sparse or negative, floats, booleans, temporal values, utf8, binary, dictionary and decimal, several columns folded together — mapped to dense ids on the GPU by GroupByKeys, with GroupBy's dense-integer fast path kept underneath.
• Selection and ordering: filter (including a fused predicate form), take, slice, drop_null, scatter, inverse_permutation, single- and multi-key sort_indices / lexsort_indices, top_k / select_k_unstable (a GPU radix select for any k, with a per-threadgroup path at k ≤ 1024), partition_nth_indices, rank, dense_rank, row_number, rank_quantile, rank_normal, winsorize.
• Strings: length, equality, prefix/suffix/containment, count_substring, find_substring, murmur3 hash, dictionary_encode, the case and title transforms (ASCII and Unicode), padding and centring, trimming (ASCII and the Unicode whitespace class), slicing, replace-slice, repeat, reverse, joining, is_in/index_in, and the full ascii_is_* / utf8_is_* predicate set. The regex family, SQL LIKE, splitting, Unicode normalisation and the float/boolean casts run on the host behind the same API, with a GPU fast path for patterns that are really literals.
• Temporal: every extractor (year … nanosecond, iso_calendar, year_month_day, subsecond, week with all its options, us_week, us_year), ceil/floor/round_temporal, every *_between difference including the three interval-valued ones, add_interval, strftime/strptime, and the three timezone functions (assume_timezone, local_timestamp, is_dst) on the host, where the IANA database lives.
• Structural and conditional: if_else, case_when, choose, coalesce, replace_with_mask, fill_null, fill_null_forward/_backward, is_null/is_valid/true_unless_null, indices_nonzero, make_struct, struct_field, list_value_length/list_flatten/list_element/ list_slice/list_parent_indices, map_lookup, pivot_wider, run_end_encode/run_end_decode.
• Windows: pairwise_diff (and its checked form), the cumulative family (sum, prod, min, max, mean, and the checked sums and products), shift, and rolling sum/mean/min/max.
• libArrowMetalC: a C ABI over everything above, and a ctypes Python package that speaks the Arrow PyCapsule protocol.
• Float64 on the GPU even though Metal has no double: compare, min, max, filter, take and slice use an order-preserving map of the IEEE bit pattern; sum, add, subtract, multiply, divide, the segmented group-by sums and means and the whole transcendental family use a software IEEE-754 binary64 implementation on 64-bit integers. The arithmetic (+ − × ÷) is bit-exact against Swift's Double over millions of random and edge-case inputs, subnormals and NaN included; sum and the grouped sums use the same adder but reassociate over threadgroups, so they are checked to a tolerance. sqrt is correctly rounded; exp, ln, log10, log2 and power measure a worst case of 1 ulp against libm and are asserted within 2; the trigonometric and hyperbolic families are within 5.
• NaN: min/max skip NaN and return null if only NaN remains; sum propagates NaN; comparisons follow IEEE.
• C Data Interface import/export for every type above and for struct (+s) record batches, C Stream Interface import, C Device Data Interface import/export, MTLBuffer recovery from our own exports.
• ArrowIPCReader / ArrowIPCWriter: the Arrow IPC streaming and file formats, including a minimal FlatBuffers reader and builder, with no dependencies. Cross-checked against pyarrow in both directions.
• Parquet decoded on the GPU, structs, maps and lists nested to any depth included, with the stored Arrow schema applied and page-index and bloom-filter skipping (Parquet on the GPU).
• Delta Lake and Apache Iceberg tables (am.read_delta, am.read_iceberg): metadata on the CPU, data files through the GPU Parquet reader (Delta Lake and Iceberg).
• CSVReader / am.read_csv: CSV parsed and typed on the GPU with pyarrow's rules, tested against pyarrow.csv.read_csv (CSV on the GPU).
• JSONReader / am.read_json: newline-delimited JSON parsed on the GPU with pyarrow.json.read_json's types, field order and error texts (JSON on the GPU).
• A CPU reference implementation behind the kernels, used as the oracle in tests.

Design notes

• Runtime shader compilation. Kernels are MSL strings generated per element type and cached. No .metal files, no metal toolchain, no Xcode required to build.
• No atomics in reductions. Each threadgroup writes a partial; the host finalises a few thousand values. Deterministic results, works for 64-bit integers where Metal atomics are limited.
• Bitmaps as 32-bit words. Compare writes one word per thread; filter uses popcount + simdgroup prefix sums. Buffers are always padded so whole-word reads are in bounds.
• Page-aligned allocation. Every buffer ArrowMetal hands out can be re-wrapped with makeBuffer(bytesNoCopy:) by another Metal consumer.
• Lifetime discipline. Raw pointers are only valid while their owning object lives. Prefer the closure accessors (withValues, withTyped).

See ROADMAP.md for the project's scope and where its measured record is kept, and CONTRIBUTING.md to get involved.

Questions, and timings from Macs we have not measured: Discord or a GitHub issue, whichever you prefer.

Citing

If you use this project in research or commercial work:

ArrowMetal: Apache Arrow compute on Apple silicon GPUs via Metal.
2026. https://github.com/singhpratech/ArrowMetal

License

Apache License 2.0. Apache Arrow and Apache DataFusion are trademarks of the Apache Software Foundation; Polars and DuckDB and their logos are trademarks of their respective owners. This project is independent and not affiliated with or endorsed by the ASF, Polars, DuckDB or Apple.

Design

How ArrowMetal is put together and what limits it today.

Layers

Python / Rust / Go / C# / R / C++  ──(Arrow C Data Interface)──►  libArrowMetalC (C ABI, handles)
                                                                          │
Swift apps ────────────────────────────────────────────────────►  ArrowMetal (Swift API)
                                                                          │
                                       MetalContext ── pipeline cache ── BufferPool
                                                                          │
                                   MSL kernels generated per element type, runtime compiled
                                                                          │
                                       Metal shared-memory buffers (page aligned, unified memory)

• Buffers are the Arrow columnar layout verbatim: validity bitmap + values (+ offsets for variable-length types). The same bytes serve CPU consumers (C Data Interface) and GPU kernels (C Device Interface): copy-free out always, and copy-free in when the producer's buffers are page aligned — one copy otherwise, which the importer reports (ImportResult.zeroCopy).
• Copy-free import wraps a producer buffer with makeBuffer(bytesNoCopy:) after checking that every page of it is mapped and readable. The check asks the VM map for the regions that cover the buffer (mach_vm_region, one call per region), so its cost does not grow with the page count; see Chunked columns and the import for what it replaced.
• Kernels never assume a length multiple of anything: every buffer is padded to a page so trailing 32-bit bitmap words are readable; kernels bounds-check the last word/element.
• Nulls ride along as bitmaps. Element-wise ops share the input's validity buffer (zero-copy); binary ops AND the two bitmaps on the GPU; compaction repacks.

Where the time goes today (M4 Max, 50M rows)

Source: swift run -c release arrowmetal-bench, recorded 2026-09-06 on the Apple M4 Max (16 cores, 64 GB unified memory, Darwin 25.6.0) that every table on this page was measured on; the dated rounds are in Benchmark method.

Device peak on M4 Max is ~546 GB/s. Single-pass kernels sit at 55-70% of peak; the rest is command buffer setup and the CPU wait.

Group-by over string keys: a hash table instead of a sort

GroupByKeys turns a key column into dense ids 0 ..< K before any aggregate runs. For utf8 and binary columns that used to mean the GPU string dictionary: hash every row to 64 bits, argsort the hashes, mark run boundaries by comparing the bytes of adjacent sorted strings. Correct, but the argsort is eight radix passes over 50 million 64-bit keys — ~185 ms — and it costs the same whether the column holds a thousand distinct keys or ten million. Real group-by keys are low cardinality, so the sort was paying row-count prices for distinct-count work.

Kernels/StringHashTable.swift replaces it with an open-addressing table sized to the cardinality:

1. Hash every non-null row to 64 bits in one pass over the bytes (sht_hash64).
2. Estimate the distinct count by building a throwaway table over a 1/64 slice of the hash space — a row takes part only when (h >> 40) & 63 == 0. Each distinct string makes that decision once, so the occupied-slot count times 64 estimates the cardinality no matter how skewed the row frequencies are (a row sample would not: it sees the frequent keys and misses the rest). The estimate only sizes the real table, and being wrong costs a retry, never a wrong answer.
3. Build the table with three slots per estimated distinct value: linear probing, slots[s] = row + 1 so every atomic stays 32-bit (Metal has no 64-bit atomic add; it has 64-bit atomic min and max only, see UPSTREAM.md), and the key of an occupied slot is the string of row slots[s] - 1. Equality is decided by comparing the bytes of the candidate against that representative, the same rule is_in uses, so a 64-bit hash collision costs one extra probe and can never merge two different strings. Each row also records the slot it landed in and folds itself into that slot's lowest row index with an atomic_fetch_min guarded by a plain load.
4. Rank the occupied slots with the same GPU scan unique() uses, compact one representative row per slot, and relabel into first-seen order with two argsorts over the distinct count.

Two design choices are worth stating plainly. Nothing is cached beside the slot — not the hash, not a key prefix — because MSL only guarantees memory_order_relaxed: a companion array could be read before its writer had published it, and a probe that then walked past its own bucket would give one string two group ids. Reading hashes[slots[s] - 1] cannot go stale, because the hashes are written by an earlier kernel and never change. And because byte comparison decides equality, this path needs no re-hash retry and no host fallback, unlike the sort path it replaces. The only retry is a table that turned out too small; the last attempt gets two slots per row and an unbounded probe budget, so it cannot fail.

The byte comparison was measured against deciding equality on the 64-bit hash alone: 17.5 vs 13.7 ms at a thousand distinct keys, 22.9 vs 17.0 ms at 100k, and no difference at all at 10 million (where the random slot access dominates). A 20-30% saving is not worth an answer that is only correct to the birthday bound, so there is no unverified mode.

The result is identical to the sort path — same group ids, same group keys, same first-seen dictionary order, same null semantics (a null key still forms its own group, id K) — which is what Tests/ArrowMetalTests/StringHashTableTests.swift asserts, case by case, against both the host hash map and the old path.

Measured (M4 Max, 50M rows, 12-byte utf8 keys, best of 5), 2026-09-06

Group-by sum over a utf8 key column, key mapping included. Metal from Swift; Polars 1.44 (16 threads), pyarrow 25 and the 16-core Swift hash group-by on the same buffers.

dictionary_encode on the same column, which is the mapping plus the first-seen relabel and the gather of the distinct strings:

At a thousand keys the whole table is 4096 slots — 16 KB — so every probe is a cache hit and the pass is memory-bound on the key bytes. At ten million the table is 32 M slots (128 MB) and the cost becomes the random slot access, which is why the margin narrows from 10x to 2x. The 1000- and 100k-key cases are now 2.8x and 7.2x faster than pyarrow, where the sort path had been 3.5x and 1.2x behind it, and they cost the CPU about 4 ms against pyarrow's 700-2000.

Integer, boolean, temporal and dictionary key columns still take the range path (~22 ms at 50M rows); they never enter the hash table when they are group-by keys.

The same table for the distinct-value functions

unique, value_counts, count_distinct, mode and dictionary_encode over a primitive column had the same problem for the same reason: all five began with an argsort of every row, so all five cost the same whatever the cardinality, and all five were behind the CPU libraries that use a hash table (0.11x to 0.71x of pyarrow/Polars/pandas at 10M rows before this change). Kernels/HashTable.swift is the string table with the byte comparison removed — the key is the 64-bit value, so equality on it is exact — and the estimate, the growth retry, the rank scan and the id pass are literally the same code.

Turning a column into that key is one kernel: integers widen (sign-extended, so the map is injective), and floats are normalised first — every NaN to one bit pattern, -0.0 to 0.0 — so bit equality means Arrow value equality, which is what unique's sort path achieves by normalising before the sort. The order does not change either: the distinct values come back ascending, because the last stage argsorts the K representative values. Sorting K values instead of n rows is the whole trick. count_distinct skips even that — it is the occupied-slot count, so it never gathers or orders anything.

Because the table keeps the lowest row per slot, first-appearance order is available for one argsort of K int32s (HashGroups.firstSlotOrder, exposed as HashDistinct.firstRows) rather than another pass over the rows, which is what order="first_appearance" uses.

The threshold is 1 << 16 rows: below it the sort is a handful of small passes and the table's extra round trips do not pay for themselves. ARROWMETAL_NO_HASH=1 forces the sort path in a release build, which is how the before column below was measured.

M4 Max, 50M int64 rows, Swift, best of 3, 2026-09-06 (the "before" column with ARROWMETAL_NO_HASH=1):

Against the CPU libraries on the same buffers, called from Python through Benchmarks/python_gpu_bench.py (M4 Max, 50M rows, wall ms), from Benchmarks/results/python_gpu_bench_50000000_2026-09-07.txt:

Two rows need a note. unique at ten million distinct is 1.7x Polars because ArrowMetal orders its output and Polars does not: part of its 100.55 ms is the argsort of the ten million distinct values, which a hash-order unique would not pay. And mode is measured through a Python binding that computes it twice (_mode in python/arrowmetal/__init__.py calls the reduction once for the value and again for the count), so its row carries two reductions. At 1,000 distinct pyarrow's mode is ahead because the values span a narrow range and it counts into a direct-indexed table rather than a hash map.

Strings in two layouts: offsets + bytes, and views

A string column is held in one of two Arrow layouts (MetalStringArray):

• offsets + bytes (utf8 / binary): int32 offsets, one more than the rows, into one data buffer.
• views (utf8_view / binary_view, Sources/ArrowMetal/StringView.swift): 16 bytes per row and any number of data buffers. The first four bytes of a view are the length. A string of 12 bytes or fewer sits in the other twelve; a longer one keeps a 4-byte prefix there, then the index of a data buffer and its byte offset inside it. This is Polars' own layout for String and Binary columns, with data buffers that grow to 16 MB each.

Import. The C Data import takes vu / vz as they are: the views and every data buffer are wrapped in Metal shared buffers without a copy. A view buffer or data buffer does not have to start on a page: MetalArrowBuffer.wrapCovering wraps the pages that hold it and binds the buffer at its offset into the first one (the kernels read it byte-wise or 16 bytes at a time, never in page-sized units), so Polars' buffers, whose starts are not on a 16 KiB page, import without a copy. The validity bitmap keeps the page-aligned rule, because bitmap kernels read it in 32-bit words; it is copied when it is not page aligned or carries a bit offset. The import also totals the bytes of the non-null rows in one GPU pass over the views; a column over 2 GB is refused, the limit the large_utf8 import has. Export hands the same buffers back as vu / vz, so a Polars column crosses in both directions without a copy. An array with more than 64 data buffers (an arrow-rs or pyarrow concatenation of view arrays keeps every input's buffers) has its data buffers copied back to back into merged buffers of under 2 GB each and its views rewritten to point into them, since binding thousands of buffers for every kernel costs more than one parallel copy.

How a kernel reads either layout. Every kernel that reads strings reads them through one accessor, row(i, len), which returns a pointer to row i's first byte and its length (StringLayoutSource): StrOff over offsets + bytes, StrView over views. The kernel body is a template over the accessor, and StringLayoutSource.variants instantiates it once per layout (name and name_v; name_ov, name_vo and name_vv for a kernel over two string columns). A view kernel binds the views where the offsets kernel binds its offsets, and a small table where it binds its data: {0, count} followed by one {GPU address, byte size} per data buffer (MTLBuffer.gpuAddress), so a view that points into any buffer is one indexed load away. The encoder is told about the data buffers with useResources, since nothing binds them directly. The accessor checks every out-of-line view against its buffer's size and reads a view that points outside it as the empty string, so a malformed view, or the unspecified view under a null slot, cannot make a kernel read out of bounds (StringViewTests.testMalformedViewsReadAsEmpty).

What reads views. Lengths, hash32, the four pattern predicates, str_eq against a scalar and against another column, is_in / index_in (either side a view), the gather behind filter / take (the result is offsets + bytes), slice (the result stays a view), the sort keys, the string hash table (group-by keys, dictionary_encode, unique, value_counts), the Unicode and ASCII case transforms, the trims, replace, repeat, slice_codeunits, the pads, str_reverse, count_substring / find_substring, the ascii_is_* / utf8_is_* predicates, utf8_center / utf8_replace_slice, the fused expression kernels (am.query, the engines' filters and string predicates), concatenation of view columns (join keys, unions: the views and data buffer lists are appended and the buffer indices of the later parts moved, no string byte is copied) and every host-side row pass (regex, normalisation).

What converts. Anything else reads offsets / data, and the first such read converts the column on the GPU in a command buffer of its own (lengths, scan, byte copy) and keeps the result, so a column is converted at most once. That is str_concat, the splits, match_like, strptime, the byte-counting pads and utf8_zero_fill, binary_slice / binary_reverse / ascii_reverse, casts from strings and the IPC, Parquet, CSV and JSON writers. MetalArray.string_layout reports "view (converted)" for a converted column, am.string_view_conversions() counts conversions process-wide, and ARROWMETAL_TRACE_VIEW_CONVERSION=1 prints the call stack of each one.

Group-by

Dense group ids 0 ..< K come out of GroupByKeys; everything below aggregates over them. There are four shapes, and the point of the current design is that none of them sorts the key column.

Atomic accumulation (Kernels/GroupBy.swift, GroupBySource.swift) is one linear pass. For K <= 1024 each threadgroup keeps a private table in threadgroup memory and merges it into the device table once; above that the updates go straight to device memory. 64-bit sums are a pair of 32-bit atomic adds with an explicit carry, because Metal has no 64-bit atomic add (only min and max, UPSTREAM.md).

Correctly rounded Float64 sums (Kernels/GroupSumExact.swift, GroupSumExactSource.swift) give sum and mean over Float64 values without any group order. A correctly rounded sum does not depend on the order of its terms, so the rows are added where they lie, in two passes of 32-bit atomics. Pass one takes each group's largest exponent (an atomic max of the 11-bit field), counts its values and notes NaN, ±inf and +0.0 in a flags word. Pass two writes each value as an integer in a fixed-point window whose unit is 2^-G of the group's largest exponent's last bit and adds it to a 128-bit two's-complement accumulator, one 32-bit word at a time with the carry (or borrow) taken from each word's own read-modify-write, so the words hold the exact sum modulo 2^128 however the rows interleave. G = 74 - bitlength(rows): 53 significand bits, G guard bits and bitlength(rows) of headroom stay below the sign bit, so no sum can overflow it. A value whose last bits fall below the window is truncated and counted. The finalize, a thread per group, rounds once to nearest-even: with nothing truncated the accumulator is the exact sum, and the mean is the exact sum divided by the count in a long division, rounded once. With d values truncated the exact sum lies strictly between W - d and W + d units; the result is taken when both ends round to the same double, and the rare group whose rounding the window cannot settle (or that holds a NaN other than the canonical one) is summed exactly on the host, from its rows alone. Both come out of one call, which the GroupBy keeps, so a sum and a mean of the same column cost one pair of passes. Up to 1,024 groups each threadgroup accumulates in threadgroup memory and merges once, as the atomic accumulation does.

Sort-free extremes (Kernels/GroupByExtrema.swift) is how min, max and hash_min_max avoid the missing 64-bit atomic. Every element type maps order-preservingly into a 64-bit unsigned key. Pass one takes the extremes of that key's high word with 32-bit atomics; pass two takes the extremes of the low word among only the rows whose high word already equals the winner. Some row attains the winning high word, and among those the smallest low word is the overall minimum, so the answer is exact. Types four bytes wide or narrower skip pass one — their whole key is the low word. A group's four words sit together in one 16-byte entry, a row reads them before it writes and issues an atomic only when it improves on what it read, and no count is kept: a group has a value exactly when its minimum key is at or below its maximum key, which the empty identities never are. Both extremes come out of one call, which the GroupBy keeps, so a min and a max of the same column cost one pair of passes. first / last ride the same kernel over a masked row index.

Counting sort by group id (Kernels/GroupOrder.swift, GroupOrderSource.swift) replaces the argsort for the aggregates that need each group's rows together — list, distinct, product, the central moments, the order statistics. Histogram, scan, scatter: one pass over the keys rather than four over keys and payload. Two scatters share the scan:

• chunked, when a per-block histogram fits the budget. A "block" is one simdgroup, so a row is ranked against 32 lanes rather than 256, and each block owns a reserved slice of every group's run and fills it in row order. Stability is exact and needs no atomics.
• atomic, when the group count is large. One atomic bump per row places the rows, then cs_fix (a thread per group) or cs_fix_tg (a threadgroup, bitonic) sorts each run back into row order. Cheap exactly while runs are short, which is the case that forced this path; a longer run falls through to the argsort, so no input is worse off than before.

The ordering is cached on the GroupBy, so N aggregates over one grouping pay for it once.

Segmented reductions come in two shapes. _wide gives a threadgroup to a group and tree-reduces in threadgroup memory; _narrow gives a single thread to a group. Ten million groups of five rows want the second one — the first would launch a quarter of a billion threads to do fifty million adds.

Grouped variance, stddev, skew, kurtosis (Kernels/GroupMoments.swift) run the two-pass shifted algorithm over that ordering in true binary64: an exact d_add sum for the per-group mean, then the sums of d = x - mean and d^2 (and d^3, d^4) about it, finished as m2 = (sum d^2 - (sum d)^2 / n) / (n - ddof). Shifting first is what makes it accurate — sum x^2 - (sum x)^2 / n cancels catastrophically on data far from zero — and the (sum d)^2 / n term removes the error in the mean itself. On 200k float64 values offset to 1e9 in 997 groups, the largest relative error against an exact rational reference is 1.9e-16; pyarrow's own answer on the same input is 2.5e-10 off. The Float32 deviations this replaced cost about 1e-5.

Finalizers write the output column and its validity on the GPU. A Swift loop over the group count is invisible at a thousand groups and is the whole cost at ten million.

Before and after (Apple M4 Max, best of five, 10M rows, one group_by per call), 2026-09-06

At 50M rows, against the fastest idiom of Polars / pyarrow / pandas (Benchmarks/results/full_matrix_2026-09-07-parallel.csv): min 3.32x / 4.93x / 4.57x at 1k / 100k / 10M groups, where it was 0.25x / 0.56x / 0.21x; variance 0.96x / 2.52x / 4.06x, where it was 0.05x at 100k; first 7.95x / 5.81x / 5.12x; list 10.85x / 9.72x / 5.70x. Two caveats on those ratios. At a thousand groups every numeric grouped aggregate lands near 3.3x to 4.1x, sum and count included, because a fresh group_by([...]) spends about 7 ms of the 10 ms rebuilding the dense key mapping — reuse the object and the aggregate itself is 3 ms. And count_distinct by key was the remaining shortfall and is now 3.4x to 7.2x the fastest CPU idiom (Where ArrowMetal needs to improve) — the sort of the key column is gone from every aggregate.

Latency (small inputs)

Measured floor on M4 Max: an empty kernel with encode + commit + wait costs about 60-70 µs measured (The persistent-kernel result), 110-160 µs per call for sum and filter at 1,000 rows in the matrix's latency family; ten kernels in one command buffer cost ~100 µs in total. So the fixed cost is the round trip, not the kernel, and the only lever is fewer round trips.

Unbatched, every public call is one command buffer. filter already fuses its three kernels into one command buffer with a GPU scan.

Batched execution (MetalContext.batch { }, am_batch_begin/end in C, with am.batch(): in Python): kernels append to one serial compute encoder; results are created immediately with their buffers, but a result whose length the GPU decides (filter) or whose null count needs a bitmap read is marked pending. Any CPU-side read (a reduction, length, nullCount, subscripts, export) triggers a sync point: the open command buffer is committed and waited once, deferred fix-ups run (lengths, null counts, take bounds errors), and a fresh batch is opened so the caller keeps batching. While any batch is open the pool parks returned buffers instead of recycling them, so pending GPU work can never observe a reused buffer. Per-thread batches; nested batch calls join the outer one.

Lengths flow on the GPU. Every kernel takes its element count as device const uint* nPtr. A pending filter result binds the GPU-written total as that pointer and is dispatched at its maximum possible size, so the next kernel (compare, arithmetic, cast, bitmap ops, another filter, a reduction's partial pass) never needs the CPU to know the length. A reduction still syncs once to read its partials.

Float64 without hardware doubles. Kernels/DoubleMath.swift carries a software IEEE-754 binary64 (d_add, d_sub, d_mul, d_div, d_sqrt) over ulong bit patterns, correctly rounded. Sum accumulates with d_add; arithmetic kernels run one element per thread. d_div is a Newton reciprocal seeded by one hardware float division, with the exact 128-bit remainder N - q·D settling the last bit; d_sqrt refines a hardware rsqrt seed with Newton steps in fixed point and settles the last bit with the exact remainder. Both are correctly rounded rather than close, and DoubleMathTests holds them to Swift's own Double bit for bit. add and multiply run at this ~390 GB/s these single-pass rows reach at 50M rows and divide within 15% of it (331 GB/s), so the software arithmetic is all but invisible in a bandwidth-bound query. cast is the one float64 path that still runs on the host, and modulo is the one binary operator with no float64 kernel — both say so in Coverage summary.

The transcendentals are a different story, and worth being explicit about. Kernels/DoubleTranscendental.swift (expm1, log1p, logb, hypot, the ten RoundModes) and Kernels/DoublePower.swift (exp, ln, log2, log10, pow) evaluate in binary64 from end to end, over that same software arithmetic — not by narrowing the column to float, calling Metal's own library and widening back, which is what these kernels used to do for about seven correct significant decimal digits out of sixteen. pow is the one that sets the bar: a 1-ulp result needs the product y·log2 x, which reaches 1024 in magnitude, accurate to 2⁻⁶¹ absolutely — more than a double holds — so log2 x is carried as an unevaluated hi/lo pair whose high part keeps 21 significant bits, and y is split the same way so that y₁·t₁ is an exact double. That is fdlibm's layout, and its Remez coefficients and hi/lo constants are reused verbatim; the three logarithms fall out of the same reduction, more accurately than a direct series would give them.

Accuracy is measured, not derived. DoubleTranscendentalTests compares each function with Foundation over 10⁶ random inputs drawn across its whole domain and prints the ulp histogram; these are those numbers, measured 2026-09-06. The measured column is what the run reports; the test asserts a 2-ulp bound on everything but sqrt, which it holds to bit equality, so a regression of one ulp shows up as a changed number here before it fails the suite. The trigonometric family is a separate kernel and a separate budget: measured 2-5 ulp against a 6-ulp assertion (TrigTests, Coverage summary).

The C99 edge table — x^0, 0^y, 1^y, (-1)^int, infinity and NaN propagation — matches libm bit for bit, and the _checked twins are the unchecked kernel plus a read-only check pass, so they inherit every value and raise at exactly the same boundaries.

The price is throughput, and it is the cost of the accuracy. At 50M rows on an M4 Max, the old float-detour ln ran at 250 GB/s because it was memory bound; the binary64 one runs at 10 GB/s because it is compute bound on forty-odd software operations per element. That is a 25x drop in throughput for nine more correct digits, and it is the right trade for a library whose whole claim is that a Float64 column means Float64. The float32 kernels are untouched and still take the hardware path.

Async

MetalContext.batchAsync is batch { } without the wait. It records body's kernels into one command buffer exactly as batch does, then commits and hangs an MTLCommandBuffer.addCompletedHandler off it instead of spinning. The deferred fix-ups — afterFlush closures (pending lengths, null counts, take bounds errors), pool.releaseParked(), the openBatches decrement — run on the GPU's completion thread, so the calling thread is released the moment the work is committed. Two forms:

let kept = try await ctx.batchAsync { try amount.filter(where: .gt, 100) }   // suspends, no thread held
ctx.batchAsync({ try amount.filter(where: .gt, 100) }) { result in ... }     // returns immediately

flush is split into detachBatch() (end encoding, unhook from the thread) and finishBatch(_:) (the post-completion fix-ups, which touch no thread-local state); the synchronous path calls both back to back around its wait, the async path puts the completion handler in between. Batches stay per-thread: body runs synchronously, before the first suspension, so the batch is opened, filled and detached without a thread hop. Nested calls join the enclosing batch, as with batch.

Sync points still exist inside body. A reduction, length or nullCount of a pending filter result, a subscript or an export commits the open batch and blocks right there, exactly as in batch { }. So body should return something that does not force a sync — a pending array from filter, compare, take, cast or arithmetic. Its length, null count and contents are resolved by the time the await returns. For a scalar there is an async accessor: MetalArray.sumAsync() (and meanAsync()) records the reduction kernel in an async batch and combines the per-threadgroup partials on the completion path, so nothing blocks. Deferred errors are thrown from the await (or delivered as .failure), not from the recording.

Not available: async accessors for min/max/group-by, and cancellation (a committed command buffer runs to completion).

CPU/GPU router

Every public call without a batch pays one command-buffer round trip (above), and on small inputs that floor is the whole cost. The router is the answer the engine gives to that: for seven operations it has a second implementation, a tight single-threaded CPU loop, and it runs that loop when its crossover table says the CPU is faster for an input of this size (the table and what its CPU side measured are below). The output is the same Arrow array either way, byte for byte.

What is routed. sum, min, max (and mean, which is sum over the valid count), compare (scalar and array), add / subtract / multiply (scalar and array), filter by a mask and the fused filter(where:), and GroupBy.sum over integer values (and GroupBy.sumUnsigned, uint64 kept unsigned, which the C ABI and Python use for uint64) with at most Router.groupBySumMaxKeys (1,024) keys, which is also what hash_sum reaches through the dense-key mapping. Every other operation is unchanged and always runs its GPU kernel.

Where the decision is made. Once, at the top of each routed operation's public entry point (Kernels/Reductions.swift, Compare.swift, Arithmetic.swift, Filter.swift, GroupBy.swift), before anything is encoded: if Router.decide(.sum, self).isCPU { return RouterCPU.sum(self) }. Not in MetalContext.run, which sees an encode closure and neither the operation nor the row count. The code is in Sources/ArrowMetal/Router/.

The decision, in this order (Router.decide):

1. No CPU path, no choice. If the operation has no CPU loop for this input the GPU runs it, as before the router, and the decision records why ("no alternative: ..."). That is Float32 min / max, Float32 arithmetic and the fused Float32 filter(where:) predicate, which compute in hardware float on the GPU, where subnormals are flushed (Evaluation method, finding 1), so a CPU loop would not give the same bits; divide, which is not routed; and a group-by sum over more than 1,024 keys.
2. Pending input. An input whose length the open batch is still deciding stays on the GPU.
3. Mode gpu runs the GPU.
4. Inside a batch, the GPU. MetalContext.batch shares one command buffer's floor across every kernel in it, and queued kernels may be producing this very input; a CPU read would have to commit and wait for them, which is what batching exists to avoid. So any open batch keeps the routed operations on the GPU, including under mode cpu.
5. Mode cpu runs the CPU loop.
6. Mode auto (the default) looks the operation up in the crossover table: below the crossover the CPU loop runs, at or above it the GPU. The table was measured on int64 columns; integer columns of every width use those rows, and floating-point columns, which have no measured crossover, stay on the GPU. The arithmetic row was measured on add, and subtract uses it. multiply, whose 64-bit integer form costs more per element on the CPU than an add, has its own row (Router.crossoverRows(arithmetic: .mul)), fitted from the shipped multiply loop (below).

The router does not adjust for residency: every routed call starts from Metal-shared buffers (an imported column is mapped before the operation is called), which is the state the table was measured in. It keeps no state across calls beyond the table in force, the modes and the last decision, and it never times or estimates anything at call time: Router.route, the rule every routed call goes through, is a pure function of the operation, the value type, the row count, the mode, the batch state and the table row.

The table is data generated from the measurement, not typed: Benchmarks/router_table.py --from-check Benchmarks/results/router_check_2026-09-24.csv fits it from a Benchmarks/router_check.py run, which times every routed operation with the router pinned to gpu and to cpu, so the CPU side is the shipped RouterCPU loops, and writes Sources/ArrowMetal/Router/RouterTable.swift, a Swift literal, so there is no resource bundle to load. The step alone would send every size between the last measured size where the CPU was ahead and the first where the GPU was ahead to the CPU, so the script fits the crossover once, offline: inside that bracket both paths are taken as straight lines between the two measured points and the crossover is where they meet. It never leaves the measured bracket. multiply has its own row, fitted the same way from the multiply(int64, 3) rows, because a 64-bit integer multiply costs the CPU more per element than an add. router_table.py --check fails when the committed table no longer matches the results file it names, and python/tests/test_router.py runs it. The values are in the generated file, in Router.crossoverRows(_:) and Router.crossoverRows(arithmetic:) from Swift, am_router_crossover and am_router_multiply_crossover from C and am.router_crossovers() from Python.

Benchmarks/results/router_check_2026-09-24_after_refit.csv is the same check run against the refitted table: auto took the faster path in 71 of 72 cases. The one miss is compare(int64 > int64) at 3M rows, by a margin of 1.10, because the compare row is fitted on the scalar compare. Both runs compare the GPU with a single-core CPU loop; a background process holding one of the sixteen cores does not enter either side.

The table can also be generated from Benchmarks/results/router_2026-09-17.json, the size sweep behind The crossover (router_table.py --json), whose CPU side is the bench's own single-core loops in Sources/ArrowMetalBench/main.swift rather than RouterCPU; the generated file's header says which source it came from, and --check regenerates from that source.

A table per machine. python -m arrowmetal.router calibrate [--quick] runs the same sweep on the machine it is run on, fits it with the same code (python/arrowmetal/_router_fit.py, which router_table.py loads too) and writes ~/.arrowmetal/router/<chip>.json. At first use the router takes ARROWMETAL_ROUTER_TABLE (a path, or shipped), else that file for this chip, else the shipped table; python -m arrowmetal.router explain <op> <rows> prints a decision with the table row it came from. The crossover describes the pipeline and the JSON format.

The CPU side (Router/RouterCPU.swift) is one generic loop per shape, over the Arrow layout: reduce (a branch-free fold where a null slot feeds the operation's identity), map (arithmetic, every slot including those under nulls, as the GPU computes them), map-to-bitmap (compare, 32 bits per word, bits past the length zero), compact (filter, by 64-bit selection words), and a dictionary loop for the group-by sum (a keyCount-slot table of wrapping 64-bit sums and counts). They are single-threaded; CROSSOVER.md also records the bench's all-core loop for reference. CPUReference stays the tests' oracle; its closure-per-element walk is one to two orders slower than these loops (CROSSOVER.md, cpu-ref).

Byte-identical output. For every routed operation the CPU loop returns what the GPU kernel returns: the same values, the same validity buffer (the scalar forms share the input's bitmap exactly as the GPU does; the array forms AND the two bitmaps), the same null count and the same buffer sizes. Integer arithmetic wraps as the GPU does. Float64 arithmetic is the hardware add, subtract or multiply, which is correctly rounded like the software d_add / d_sub / d_mul, with the software kernels' NaN rule applied when the result is a NaN (the first NaN operand, quieted; otherwise the positive quiet NaN). Float64 min / max compare the GPU's order-preserving keys, so NaN is skipped and -0 and +0 are one key. Float sums reproduce the GPU's summation order exactly rather than adding in row order: thread t of the groups x 256 grid adds its 4-element blocks t, t + grid, ... and then tail element t, each threadgroup folds its 256 accumulators as the kernel's tree does, and the host adds the group partials in order, as finaliseSum does. The sum is therefore the GPU's to the bit, NaN payloads included, with no Accelerate and no tolerance. RouterTests (Swift) and test_router.py (Python) hold both paths to this on every type, on sizes across the word and threadgroup boundaries, with 0%, 10% and 90% nulls, on slices at an offset, with signed zeros, infinities, quiet and signaling NaNs with payloads and subnormals, and on float sums past the point where every GPU thread takes more than one block.

Overrides and visibility.

The per-thread override wins over the process mode. Bindings other than Python reach the router through the C functions above; no operation signature changed.

What the router does not do. It does not remove the floor, and it does not touch the operations whose cost is not the floor (memory-bound single passes, software binary64 transcendentals, host regex, views, temporal extraction: Where ArrowMetal needs to improve). It chooses between ArrowMetal's own two paths, never a CPU library. Strings are not routed: the string rows are behind a 12-core Acero, and a single-threaded loop would not change that.

Top-k and order statistics: GPU radix select

top_k, quantile and approximate_median all ask the same question — which rows sit at which rank — and all of them used to answer it by sorting the whole column. Sorting 50M Int64 is eight LSD radix passes and a gather, about 130 ms; the answer needs a few hundred bytes of it. Radix select replaces the sort with two passes over the column, whatever k is.

The algorithm

Everything runs on the order-preserving key: the same map argsort uses (TopKSource.keyMap), so nulls, NaN, -0.0 and the ascending/descending flag behave identically and the answer is exactly argsort(descending:)[..<k], index for index.

1. Histogram (rs_histogram, Kernels/RadixSelectSource.swift). One pass counts the top 8 bits of every non-null row's key. The counts are kept per simdgroup-sized sub-block — one 256-bin private histogram per simdgroup in threadgroup memory, so the only atomic contention is between 32 lanes — and written digit-major as counts[digit * subBlocks + sub]. Summed over sub-blocks that is the global histogram; left per sub-block it is exactly the offset table step 3 needs. That double duty is what keeps the fast path at two passes instead of three.
2. Pick the bin. The host walks the 256 totals to the first digit whose running total reaches k. Rows with a smaller digit are guaranteed winners — there are fewer than k of them — and the rest of the answer lives in that one bin, about n/256 rows for well-spread keys.
3. Compact (rs_scatter). A second pass writes the winners and then the bin's rows, both in ascending row order, into one array of (key, row) pairs. Every winner's key is smaller than every bin key, so a stable sort by key alone turns that array into (key, row) order — the total order — with no second sort key and no 96-bit comparisons. Because each sub-block owns a contiguous slice of the output, the scatter needs no threadgroup barriers: a simd_prefix_exclusive_sum per 32 rows is the whole ranking.
4. Refine. The bin is still ~n/256 rows, far more than the answer, and it would dominate the final sort. So steps 1-3 run again on the compacted array and the next digit — microseconds of work that shrinks the bin by another factor of 256. One refinement round is enough; the loop stops when the bin is under TopK.refineCap (4096) rows.
5. Order. Up to TopK.bitonicCap (2048) survivors are sorted by a bitonic network in a single threadgroup (rs_sort_small, one dispatch, the slot index as the tie-break), which covers every k up to about 1024. Above that the stable LSD radix sort orders them and the first k rows are the answer.

Ties. Heavily tied data can put nearly every row in the chosen bin, which would make step 3 copy the whole column. Step 2 therefore repeats on the next digit without compacting while the bin is over the candidate budget (1M rows). When the key runs out of digits every remaining row is an exact tie, and ties are settled by row order, so the scatter writes only the first k - winners of them: the output is exactly k pairs however many rows share the key.

The same in-place narrowing covers the other skewed case: a float64 key's top byte is the sign plus seven exponent bits, so a column of uniform doubles concentrates in a handful of bins rather than spreading over 256, and a rank in the middle of one of them costs one extra pass before the bin fits the budget.

Nulls. The histogram and the scatter both skip invalid rows, so the selection is over the non-null rows only. When fewer than k rows are non-null the answer has to reach into the null rows, which only the sort places — that case, and k above half the non-null rows (where the compaction would copy most of the column anyway), fall back to argsort.

Order statistics: selecting a value, not rows

A quantile wants the value at a rank, not the rows, so Kernels/RadixSelectValue.swift runs the same narrowing with the winners counted but never written (wantLt = 0). Once the bin is under 4096 keys it is read back and finished on the host, which ends the round trips immediately instead of after another six digits. kthElement(k, largest:) is that search plus the inverse key map; quantile(q) asks for the one or two adjacent ranks that bracket the interpolated position, and because they are adjacent one search usually settles both. approximateMedian() is quantile(0.5).

The reconstructed value is exact for every integer type. For floats it differs from indexing a sorted copy on exactly the two values the key deliberately merges: -0.0 comes back as +0.0 and a NaN loses its payload. Both are values Arrow's total order calls equal, and quantile returns a Double, so its answers are unchanged.

tdigest still sorts: its centroids are defined by the whole sorted sequence, not by one rank, so selection cannot produce the same digest.

Which path runs

The crossover at 2^19 rows is where radix select's two passes plus one host readback start beating a single selection dispatch. Below it the per-threadgroup kernel is one command buffer; above it, it is bandwidth that matters and radix select reads the column at close to peak.

Numbers (M4 Max, 50M rows)

Called from Python on the same in-process data (PYTHONPATH=python python Benchmarks/python_gpu_bench.py 50000000 5), against pyarrow 25, Polars 1.44 (16 threads) and numpy 2.5. Wall ms, from Benchmarks/results/python_gpu_bench_50000000_2026-09-07.txt. Before radix select, top_k(10 000), top_k(100 000) and both quantiles ran a full argsort or sort of the column.

That is 9.3x / 11.5x / 49.0x pyarrow's select_k_unstable at the three k, and 108x / 156x pyarrow's quantile. The target was 3x. top_k is flat in k, because k only changes how many of the compacted candidates the final ordering has to sort, not how much of the column is read — while pyarrow's heap-based select_k_unstable goes from 24.6 ms to 154.8 ms over the same range.

The column is read twice, so the useful bandwidth figure is 2 x 400 MB over the wall time: 302 / 265 / 253 GB/s for the three top-k, against a device peak of ~546 GB/s. The gap is the mid-flight readback — the host reads 256 counts to pick the bin, so the two passes cannot share a command buffer — plus the final ordering, which grows with k.

The CPU stays free throughout: 1.4-1.8 CPU-ms for the top-k calls and 0.9-1.2 for the quantiles, against 24.6-362.3 CPU-ms for the CPU libraries on the same questions (pyarrow, Polars and numpy; Benchmarks/results/python_gpu_bench_50000000_2026-09-07.txt, rows 259-278).

The radix sort's block size is also now adaptive below ~256k rows: a fixed 4096 elements per block left a 20k-element sort — the size top-k's final ordering lands on — running on five threadgroups. Inputs above ~256k rows are unaffected, so the 50M argsort is unchanged.

Index arrays: UInt32 row numbers

Every call that returns row numbers into its input returns them as UInt32 (Arrow uint32, format I): argsort (numeric, boolean, temporal, utf8 and binary), topK, partitionNthIndices, the lexsort, the hash join's index pairs (hashJoin, JoinExtra.indices), the ranks (rowNumber, rank, denseRank, maxRank, rank(tiebreaker:), and the engine's row_number / rank / dense_rank window columns), GroupByKeys.representativeRows() and GroupByKeys.rowIndices(_:_:). UInt32 is the width the Int32 arrays had, so the kernels, the bytes they move and the values they write are the same; it is the type arrow-rs sort_to_indices and Polars' IdxSize use. pyarrow returns uint64 for the same functions, with the same values. Before, a row number of 2^31 or more came back negative in an Int32 array.

Arrays hold at most 2^32 - 1 elements, so every row number fits. A call whose row numbers would pass 2^32 - 1 is refused rather than wrapped, with the error "<op>: row N does not fit the UInt32 index type (row numbers go up to 4294967295)" (Dispatch.checkIndexRows). take accepts Int32, Int64 and UInt32 indices, as Arrow's does; an Int64 index the 32-bit gathers of nested, fixed-size-binary and interval columns would have to narrow is checked against the range first, so it is an error and never another row.

The integer outputs that are not row numbers into the input keep their types: dictionary codes and index_in positions are int32 (Arrow's defaults), group ids are int32 (a group-by holds at most 2^30 groups), inverse_permutation is Arrow's signed int32 and refuses an index column of more than 2^31 rows, list_parent_indices() is int32 and refuses a list column of more than 2^31 rows (listParentIndices64() is int64), and indices_nonzero is uint64, as in Arrow. The hash join takes at most 2^31 - 1 rows per side and returns at most 2^31 - 1 pairs.

Timed against the Int32 build, alternating in one process (Benchmarks/index_type_bench.py, Benchmarks/results/index_type_2026-10-01.csv): an untimed warm-up of at least 100 ms per call, 8 rounds of up to 30 calls, and the three calls whose median moved by more than 1% timed again alone, 20 rounds of 100 calls (marked *). No call is more than 1% slower in both best and median time:

A median of a call of a few milliseconds moves by several percent from one block to the next (top_k 10 at 10M: 5.453 → 5.012 in the first block, 4.878 → 5.243 alone); a loss shows in both columns.

Chunked columns and the import

A column often arrives as many Arrow arrays: a pyarrow ChunkedArray, a multi-chunk Polars Series, one column of a stream of arrow-rs RecordBatches. am_import_chunks (Sources/ArrowMetal/ChunkedImport.swift) takes all of them in one call and writes each chunk straight into the final page-aligned Metal buffers, with no concatenated copy in between. The work is split by output position, not by chunk, into pieces that the copy threads take one after another, so 6,104 chunks of 8,192 rows and 16 large chunks spread alike, a thread on a slower core takes fewer pieces, and no two threads write the same byte:

• fixed-width values: one memcpy per chunk piece, from the chunk's Arrow offset;
• bitmaps (validity, boolean values): assembled 64 output bits at a time from each chunk's bits at its own bit offset, since chunk boundaries are generally not byte aligned; a chunk without a bitmap, or with a null count of 0, contributes set bits, and the null count is counted from the merged bitmap;
• utf8 / binary and their large variants: offsets rebased by a prefix sum of the chunks' byte lengths and narrowed to int32 (a total of 2 GB or more is refused, as for one large_utf8 array), bytes copied;
• utf8_view / binary_view: views copied with each out-of-line view's buffer index and offset rewritten. Data buffers shared by several chunks (the slices of one array) are taken once; up to 64 distinct buffers are wrapped without a copy, more are copied back to back into merged buffers. A single utf8_view array with more than 64 data buffers takes the same path.

Dictionary, nested, run-end encoded and extension types are not taken: the call returns 3 without reading or moving anything, and the bindings concatenate those columns first, as before.

The view pass. The views are rewritten by am_rewrite_views (Sources/CArrowMetalCopy), C with NEON intrinsics: vld4q_u32 loads four views and splits them into lengths, prefixes, buffer indices and offsets; every lane is rewritten with vector selects (an inline view is kept as it is, an out-of-line one gets its merged buffer index and its offset plus its buffer's base) and checked in the same instructions: length and offset below 2^31, buffer index below the chunk's buffer count, offset plus length within its buffer. vst4q_u32 writes the four back, and the lengths of the valid rows are summed for the column's byte total. Nothing branches per view. A chunk with one data buffer uses its target as a constant; a chunk with several loads each lane's target by its buffer index. A block of 2,048 rows with a view that fails the check is rewritten again view by view, which gives that view an index no buffer has, so it reads as the empty string, as after the single-array import. On one thread (ARROWMETAL_IMPORT_THREADS=1) three 50M-row utf8_view columns in 16 chunks import in 85.5 ms, against 339.3 ms with the view-by-view loop it replaces (6,104 chunks: 134.5 against 381.6 ms; best of three alternating rounds, Benchmarks/results/chunked_import_2026-09-30.csv).

Copy threads. The number of threads is a measured policy (ImportThreads). 1, 2, 3, 4, 6, 8, 12 and 16 threads were timed with Benchmarks/chunked_import_bench.py --threads at 100,000 to 50,000,000 rows (Benchmarks/results/import_threads_2026-09-30.csv), and Benchmarks/import_threads_policy.py turns the sweep into two tables with one rule per import: a thread count is allowed when its CPU time is at most 32 ms or at most twice the one-thread import's, whichever is larger, and the policy takes the fewest threads that come within 5% of the fastest allowed count. Copies have no CPU limit from the size where the limit costs wall time: the smallest measured size from which, at every larger size too, some type and chunk layout is more than 5% slower under the limited count than at its own fastest count. In the sweep that is 381 MiB per column (50M int64 rows in 16 chunks: 4 threads within the limit, 7.0 ms, against 6.5 ms on 16), and the sizes below it, up to 305 MiB, lose nothing to the limit; from there a copy takes the fastest count. Sizes are grouped by the bytes an import writes (both chunk layouts together), and between two measured sizes the boundary is their geometric mean:

Below about 2 MiB the copies are split into pieces of at least 1 MiB, so the first row's two threads run one piece; 250,000 int64 rows in 16 chunks take 0.10 ms on 1 to 16 threads. A memory copy of the largest columns stops gaining wall time at four threads until all sixteen cores run it, and those sixteen take 5.6-6.0x the one-thread CPU time in 16 chunks (4.2-4.5x in 6,104 chunks); the view pass, bound by its arithmetic, divides its wall time by the thread count at almost no CPU cost up to six. Wall ms / CPU-ms, three columns, in 16 chunks and (last column) one utf8_view array with 6,104 data buffers, best of 5 runs (Benchmarks/results/import_threads_2026-09-30.csv; the last column is its single_import columns):

am_set_import_threads(n) (Python am.set_import_threads, Rust arrowmetal::set_import_threads, Swift ImportThreads.limit) or the environment variable ARROWMETAL_IMPORT_THREADS caps every step at n threads; 1 keeps the import on the calling thread, 0 is the policy.

The page probe. A copy-free import has to know that every page it wraps is mapped. The probe used to be mincore, which reports on every page. mach_vm_region answers per VM region, and a large allocation is one region. Three page-aligned pyarrow int64 columns of 50M rows (400 MB each) imported in 15.71 ms and 15.7 CPU-ms with mincore and import in 0.05 ms with mach_vm_region; at 10M rows 3.10 against 0.02 ms, at 1M rows 0.32 against 0.01 ms (best of two rounds, the single_import columns of the int64 rows in 16 chunks of Benchmarks/results/chunked_import_2026-09-28.csv, where old is the build with mincore). A buffer that is not page aligned is still copied.

In practice pyarrow's buffers of 1M rows and more (pa.array from numpy, compute results, combine_chunks) start on a 16 KiB page and are wrapped; so are arrow-rs concat results of fixed-width and utf8 columns at 1M rows and more.

Measured (M4 Max, Benchmarks/chunked_import_bench.py, three columns per case, the previous build (c4e2f6f, which copied on all sixteen cores) and this one alternating; Benchmarks/results/chunked_import_2026-09-30.csv). At 10M rows: three rounds, best of 5 runs per round, best of the rounds. At 50M rows: each case re-timed alone, eight rounds of 100 runs, best (the 1-minute load was 2.8 when the block started). "combine + import" is combine_chunks() and the import of the combined column (one round); "chunked" is am.array(chunked_array):

Every case in the table is as fast as on the previous build or faster on the best run, and on the median run every case but utf8 at 50M in 16 chunks, whose median run is 14.77-14.82 ms in each of the eight rounds against 14.21-14.45 ms (Benchmarks/results/chunked_import_2026-09-30.csv, as are the figures below). CPU time: the 50M copies, where the policy has no CPU limit, take 1.09-1.11x the previous build's (and 4.8-6.0x the single-threaded combine_chunks and import, for 0.38-0.65x its wall time); the 10M copies take 0.54-0.88x; the utf8_view pass takes 0.28x at 50M in 16 chunks and 0.41-0.56x elsewhere in the table. At 20M and 30M rows (three rounds) every copy case is within 1.3% of the previous build's best time or faster, at 0.81-1.22x its CPU time (utf8 at 30M in 16 chunks: 8.67 against 8.56 ms, 128 against 104 CPU-ms); at 1M rows within 0.01 ms or faster (utf8_view 0.52 against 0.95 ms in 16 chunks). utf8_view imports 1.08x the time of utf8 in 16 chunks (14.26 against 13.21 ms) and 1.83x in 6,104 chunks (38.96 against 21.27 ms): there its 6,104 data buffers are copied into one merged buffer, so the column moves 16 bytes of view per row plus the string bytes, against 4 bytes of offset per row plus the string bytes for utf8.

One utf8_view array with 6,104 data buffers (combine_chunks of the 6,104-chunk column above) imports in 26.5 ms and 274 CPU-ms (previous build: 44.8 ms and 548 CPU-ms; the single_import columns of Benchmarks/results/chunked_import_2026-09-30.csv; the build before the merged copy path: 150.5 ms and 143 CPU-ms, Benchmarks/results/chunked_import_2026-09-28.csv); at 10M rows in 5.6 ms and 59 CPU-ms against 9.2 ms and 112 CPU-ms (chunked_import_2026-09-30.csv).

Through the engines, with the previous build alternating: Benchmarks/polars_engine_bench.py --sizes 2000000,50000000 --cases a,c,i,t1,w1,x2, MetalEngine with the import cache cleared, takes 0.98-1.01x of the previous build's best time at 50M rows (three rounds) and 0.73-1.06x at 2M rows (the 1.06x, the 200-group sum under the default engine, takes 0.91x of its median); the two 2M cases that differed more with every shape taken, the 100,000-group sum and the join, re-timed alone over eight rounds of 100 runs, take 0.92x and 0.98x of its best time and 1.02x and 1.00x of its median (Benchmarks/results/chunked_import_polars_2026-09-30.csv). The DataFusion rule's sorts (datafusion/examples/bench.rs --families sort, rule on, two rounds, datafusion/results/datafusion_sort_import_2026-09-30.csv) take 0.93-0.99x of the previous build's time at 50M rows in both table layouts, at 0.91-1.04x its CPU time (int64 key in 8,192-row batches: 68.84 against 70.63 ms), and 0.95-1.05x at 10M rows with 0.54-0.88x its CPU time.

• Not a query planner or SQL engine. It is the compute layer that DuckDB, DataFusion, Polars plugins or an app can call.
• Not a tensor library; MLX is that.

Fused expression queries

One Metal kernel for a whole Arrow compute expression, instead of one kernel per operator.

(a * 2 + b) / (c + 1) - d is five Arrow kernels: five reads of a column, five writes of a temporary, five command buffers (or five dispatches inside one). At 50M rows on an M4 Max that costs 24 ms even when every kernel runs at memory bandwidth, because the data is moved five times.

The expression compiler lowers the whole tree — arithmetic, comparisons, null logic, casts, string predicates — into one runtime-generated MSL kernel, so the inputs are read once and the outputs are written once. Same expression, 16 ms, and the CPU does 0.7 ms of work instead of 2.8 ms.

import ArrowMetal

let batch = try MetalRecordBatch(names: ["region", "amount"], columns: [.int32(region), .float32(amount)])
let total = try batch.query(
    query().filter((col("region") == 2) && (col("amount") > 100)).sum(col("amount"))
).onlyScalar                                   // .double(...)

import arrowmetal as am
am.query(table, am.filter((am.col("region") == 2) & (am.col("amount") > 100)).sum(am.col("amount")))
am.query(table, ((am.col("a") * 2 + am.col("b")) / (am.col("c") + 1) - am.col("d")).alias("r").project())
am.query(table, am.group_by(am.col("k"), 1000).sum(am.col("v") * 3 + 1, "s"))

---

What it is

How the lowering works

The emitter walks the tree once, bottom up, and appends straight-line MSL to one inline void am_row(...) function. Every node becomes two registers: the value and a bool validity.

• Common subexpression elimination is keyed on the node's canonical text, so a subtree that occurs four times is evaluated once. ExprTests.testDeepMixedTreeWithSharedSubtrees builds a 20+ node tree with a shared subtree used four times and checks it against a Swift oracle.
• Validity is compiled in, not carried in buffers. A node that cannot be null carries the literal true instead of a register, and the Metal front end folds the whole null chain away. A node that can be null gets one bool; the input bitmaps are read one 32-bit word per 32 rows into a register (am_vword) and the output bitmap is built in a register and stored once per 32 rows.
• Vectorised loads. The kernels are word shaped: one thread owns 32 consecutive rows (the same shape the existing compare and filter kernels use). When every leaf is a plain numeric column the thread loads its 32 rows as eight four-wide vector loads per column; a boolean or utf8 leaf, whose access is not a contiguous vector, falls back to the scalar path.
• Float64 has no hardware support on Apple GPUs, so it travels as a raw 64-bit pattern in a ulong and goes through the correctly rounded software binary64 of Kernels/DoubleMath.swift (d_add/d_sub/d_mul/d_div) and the transcendentals of Kernels/DoubleTranscendental.swift. d_add/d_sub/d_mul/d_div and d_sqrt are correctly rounded, so those results are bit identical to Swift's Double — ExprTests.testFloat64ArithmeticIsBitExact asserts add, sub, mul, div, negate, abs and round on bit patterns, and test_expr.py asserts against pyarrow. The transcendentals are bounded to 1-2 ulp, not correctly rounded.
• Pipeline caching. A compiled query is cached on the canonical query text plus the referenced columns' types and nullability, and the pipeline itself on a hash of the generated source (the same process-wide cache every other kernel uses). Running the same shape again costs one dictionary lookup: ExprTests.testPipelineCacheHit asserts the compile counter does not move.
• Batching. A query joins an open MetalContext.batch { } like any other kernel, and a filtered project inside a batch produces pending arrays whose length the GPU decides, so a second query can consume them without a CPU round trip (ExprTests.testChainedQueriesInOneBatch).

Kernel shapes

The filtered project reads its inputs twice on purpose: the output positions are only known after the scan, and re-evaluating the expression for the selected rows is cheaper than materialising every intermediate. The predicate is compiled into the counting pass, so no boolean array is ever built.

Group-by privatises a table of keyCount × aggregates slots in threadgroup memory while that fits in 2048 slots (24 KB of the 32 KB budget); above that it uses device-wide atomics, as Kernels/GroupBy.swift does. Sums are 64-bit through a lo/hi carry pair, because MSL has no 64-bit atomic add.

The grammar

Every query serialises to an s-expression. It is what Expr.description prints, what ExprQuery(text:) parses, what am_query takes over the C ABI, and what Python's .sexpr() produces. include/arrowmetal.h carries the same grammar for C consumers.

query   := "(query" filter? group_by? terminal ")"
filter  := "(filter" expr ")"
group_by:= "(group_by" INT "\"name\"" expr ")"
terminal:= "(project" ("(as \"name\"" expr ")")+ ")"
         | "(aggregate" agg+ ")"
agg     := "(" ("sum"|"min"|"max"|"mean"|"count") "\"name\"" expr? ")"

expr    := "(col \"name\")"
         | "(int" INT ")" | "(float" NUM ")"              -- untyped: adapts to the other operand
         | "(i8"|"i16"|"i32"|"i64"|"u8"|"u16"|"u32"|"u64" INT ")"
         | "(f32"|"f64" NUM ")" | "(bool" true|false ")" | "(str \"...\")" | "(null" TYPE ")"
         | "(" BINOP expr expr ")" | "(" UNOP expr ")"
         | "(cast" expr TYPE ")"
         | "(if_else" expr expr expr ")" | "(coalesce" expr+ ")" | "(fill_null" expr expr ")"
         | "(is_null" expr ")" | "(is_valid" expr ")" | "(is_in" expr literal+ ")"
         | "(str_eq"|"starts_with"|"contains" expr "\"pattern\"" ")"
BINOP   := add sub mul div | eq ne lt le gt ge | and or and_kleene or_kleene
         | bit_and bit_or bit_xor shl shr
UNOP    := negate abs sqrt exp ln round not bit_not signbit is_nan
TYPE    := i8 i16 i32 i64 u8 u16 u32 u64 f32 f64 bool str

Example:

(query (filter (and (eq (col "region") (int 2)) (gt (col "amount") (int 100))))
       (aggregate (sum "total" (col "amount"))))

Checked against pyarrow.compute (test_expr.py), and applied to both operands of every binary operator:

1. float64 with anything numeric → float64.
2. float32 with any integer → float32 (Arrow's rule; int64 + float32 is float32, not float64).
3. Two integers of the same signedness → the wider one.
4. Mixed signedness → a signed type of max(signed width, unsigned width × 2), capped at int64 (so uint8 + int8 → int16, uint32 + int32 → int64, uint64 + int64 → int64).
5. boolean and utf8 have no numeric common type: mixing them is an error naming both types.

An untyped literal (col("x") > 100, (int 100)) takes the type of the other operand when that is numeric, so comparing a float32 column with 100 stays in float32. Pin a literal with Expr.typedInt(100, .int32) / am.lit(100, "int32") when you want a specific width.

It takes the other operand's type only when it fits there; a literal that does not fit widens the pair instead of being truncated to it, because truncating would silently answer a different question. An integer literal outside the column's range promotes as if it carried its own smallest type (int8_column > 200 compares in int16, not against (char)200 == -56; uint8_column == -1 compares in int16 and is false everywhere), and a floating literal against an integer column promotes the pair to float64 under rule 1 (int64_column >= 2.5 really does compare against 2.5). The same rule applies inside if_else, fill_null, coalesce and is_in.

sqrt, exp and ln on an integer column promote to float64, as Arrow's do. round puts halves away from zero (Arrow's half_towards_infinity), matching ArrowMetal's existing round.

Sign bit and NaN

(signbit x) and (is_nan x) are boolean, null where x is null, and read the value's bits as stored:

is_nan is Arrow's is_nan; signbit is NumPy's signbit (Arrow's sign gives 0 for both zeros and NaN for NaN, so it cannot tell -0.0 or a negative NaN apart). Swift: col("x").signBit, col("x").isNaN; Python: am.col("x").signbit(), am.col("x").is_nan(). A float literal keeps its sign in the text form ((f64 -nan), (f64 -0.0)); a NaN literal's payload is not kept.

totalOrder comparisons

The comparison operators are IEEE: a NaN compares false (and ne true), and -0.0 equals +0.0. Arrow's Rust implementation compares floats by IEEE 754 totalOrder instead: -NaN < -inf < … < -0.0 < +0.0 < … < +inf < +NaN. DataFusion 55.1 uses those kernels after rewriting -0.0 to +0.0 on both sides (normalize_float_zero), so for DataFusion the two zeros are equal and the "finite non-zero" row below is the one to use for a zero literal; the -0.0 and +0.0 rows are arrow-rs's order. With signbit and is_nan, a column against a literal c in totalOrder is one expression (x the column, s = (signbit x), n = (is_nan x)):

x != c is the not of the x == c column (for a non-zero c that is (ne x c)). Every entry is null where x is null, so a filter drops those rows as DataFusion does. A NaN literal is not in the table: totalOrder tells NaN payloads apart, and the grammar has no bit-pattern equality. ExprSignBitTests checks every cell against the totalOrder keys of Float32 and Float64 columns holding ±0.0, ±inf, subnormals, NaN of both signs with four payloads each, and nulls; python/tests/test_expr.py checks the same against NumPy.

How nulls are compiled

Integer division by zero yields 0, which is what ArrowMetal's other integer kernels do (pyarrow raises). A shift amount outside [0, width) yields 0.

Numbers

M4 Max, 50M rows, best of 5, in-process against Polars 1.44 (16 threads), pyarrow 25, pandas 3, numpy 2.5. Benchmarks/expr_bench.py. "op-by-op" is the same expression built from ArrowMetal's per-operator kernels, all batched into one command buffer — today's best without fusion.

At 100 000 rows the fixed cost dominates and the GPU is the wrong tool:

Reading the small-input row: the command-buffer round trip is about 60-70 µs measured (The persistent-kernel result), 110-160 µs per call for sum and filter at 1,000 rows in the matrix's latency family, and it is paid once per query however many operators the expression has — that is what fusion buys at the low end. What it cannot buy back is the round trip itself, nor the cost of allocating and first-touching a fresh output buffer (the reason the float64 project costs more than the aggregate, which only writes threadgroup partials). Crossover against Polars for these shapes is around 1M rows.

Where fusion gains most is where the operator count is highest relative to the bytes moved: (b) is 1.5× the batched op-by-op chain, (d) is 1.8×, and (a) is 1.2× — while (c), which is compaction bound rather than operator bound, is a wash. Against the CPU engines in these tables — Polars, pyarrow and pandas — the margin is 4× to 62×, at a sixtieth to a twenty-four-hundredth of the CPU time; against numpy it is 2.6× to 71×.

Limits

• No string outputs. String columns can be read by str_eq / starts_with / contains against a literal pattern; the compiler will not materialise a utf8 output column, and rejects a projection that asks for one by name.
• Only these column types are read: int8–int64, uint8–uint64, float32, float64, boolean, utf8 (offsets + bytes, or utf8_view views read in place: the kernel reads each row through the view accessor of Design). Temporal, decimal, dictionary, list, struct, map and union columns are rejected with an error naming the column and its Arrow format. Extension columns are read through their storage.
• Group-by needs a dense integer key in [0, keyCount), exactly like GroupBy. Hash the keys with GroupByKeys first if they are not dense.
• Group-by aggregate types are limited to the aggregates 32-bit atomics support (MSL has no 64-bit atomic add): sum/mean accept any integer or float32 (not float64 — cast first); min/max accept a 32-bit or narrower integer or float32. Both errors name the aggregate and say what to cast.
• is_in takes a small literal set and expands to a chain of comparisons; it is not a hash lookup, so keep the set to a few dozen values. A null input is false.
• No window functions, no joins, no sorts, no aggregates over strings inside a query.
• A count(expr) of a boolean expression is allowed; sum of a boolean is not (cast it).
• pyarrow.compute.Expression is not accepted. Converting one means walking str(expr), whose rendering is ambiguous for string literals and field names with spaces, or going through Substrait, which is a larger dependency than this feature warrants. Build the expression with am.col instead — the two read almost identically.
• Arrays above 2^32 elements are not supported, as elsewhere in the package.

Correctness

• Tests/ArrowMetalTests/ExprTests.swift — 20 test cases against Swift oracles at 0 / 1 / 33 / 4097 / 1 000 003 rows with nulls: every operator alone, deep mixed trees with shared subtrees, bit-exact float64 arithmetic and comparisons (including ±0, subnormals, infinities and NaN), Kleene predicates, all four terminals, batched vs unbatched, the pipeline-cache hit, the error cases, and the canonical text round trip.
• python/tests/test_expr.py — 40 cases against pyarrow.compute compositions and Polars lazy for the same expressions, including a bit-identical check of the six-operator float64 expression.

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test --filter ExprTests
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test -c release --filter ExprTests
swift build -c release --product ArrowMetalC
PYTHONPATH=python python -m pytest python/tests/test_expr.py -q
PYTHONPATH=python python Benchmarks/expr_bench.py

The query engine

A lazy, optimizing query engine that runs on the GPU: filter, select, group_by, sort, join, join_asof, window functions, unique, explode and concat, planned as a whole and executed in as few Metal command buffers as the sync points below allow — usually one.

import arrowmetal as am

q = (am.scan(table)
       .filter((am.col("amount") > 0) & (am.col("qty") > 5))
       .group_by("region").agg(am.agg.sum("amount", "revenue"), am.agg.count("orders"))
       .sort("revenue", descending=True)
       .limit(10))
q.explain()      # the optimized plan, the way Polars prints one
q.collect()      # -> pyarrow.Table

let df = LazyFrame(PlanSource(name: "sales", batch: batch))
    .filter(col("amount") > 0)
    .groupBy(["region"], [ExprAggregate(.sum, col("amount"), name: "revenue")])
    .sort([SortKey("revenue", descending: true)])
    .limit(10)
let out = try df.collect()

---

Why this layer exists

docs/EXPR.md describes the fused expression compiler: it turns a whole element-wise tree plus one terminal (project, filter+project, aggregate, group-by over dense keys) into one runtime-generated Metal kernel. That is the fast half of a query engine, and it was already here.

What was missing is everything that moves rows around — sorting, joining, grouping by keys that are not already dense, windows, limits — and, more importantly, the layer that decides: which parts of a query become one kernel, which columns are read at all, where a predicate should run. Without it, a caller writing a real query gets one Arrow kernel per operator and a CPU round trip between each.

So the engine is a plan, three layers of decision over it, and one of execution:

Operators

Aggregates are sum, min, max, mean and count, over any expression, whole-table or per group. Window functions are row_number, rank, dense_rank (uint32 columns), lag, lead, cum_sum, rolling_sum / rolling_mean / rolling_min / rolling_max, and any aggregate broadcast over its partition.

The optimizer

Every rule preserves the result's rows and nulls exactly; join_reorder alone permutes row order, and only fires where that order is unobservable. explain() lists the ones that fired.

Two rules are deliberately absent. A filter is not pushed below a limit, because that changes which rows survive. A filter is not pushed below a window, because a window function's value depends on the rows in its partition.

Cardinality comes from PlanStats: row counts flow up from the scans, a filter is assumed to keep a quarter of its rows, a group-by to fold by 16 per key, an inner join to produce min(left, right). Setting stats.useGPUDistinctCounts replaces the per-column guess with a real GroupByKeys pass; it costs a pass over the column, so it is off unless the plan will run for far longer than the estimate.

explain()

LOGICAL PLAN
  LIMIT 10
    SORT BY [total DESC]
      GROUP_BY [region] AGG [(sum "total" (col "amount")), (count "n")]
        FILTER (gt (col "amount") (int 100))
          SCAN sales [region, amount] 2/5 columns, 1000 rows

PHYSICAL PLAN
  HEAD 10
    TOP-K 10 BY total DESC
      HASH-AGGREGATE [region] AGG [(sum "total" (col "amount")), (count "n")] (per-aggregate kernels)
        FUSED-FILTER-PROJECT [region, amount] FILTER (gt (col "amount") (int 100))
          SOURCE sales [region, amount] 2/5 columns, 1000 rows

RULES APPLIED: projection_pruning

Three of the five source columns are never read. The filter and the projection are one kernel. The sort + limit became a top-k selection, which touches each row once instead of running eight radix passes over all of them.

A join, with the predicate pushed into the side it belongs to:

LOGICAL PLAN
  LIMIT 5
    SORT BY [w DESC]
      SELECT [region_name, (mul (col "amount") (col "weight")) AS "w"]
        JOIN INNER ON [region]
          FILTER (gt (col "qty") (int 5))
            SCAN sales [region, amount, qty] 3/5 columns, 1000 rows
          FILTER (gt (col "weight") (float 1.0))
            SCAN dim [region, region_name, weight] 3/3 columns, 200 rows

PHYSICAL PLAN
  HEAD 5
    TOP-K 5 BY w DESC
      FUSED-PROJECT [(mul (col "amount") (col "weight")) AS "w", region_name]
        HASH-JOIN INNER ON [region]
          FUSED-FILTER-PROJECT [region, amount, qty] FILTER (gt (col "qty") (int 5))
            SOURCE sales [region, amount, qty] 3/5 columns, 1000 rows
          MASK-FILTER (gt (col "weight") (float 1.0))
            SOURCE dim [region, region_name, weight] 3/3 columns, 200 rows

RULES APPLIED: predicate_pushdown, projection_pruning

As written, that query was a join of two full tables followed by one and of two predicates. The dim side falls back to MASK-FILTER — one fused kernel for the predicate, then the ordinary compaction kernels — because it has to carry a utf8 column through, and the expression compiler will not materialise a string output.

Fusion planning

The physical planner decides what becomes one kernel. The rule is short, because the expression compiler already defines what it can do:

• An element-wise region — arithmetic, comparisons, null logic, casts, string predicates — is one kernel (FUSED-PROJECT).
• A filter immediately under a select or an aggregate joins that kernel, so the predicate is evaluated inside the same pass that computes the outputs and no boolean column is ever written (FUSED-FILTER-PROJECT, FUSED-AGGREGATE).
• A select output that is a bare column reference is not a kernel at all: the column is carried by reference (CARRY).
• A region stops being fusable the moment an output would be utf8, or a column it must read is temporal, decimal, list, struct, map, union or dictionary encoded. Then the predicate becomes a mask (MASK-FILTER) and the ordinary per-column compaction kernels run — which is what the package did before fusion existed, so nothing is lost, only the extra pass.
• Everything that moves rows is its own node.

Execution

The whole tree runs inside one MetalContext.batch { }, so every kernel goes into one command buffer and the round trip — about 60-70 µs measured (The persistent-kernel result), 110-160 µs per call for sum and filter at 1,000 rows in the matrix's latency family — is paid once for the query rather than once per operator. Where an operator's length is decided by the GPU — a filter — the result is a pending array whose length lives in a device buffer and the next kernel binds that buffer instead of a CPU-known count (docs/DESIGN.md, "Lengths flow on the GPU"), so a filter feeding a projection feeding another filter never returns to the CPU.

Five operators are unavoidable sync points, because the CPU has to know a count before it can size the next dispatch: GroupByKeys (the number of groups), the hash join (the number of pairs), the top-k selection, and explode and slice (which read offsets). flush(reopen: true) commits, waits and reopens the batch at each, so batching resumes immediately afterwards. Intermediate buffers come from and go back to MetalContext.pool, which parks rather than recycles while a batch is open.

Getting the data in

An engine that runs a query in 10 ms can spend longer than that getting to it, and this one did. am.scan(table) handed the plan its columns as pyarrow arrays and collect() imported every one of them into Metal memory, on every run. At 20M rows that was 6 ms of makeBuffer(bytesNoCopy:) and page residency per call — and worse, the allocation and teardown of a fresh 240 MB import each time churned the buffer pool, so a query whose kernels take 10 ms took 25.

So the Python side imports once. am.scan() imports nothing at all; the first collect() imports the columns the plan reads and caches them on the object it scanned, keyed by identity and held by a weak reference, so the cache dies with the table and a second scan of the same table costs nothing. An imported MetalArray owns the Arrow C Data Interface release callback of the array it came from, so it keeps the producer's buffers alive by itself; caching it is safe even after the pyarrow.Array is gone. A chunked column is imported on first use rather than on the way into every query, chunk by chunk through am_import_chunks (each chunk copied straight into the final Metal buffers, with no combine_chunks copy first) for every type that path takes; dictionary and nested columns are combined first.

Which columns get imported is worked out from the plan text before the plan is sent, so a 200-column table scanned for two of them imports two. It is the same question projection_pruning answers in the optimizer, asked one step earlier — the optimizer can only prune columns that are already on the GPU. The Python answer is deliberately conservative: a superset is always safe, and anything it does not recognise (a renaming join, an escaped column name, a hand-built plan node) means "every column". am.scan(table, columns=[...]) names them by hand for the cases it gives up on.

LazyFrame.columns and LazyFrame.warmup() register a prefix of each source, taken before the import, so neither touches the whole table. explain() is the exception: it registers every column and every row, because the n/m columns it prints and the cardinalities that pick a join order are statements about the real table.

A source can also be columns that were never anywhere but the GPU. A _Source takes MetalArrays as well as Arrow arrays, and a MetalArray is used as it is, with no import. That is how the Polars engine runs a Parquet scan (Polars, "Parquet scans"): it reads the columns Polars' projection names with am.read_parquet through the open-file cache (Parquet on the GPU, "The open-file cache"), the decoder writes them straight into Metal shared memory, and the plan scans those arrays. The row groups and pages the reader skipped by statistics never reach the plan; the plan's filter still runs over every row the reader returned, so the answer does not depend on what was skipped.

That leaves the one cost that cannot be cached away: the first collect() in a process generates and compiles MSL for the kernels its plan lowers to; every later run finds them in MetalContext's pipeline cache. q.warmup() runs the plan over 4096 rows per source to move that cost off the first real query.

With the import cached, a warm collect() over a pyarrow.Table does the same work as the same query over columns already resident in Metal memory, plus the Python side around am_plan_run: am.scan, building the operators, plan_json() and materialising the result. python/tests/test_lazy.py holds the pyarrow.Table case within 3 ms of the resident one; per-stage timings are not recorded.

The join matrix

Kernels/Join.swift does inner and left over one int32 or int64 key, returning uint32 index pairs (at most 2^31 - 1 rows per side and 2^31 - 1 pairs). Kernels/JoinExtra.swift does the rest.

Making arbitrary keys joinable. A join needs an equality that is exact, not probabilistic. Rather than hashing a wide or string key into 64 bits and adding a verification pass, the key columns of both sides are concatenated and handed to GroupByKeys — the machine that already turns arbitrary key columns into dense int32 ids with an injective mapping. Two rows get the same id exactly when their keys are equal, so the int32 hash join over those ids is the original join, with no collisions to resolve. It costs one densification pass over left + right rows. The common case that needs none — a single int32 or int64 key on both sides — goes straight to hashJoin.

Nulls never match, in either direction and for every join kind, which is Arrow's, Polars' and SQL's rule. GroupByKeys gives null keys a group of their own, so the ids alone would match them up; the per-side "every key column is valid" bitmap is attached to the id array instead, and hashJoin skips a null key on both the build and the probe side.

Row order. Inner and left keep probe (left) order, and a left row's matches are contiguous. Right keeps right order. Full is left order followed by the unmatched right rows. Semi and anti keep left order and emit each row once. The order of several matches within one probe row is unspecified.

Semi, anti and the right tail of a full outer join all need the same primitive: which rows of one side the join touched. That is one jx_scatter_flag dispatch over the index pairs plus a filter of the row indices — no second hash table.

As-of. The build side is sorted by (partition, key) on the GPU with its null keys dropped, and every probe row does two binary searches — one for its partition's range in the sorted array, one for the key inside it. That is log2(n) dependent loads per probe row and no table at all, which is the whole reason an as-of join does not need a hash. strategy is backward (last key at or before), forward (first at or after) or nearest (ties to backward, as Polars does); tolerance bounds |probe − match|. An as-of join is a left join: every probe row comes out, in its original order, with nulls where nothing matched.

Window functions

Always the same shape, and it is the shape a GPU likes: sort once, work in sorted order, scatter back.

1. lexsort by (partition keys…, order keys…). Every partition is now one contiguous run and inside it the rows are in the window's order — one stable radix sort per key column.
2. Compute the answer as a function of a row's position in that order, its partition's first position, and its tie group's first position. All three are a GroupBy minimum over the row positions plus a take.
3. take the result through the inverse permutation (which is argsort of the permutation), so the output is aligned to the caller's rows.

Ranking, lag and lead fall out of step 2 with no new kernel: the index a row wants is if_else(position - k >= partition_start && position - k <= partition_end, position - k, null), which is one fused expression, and take of a null index gives a null. RANK is the tie group's first position minus the partition's first position, plus one; DENSE_RANK is a running sum of tie-group marks minus its value at the partition's first row.

cum_sum and the four rolling_* functions are the exception: their existing kernels (Kernels/Cumulative.swift, Kernels/Window.swift) are already exactly right per partition, so each partition's contiguous slice is handed to them and the results concatenated. That is one dispatch per partition — the right trade below a few thousand partitions, and capped at 8192 with an error above.

Nulls follow SQL ORDER BY x NULLS LAST, which is what lexsort does in both directions: a null key sorts after every value, and all nulls of one key form one tie group.

Sort key options

Each key of a sort carries two options besides its direction. Both default to the order the engine has always used, so a key that names neither sorts exactly as before.

• "ieee" is Arrow C++'s (and pyarrow's) order: IEEE comparison, -0.0 and +0.0 equal (a tie, kept in input order), and every NaN one value placed next to the nulls in both directions — after the values when the nulls are last, between the nulls and the values when they are first.
• "total" is IEEE 754 totalOrder, the order arrow-rs (total_cmp), DataFusion and Rust use: -NaN < -inf < … < -0.0 < +0.0 < … < +inf < +NaN, NaNs ordered by payload, only identical bits tie, and a descending key is the exact mirror, so +NaN comes first.
• "nan_largest" is Polars' (and NumPy's) order: IEEE comparison with -0.0 and +0.0 equal, and every NaN one value greater than every number, +inf included, in both directions — last ascending, first among the values descending. The null placement is independent of it.

No option adds work to the sort. A null placement is where the stable partition that takes the null rows out of the radix sort puts them; a float order is the key map in front of the radix passes — "total" is the raw bit pattern with the sign-flip transform, a pure function of the value, and "nan_largest" is "ieee"'s map with the descending key the plain complement. A single-key sort + limit stays a top-k selection with any option.

The keys of a window's order_by take the same forms and the same options (a window spec's own nulls and float_order are the defaults of its keys), and so do the keys of the streaming external sort (ExternalSortOperator.Key, am_stream_sort_ex, Stream.sort(null_placement=, float_order=)), whose merge compares by the same rules.

The plan grammar

Expressions already have a text form — the s-expression grammar of docs/EXPR.md, which is the cache key the fused compiler uses — so a plan reuses it verbatim and only spells the operators. JSON is the wrapper, because a plan is a tree of records with optional fields.

{"op": "sort", "by": [["total", true]], "input":
  {"op": "group_by", "keys": [["region", "(col \"region\")"]],
   "aggs": [["sum", "total", "(col \"amount\")"]], "input":
    {"op": "filter", "predicate": "(gt (col \"amount\") (int 100))",
     "input": {"op": "scan", "source": "sales"}}}}

A sort key is [column, descending] as before, [column, descending, {"nulls": …, "float_order": …}], or {"column": …, "descending": …, "nulls": …, "float_order": …}, every field but column optional. The sort-level nulls and float_order are the defaults for keys that do not name their own:

{"op": "limit", "count": 100, "input":
  {"op": "sort", "by": [["price", true, {"nulls": "first", "float_order": "total"}]],
   "input": {"op": "scan", "source": "t"}}}

Over the C ABI the tables are registered once (am_plan_source_create, which takes the am_array handles the caller already holds) and the plan text carries no data, so the same text can be re-run against new sources and am_plan_explain can print a plan without touching the GPU. include/arrowmetal.h carries the same grammar for C consumers.

Polars' MetalEngine (Polars, "Tier 4") writes this grammar from Polars' optimised plans. Which plans it can write, per plan shape, input dtype and null pattern, with the reason for each one it leaves to Polars, is Polars engine capabilities, generated from runs of the engine by python/tests/engine_capabilities.py and checked against a fresh run by test_polars_engine.py; which Polars entry points reach it is POLARS.md, "Collect paths".

Numbers

M4 Max, 50 M rows, best of 5, in process, against Polars 1.44.1 lazy (16 threads) and DuckDB 1.5.5 (16 threads) on the same Arrow buffers. Benchmarks/engine_bench.py; recorded in Benchmarks/results/engine_bench_50000000_2026-09-07.txt. "CPU ms" is process CPU time over the same run: it is what the query cost the machine, next to what it cost the caller in latency.

Reading it: the widest margins are where the operator count is high relative to the bytes moved — (d) is six operators over three columns and 42x, (a) is a filter plus a reduction and 7.1x — and where the work is a scan the GPU does in one pass, like (h)'s binary search. The narrowest is (f), where Polars is ahead — 3.06 ms against ArrowMetal's 3.50 — because the case is bound by writing 10M output rows rather than by compute; (b) at 2.4x is the next narrowest.

The CPU column is the other half of the story: ArrowMetal's queries cost the machine 0.6 to 5.2 ms of CPU where Polars spends 24.9 to 2305.4 ms and DuckDB 7.1 to 1727.4 ms across 16 threads. Case (c) at 3.5 CPU-ms instead of Polars' 2.3 CPU-seconds leaves the cores free for whatever else the process is doing.

Below about 1M rows the fixed cost dominates and the GPU is the wrong tool; at 2M rows on the same machine (Benchmarks/results/engine_bench_2000000_2026-09-07.txt) (a) is 0.57 ms against Polars' 0.63 ms and (f) is 2.52 ms against Polars' 1.23 ms; at that size (a), (c), (d), (e), (g) and (h) are ahead of Polars and (b) and (f) are behind, and (f) is behind at 50M as well.

Polars' lf.collect(engine=am.MetalEngine()) runs this engine under Polars: it translates subtrees of Polars' optimised plan into the grammar above and, by default, runs a subtree here only when its input rows are at or above the measured crossover of every shape class in it (sort, each join kind, unique, group-by and whole-frame aggregates per aggregate family, row-wise), per dtype class and input, fitted from Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv against Polars' own engines. A group-by is judged at the bucket of its number of groups (200, 1,000, 10,000, 100,000, 1,000,000, or at least a quarter of the rows, a bucket never taken), which a fixed-seed sample of its key columns estimates at plan time (bias-corrected Chao1, 512 sampled rows and up until the decision is settled). A shape counts as ahead at a size when its time, raised by 15% (35% for a shape with a String column), is at most the faster Polars engine's; the default takes it from 1.5 times that fitted crossover, and a shape with a String column from 5,000,000 rows at the earliest. In Benchmarks/results/polars_engine_bench_2026-10-02.csv (2,000,000 and 50,000,000 rows, best of 5) the default takes 75 of 220 case-size pairs, 55 of them group-bys, all ahead of the faster Polars engine, 1.52x to 11.54x, with every result equal to Polars' (Polars, "Which translatable subtrees it runs: the defaults").

Limits

• Strings are read, not written. The fused kernels will not materialise a utf8 output column, so a projection that computes one is rejected; a string column can be carried through a plan, filtered, sorted by (through GroupByKeys), joined on and grouped by, but not transformed inside a query. Use the per-operator string kernels for that.
• Temporal, decimal, list, struct, map, union and dictionary columns can be carried, filtered, joined on, grouped by and sorted, but not read by a fused expression. A predicate over one of them is an error naming the column and its Arrow format.
• min and max in a whole-table aggregate come back at the input's width but through a 64-bit scalar, so a uint64 maximum above Int64.max is not representable.
• Group-by aggregate types. The one-kernel group-by path is limited to the aggregates 32-bit atomics support (MSL has no 64-bit atomic add): sum/mean over any integer or float32, min/max over a 32-bit-or-narrower integer or float32. Anything else — a float64 sum, a 64-bit min — silently takes GroupBy's own per-aggregate kernels instead, which explain() says. There a Float64 sum is correctly rounded (the binary64 value nearest the exact sum of the group's non-null values, ties to even) and a Float64 mean is the exact sum divided by the count, rounded once; both come from two passes over the rows with no group order (Design). A sum and a mean, or a min and a max, of the same column in one group_by share one pair of passes.
• count(expr) is the number of non-null values of expr for a column of any type, on both paths and in a whole-table aggregate. A count never decides the path: a count of an expression the fused kernel does not read as a number (utf8, boolean, temporal, decimal, nested, dictionary, ...) is taken from the column's validity bitmap next to the kernel.
• Group order is not Polars'. GroupByKeys emits groups ascending by key for numeric, boolean, temporal and decimal keys and in first-seen order for utf8; pyarrow and Polars use first-seen for everything. Sort both sides before comparing.
• join_asof keys must be integer or temporal. Float and string as-of keys are not supported.
• explode takes one list column at a time, and builds its gather index on the CPU, so it is a sync point.
• cum_sum and the rolling window functions are capped at 8192 partitions, because they run one dispatch per partition.
• Concatenation is a memcpy, not a kernel: the buffers are unified memory, so appending is a memcpy at ~100 GB/s with no transfer, and concatenation is always a materialising boundary anyway.
• No SQL front end, no LIMIT push into a join, no spilling. Every input must fit in the working set; there is no partitioned or out-of-core execution.
• Arrays above 2^31 rows are not supported by the join, and 2^32 elsewhere, as in the rest of the package.

Correctness

• Tests/ArrowMetalTests/EngineTests.swift — 34 tests: schema inference and type checking, every optimizer rule asserted on explain(), every operator against a Swift oracle at 0 / 1 / 33 / 4097 / 1 000 003 rows with nulls, all six join kinds against a Swift oracle over an integer key at four sizes with null keys on both sides, multi-column and utf8 keys, the as-of join in all three strategies with partitions and a tolerance and against an oracle at 4097 × 1301 rows, the window functions, concat, explode, batched execution, and optimized-vs-unoptimized equivalence.
• python/tests/test_lazy.py — 56 queries against Polars lazy, pyarrow and DuckDB on the same data, including TPC-H shapes (filter + group-by + sort + limit; join + aggregate; window over partition; as-of join on timestamps), plus what the Python side costs: that a column is imported once however often it is scanned, that a query importing two of five columns imports two and one that can read any imports all, that a schema check and a warm-up import none, that rescanning a changed dict or a mutated Polars frame sees the change, and that a warm collect() over a pyarrow.Table stays within 3 ms of the same query over GPU-resident arrays.

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test --filter EngineTests
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test -c release --filter EngineTests
swift build -c release --product ArrowMetalC
PYTHONPATH=python python -m pytest python/tests/test_lazy.py -q
PYTHONPATH=python python Benchmarks/engine_bench.py

Parquet on the GPU

ArrowMetal reads Apache Parquet with Metal compute kernels: from file bytes to Arrow arrays in shared memory. The CPU reads column data only to decompress it: the ZSTD, GZIP and BROTLI pages, and the Snappy and LZ4 pages it decodes faster than the GPU (token-dense pages; the GPU takes the literal ones), all straight into the shared buffer the GPU decoders read, while the GPU decompresses its own share. Definition levels, dictionary indices, every value encoding, the delta encodings and BYTE_STREAM_SPLIT are all kernels. The host parses the Thrift footer and the per-page Thrift headers — metadata, not data — and everything after decompression runs on the GPU.

cuDF does this for NVIDIA.

• Sources/ArrowMetal/Parquet/Thrift.swift — a hand-rolled Thrift compact-protocol reader and writer, in the spirit of the FlatBuffers reader in IPC/FlatBuffers.swift: no code generator, no runtime, just the handful of structs the format defines.
• Sources/ArrowMetal/Parquet/ParquetMetadata.swift — the subset of parquet.thrift this needs.
• Sources/ArrowMetal/Parquet/ParquetFile.swift — the mapped file and the schema tree.
• Sources/ArrowMetal/Parquet/ParquetReader.swift, ParquetColumnDecode.swift, ParquetValueDecode.swift, ParquetTypeMap.swift, ParquetList.swift — the read pipeline.
• Sources/ArrowMetal/Parquet/ParquetNested.swift — structs, maps and lists at any depth, from their leaves.
• Sources/ArrowMetal/Parquet/ParquetArrowSchema.swift — the stored ARROW:schema and what it restores.
• Sources/ArrowMetal/Parquet/ParquetPageIndex.swift, ParquetBloomFilter.swift — page-level skipping with the column and offset indexes, and row-group skipping with bloom filters.
• Sources/ArrowMetal/Kernels/DecompressSource.swift / Decompress.swift / DecodeSplit.swift — Snappy and LZ4 on the GPU and on the host, and the router that splits a read's pages between them.
• Sources/ArrowMetal/Kernels/ParquetDecodeSource.swift — every decoding kernel.
• Sources/ArrowMetal/Parquet/ParquetWriter.swift — a small host-side writer, for round trips.

The pipeline

For one leaf column, across every selected row group at once:

  page headers (host, Thrift only, in parallel across the read's column chunks)
    -> decompression       SNAPPY / LZ4 / LZ4_RAW split page by page between the host and the GPU,
                           both at once; host for ZSTD / GZIP / BROTLI; nothing for UNCOMPRESSED
    -> pq_page_layout      finds each page's level and value sections *inside* the page
    -> pq_decode_levels    definition levels -> one byte per row, plus each row's rank
    -> pq_page_scan        per-page offsets into the chunk's dense value section
    -> one kernel per encoding, each writing dense values
    -> pq_scatter          dense values -> row positions (skipped when the column has no nulls)
    -> pq_levels_to_bitmap definition levels -> an Arrow validity bitmap

The file's bytes are the GPU's bytes

ParquetFile mmaps the whole file, read-only and shared (PROT_READ, MAP_SHARED). mmap always returns a page-aligned address, so any range of the mapping can be wrapped with makeBuffer(bytesNoCopy:) — the same trick MetalArrowBuffer.wrapOrCopy uses for zero-copy Arrow import. A column chunk's pages are then addressable by a kernel as byte offsets into that buffer. An uncompressed file needs no staging buffer at all: the mapped file is the page buffer the decoders read, so the values go from page cache to Arrow array without the host ever loading them. Every kernel binds the page bytes device const; nothing writes to the mapping.

Creating the wrap costs next to nothing (under 0.1 ms for 2 GB). The first kernel that reads a wrapped range makes all of it resident for the GPU, and that costs time in proportion to the range: 7-10 ms per gigabyte for this read-only shared mapping on an M4 Max, against 67-78 ms per gigabyte for a private writable one, which the reader keeps only as the fallback for a device that will not wrap read-only memory (and on virtualised GPUs). So what a read wraps matters:

• A read whose columns fill most of the bytes they span (at least four fifths) wraps that span once, up front. Column chunks are interleaved by row group, so in a file with fifty row groups one column's chunks already span nearly the whole file, and wrapping per column would map the same pages once per column.
• A sparse projection — one column of a wide file — maps its column's chunks side by side into one reserved range of address space (MAP_FIXED over each chunk, MappedView in ParquetFile.swift) and wraps only them, so the GPU makes resident the bytes it reads and not the other columns' chunks in between.

Wraps and views are cached on the handle, and so are the parsed page headers, so holding the ParquetFile across queries still saves time; see the benchmarks. Closing a file unmaps it on a background queue.

Pages are addressed with 32-bit offsets from their binding point, so file size is not a limit: a column whose chunks span more than 4 GiB of a larger file reads through a view. A column whose chunks in one read total 4 GiB or more raises ParquetError.unsupported.

ARROWMETAL_PARQUET_PROFILE=1 prints, for every open and read, the wall time and minor page faults of each phase to stderr (waiting for the GPU at each phase boundary, so a phase's kernels are charged to it).

Page layout is computed on the GPU, not the host

A v1 data page stores <4-byte length><repetition levels><4-byte length><definition levels><values>. Those two lengths live inside the page, which for a compressed column means they only exist after decompression. pq_page_layout therefore runs as a kernel: one thread per page, filling in the byte offsets of the three sections and, for a dictionary page, the index bit width stored in its first byte. A v2 page carries the two lengths in its Thrift header, so the host fills those in and the kernel only does the arithmetic.

The parallel RLE strategy

Definition levels, repetition levels and dictionary indices all use Parquet's RLE / bit-packing hybrid: a sequence of runs, each introduced by a varint header whose low bit picks the run kind (0 = a repeated value, 1 = a group of 8·n bit-packed values). Run boundaries are only known after the previous header has been read, so the headers cannot be parsed in parallel. The values inside them can, and that is where all the work is.

One threadgroup (256 threads) owns one page and walks it in batches of up to 2048 values:

1. Thread 0 scans forward over run headers only. It never touches the packed value bytes — for a bit-packed run it computes the run's byte length from the width and skips it — until it has covered 2048 values or filled its table of 256 run records. A record is (kind, value or byte offset of the packed data, values of this run already emitted, where this run starts inside the batch). A run that spills past the batch is left half-consumed and resumed by the next batch, so runs of any length work and a page of any size works.
2. A threadgroup barrier publishes the table.
3. Every thread takes a contiguous slice of 8 batch positions and binary-searches the run table (256 entries, threadgroup memory, a few cycles) for the run each position falls in, then decodes its level: a constant for an RLE run, an LSB-first bit-field read for a bit-packed one. All of the bit unpacking happens here, 256-wide.
4. The same pass scans "is this level the max definition level?" across the batch — 8 values serially per thread, then simd_prefix_exclusive_sum across the 32 lanes of each SIMD group, then a 256-wide combine. That gives every row its rank: the index of its value in the page's dense, null-free value section.

Ranks are the load-bearing idea. Because every row knows where its value sits in the dense section, every value decoder can be a pure gather, no decoder has to think about nulls, and a column with no nulls skips the scatter entirely because dense positions and row positions coincide.

A page therefore costs one serial header walk of a few instructions per run, and everything else runs 256-wide while thousands of pages run at once across the GPU.

Dense values, then one scatter

Every value decoder writes its page's values densely: value j of a page lands at nonNullOffset + j. One pq_scatter at the end moves the dense values into their row positions using the ranks. Keeping the two apart means each encoding needs exactly one kernel, and a chunk whose pages mix encodings — which is what a writer produces when a dictionary outgrows its budget and the remaining pages fall back to PLAIN — just runs two kernels over two slices of the page list into the same output buffers.

Variable-length values never get copied twice. Each encoding only fills, for every dense value slot, where its bytes are and how many, always as offsets into the page buffer (or, for DELTA_BYTE_ARRAY, into a dense staging buffer built by its own kernel). One prefix sum over per-row lengths produces the Arrow offsets buffer, and one gather moves every byte of the column in parallel.

Several row groups, one dictionary

A dictionary column is decoded across every selected row group in a single pass even though each row group has its own dictionary page: the dictionaries are concatenated, and pq_decode_rle_values adds each page's dictionary base to its codes as it writes them. The result is one Arrow dictionary array over a merged (not deduplicated) dictionary, which is exactly what Arrow allows.

Decompression

Snappy and LZ4 are byte-oriented LZ77: a stream of tokens, each either "copy N literal bytes from the input" or "copy N bytes from N' bytes back in the output". A single block cannot be parsed in parallel — but a Parquet file has thousands of pages, each an independent block, and that is the parallelism.

However a page is decoded, one worker walks its token stream from start to end. The GPU has many slow workers for that and the host a few fast ones, so a read splits its Snappy and LZ4 pages between the two, page by page, and runs both at once.

What a page costs on each side. The columns of the 50 M-row benchmark files, 2,525 pages of about 160 KB of output each (int64: id sequential, qty below 1,000; ts timestamps; price random doubles), host decoders on one core and on all 16, GPU kernels over the column's pages in one dispatch:

A token-dense page (id, qty, ts) is tens of thousands of tokens of a few bytes. On the GPU its cost is the time one worker takes to walk it: 12-15 ms on a SIMD group and 37-60 ms on a thread, the same for 32 pages in a dispatch as for 512, and a dispatch lasts as long as its slowest page. The 12,625 pages of a file that compress 1.25x or more (these three columns and two of long runs) take the SIMD- group kernel 159-190 ms and the page-per-thread kernel 51-65 ms, and 16 host threads 44 ms (Snappy) and 33 ms (LZ4) reading a freshly opened file. A literal page (price) is the other way round: it is one long copy, 1.2 GB of such pages take the GPU 10 ms, and the host 19 ms from a fresh mapping (the first touch of each 16 KB page of a mapping is a minor fault; 5.6 ms once touched). A column of literal pages split between the two read in 19.6 ms, against 12.1 ms all on the GPU and 13.7 ms all on the host: both sides copy at memory speed.

The router (DecodeRouter in Kernels/DecodeSplit.swift) sees every Snappy and LZ4 page of a read — all the columns staged together — before any is decoded, with only its page header: compressed and uncompressed sizes, hence its ratio. A page at or below 1.0 is literal; above 1.05 it is token-dense (an LCG-generated int64 column compresses only 1.05x with Snappy yet is short tokens throughout, 32-35 ms on the GPU against 8-17 ms on the host for 160 MB); far above 4 it is a few long runs again. Each class has a cost per output byte on each side (DecodeCost), from the measurements above:

The pages go in order, densest first; the host takes a prefix of them and the GPU the rest, and the router picks the prefix whose later finishing time is earliest: the host's pages spread over its cores (12 performance cores and a quarter of each efficiency core, less a tenth; 16 threads decoded token- dense pages 12.4 times as fast as one), never less than its largest page, against the GPU's slowest page plus what every other page adds (with the page-per-thread kernel for the token-dense part when that is predicted faster). ZSTD, GZIP and BROTLI pages are host-only and count against the host's side; literal pages cost the host more than the GPU and stay on the GPU. On the 50 M-row files the host takes 6,924 of the 20,250 Snappy pages and 7,102 of the LZ4 ones — the pages of id, qty and ts but for 470-650 — and the GPU the rest; on the 10 M-row, 7-column file (pyarrow's defaults) the host takes its 1,338 token-dense pages, 790 KB dictionary pages among them (97 ms each on a SIMD group), and the GPU its 2,274 literal ones.

Both at once. The GPU's pages go out as one dispatch per codec and kernel, slowest pages first, and the command buffer is committed before the host starts; the host then decodes its pages on every core (DispatchQueue.concurrentPerform, most expensive first) while the GPU runs, and the read waits once for both before any value kernel reads a page. The two sides never write the same memory: a column's staging layout puts the GPU's pages first and the host's from the next page boundary, so no page of memory holds bytes of both. A header that does not parse or a page that fails to decompress, on either side, drops the batch, and each column then stages its own pages (routed the same way, column by column), so errors are raised exactly as a column-by-column read raises them.

The result, whole-file reads through a fresh open, best of 5, phases from ARROWMETAL_PARQUET_PROFILE (the previous build decompressed every Snappy and LZ4 page on the GPU but for dictionary pages and chunks of a few pages):

In all three the GPU finishes its share before the host finishes its own. One column of token-dense pages read on its own with the handle kept went from 49.7 to 15.2 ms (id, Snappy), 53.5 to 18.1 (qty), 44.0 to 15.3 (ts), and 34.3 to 11.2, 48.0 to 12.2 and 39.4 to 10.9 with LZ4; literal columns read as before (price 7.0 and 6.8 ms). The host's share costs CPU time: a whole-file read takes 674 (Snappy) and 508 (LZ4) CPU-ms, against 80 and 78 for the GPU-only build and against Polars' 1,283 and 1,007 and pyarrow's 1,164 and 987 in the same run (Benchmarks, below).

The host decoders. SnappyHost and LZ4Host (Decompress.swift) check every read against the page's input and every write against its output slot, so a damaged page is an error, never an access outside either; LZ4Host makes the GPU kernels' checks in their order, with their outcomes, and recognises the Hadoop framing the same way. Back-references move 8 bytes at a time (a pattern shorter than 8 bytes first repeats to its first multiple of 8) and short literals 16 at a time, only where the whole step lies inside the page and the slot. ZSTD, GZIP and BROTLI are decompressed on the host too, straight into the shared-memory buffer the GPU decoders read. ZSTD pages go out in runs of up to eight, each decoded with one ZSTD_DCtx: ZSTD_decompress allocates and frees a context (about 160 KB) on every call, which for a column of thousands of small pages cost more than the decompression. GZIP and BROTLI go through Foundation's Compression framework (COMPRESSION_ZLIB is raw DEFLATE, so the gzip container is stripped first). The macOS SDK has no COMPRESSION_ZSTD, so libzstd is looked up with dlopen at first use — $ARROWMETAL_ZSTD, then /opt/homebrew/lib, /usr/local/lib, /usr/lib. When it is not installed a ZSTD column raises ParquetError.unsupported naming the missing library rather than returning wrong data; that is the documented host fallback.

The GPU decoders. One SIMD group (32 lanes) owns one page. Lane 0 walks the token stream and broadcasts each parsed token with simd_broadcast; all 32 lanes then move that token's bytes. A 256-thread threadgroup per page was tried and is worse — a threadgroup barrier per token costs more than the extra lanes are worth, because real pages have tens of thousands of small tokens rather than a few large ones. Two details do help:

• The token stream is staged in threadgroup memory. Every tag byte lane 0 reads is a dependent device-memory load, and one of those costs hundreds of cycles. All 32 lanes cooperatively stage the next 8 KB of the compressed stream into threadgroup memory and the parse reads from there; payload bytes are still read straight from device memory, where the reads are wide and coalesced and off the dependency chain.
• Overlapping back-references are still copied in parallel. When offset < length (which is how both formats encode runs) the copy repeats a pattern of offset bytes, so output byte i is out[dst - offset + (i % offset)] and every one of those source bytes was written before this token began. simdgroup_barrier(mem_flags::mem_device) before a back-reference orders the stores; a literal token reads only the input and skips the barrier.

snappy_decompress_lane and lz4_decompress_lane give each thread its own page instead: it reads the token stream 16 aligned bytes at a time into registers and keeps the last 256 bytes it wrote in its own slice of threadgroup memory, so the back-references of token-dense pages (a value or two back) are served from there rather than from device memory. They check the same things in the same order and return the same status codes.

Per-block status codes are written back so corrupt input becomes an error instead of silently wrong data. A page that decompresses to nonsense is a separate question: the value decoders clamp what they read to the page they were given (a BYTE_ARRAY length field, for instance), so damaged bytes produce wrong values but never an out-of-bounds access and never an unbounded loop.

ARROWMETAL_PARQUET_DECODE=host, gpu or lane sends every Snappy and LZ4 page of a process to one decoder; the tests read every fixture, and 240 damaged files, on each (Correctness, below).

Supported matrix

Encodings

Codecs

Types

Nesting

Structs, maps and lists nest to any depth — list<list<T>>, list<struct<...>>, struct<list<...>>, map<K, list<V>>, list<map<K, V>> — with nulls and empty lists at every level, in the three-level (group (LIST) { repeated group list { element } }) and two-level (group (LIST) { repeated element }) list shapes. Every Arrow-level field carries three numbers from the schema: the definition level at which it is present, the largest repetition level that starts a new element of it, and the definition level of its nearest repeated ancestor. Over the level entries of any one leaf beneath the field, an entry is one slot of the field when its repetition level is at most the first and its definition level reaches the third, and the slot is non-null when the definition level reaches the second. So:

• a leaf is its decoded array compacted to its own slots with the package's existing filter;
• a struct is its children plus a validity bitmap, read off one leaf's entries at the struct's slots;
• a list is its child plus offsets: at each of the list's slots, the number of child slots before it, which is a prefix sum of the child's slot flags read at the list's own slots;
• a map is a list whose child is the key/value entries struct.

Each of those is one flag kernel over the entries (pq_nest_flags), one prefix sum and one scatter (pq_nest_scatter), and every leaf is decoded once however many fields read its levels (ParquetNested.swift). A one-level list<primitive> keeps the dedicated kernel pair it had before, which reads the same levels the same way. A struct's leaves still read on their own by dotted path: am.read_parquet(path, columns=["addr.city"]) returns the city leaf as a flat column, null wherever addr or city is null.

The stored Arrow schema

Parquet has no time zone (only isAdjustedToUTC), no duration, no decimal32 / decimal64, no fixed-size list, no dictionary type, no extension types and no per-field metadata. Arrow writers — pyarrow, Arrow C++, Polars, arrow-rs — therefore store the original Arrow schema in the file's key/value metadata under ARROW:schema (base64 of an IPC Schema message), and the reader decodes it with the IPC FlatBuffers views in IPC/FlatBuffers.swift and puts back what the Parquet schema lost (ParquetArrowSchema.swift):

Time zones and durations are restored inside structs, lists and maps too. The file's other key/value metadata is the schema metadata (arrowSchemaMetadata, am_parquet_schema_metadata, f.schema_metadata), and a Parquet field_id shows up as PARQUET:field_id, as it does in pyarrow. The null type needs no stored schema: Parquet annotates a null column UNKNOWN, which reads as null, at the top level and inside lists, maps and structs (list<null>, map<string, null>, list<struct<a: null, ...>>). The stored fields are matched to the Parquet columns by position, as Arrow's reader matches them; a stored schema with a different number of top-level fields is ignored, and its ARROW:schema key stays in the schema metadata, as pyarrow keeps it. A stored type the column cannot take (a dictionary claim over a struct, say) is ignored for that column. A file without ARROW:schema reads exactly as its Parquet schema describes it. A file whose ARROW:schema is not base64, or does not decode as a Schema message, reads the same way, with the undecodable value left in the schema metadata; pyarrow refuses to open such a file (Invalid base64 input, Corrupted metadata length), and test_parquet_nested.py checks both sides. A stored field nested more than 64 levels deep is read down to 64 levels and taken as a type no column matches, so that column reads as its Parquet schema says and the other stored fields still apply, as in pyarrow (test_a_stored_field_nested_past_64_levels_is_ignored_like_pyarrow). A leaf read on its own by dotted path always reads as the Parquet schema describes it.

Projection and predicate pushdown

let f = try ParquetFile(path: "trades.parquet")
let batch = try f.read(ParquetReadOptions(
    columns: ["price", "qty"],
    rowGroups: [0, 1],
    filters: [ParquetFilter(column: "price", op: .gt, value: .int(100))]))

cols = am.read_parquet("trades.parquet", columns=["price", "qty"],
                       filters=[("price", ">", 100)])
# already on the GPU, dictionary-encoded or not
total = cols["price"].sum()

# Across several queries, keep the handle: mapping the file is a per-open cost.
f = am.ParquetFile("trades.parquet")
for day in days:
    px = f.read(columns=["price"], filters=[("day", "==", day)])["price"]

# Or let the open-file cache hold it (below).
px = am.read_parquet("trades.parquet", columns=["price"], cache=True)["price"]

Only the requested column chunks are ever touched — the other columns' pages are never even faulted in. filters is evaluated against the footer's min_value / max_value statistics per row group; a row group whose range cannot contain a match is skipped without reading a page. When the file also has a column index and an offset index, the same filters then skip pages inside the row groups that remain (next section). Either way the result is a superset of the matching rows: follow it with filter (or a fused am.query) to get exactly the rows. A filter value is a string, a boolean, an integer or a float; a date or timestamp column is filtered by its stored integer (days since the epoch, or ticks in the column's unit), and Python raises on any other value (a datetime.date, a Decimal). A literal of another kind than the column's (a string against a number, in the C and Swift filter text) never rules a row group or page out. A string value may hold any character: Python quotes it and escapes " and \, so a ;, a quote or an operator inside it is part of the value; a column name cannot hold =, !, <, > or ;, and Python raises on one that does. An integer above the int64 range stays exact and is compared as unsigned against a uint64 column's statistics, which are read as unsigned (test_uint64_literals_past_the_signed_range, test_string_literals_with_quotes_semicolons_and_backslashes). String and binary statistics are compared the way Parquet orders them, by unsigned bytes: the min / max stay raw bytes and a string literal is compared as its UTF-8 bytes, so a composed and a decomposed accent, Z before a before é, and an emoji above every BMP character are all ordered as pyarrow orders them, row groups and page index alike (test_string_statistics_are_ordered_by_bytes). An integer statistic and a float literal are compared exactly, never through a double: 2^53 + 1 is above the literal 2^53 (test_int64_statistics_against_a_double_literal_near_2_pow_53). A decimal column's statistics hold its unscaled integer, so they never rule a row group or page out (test_decimal_statistics_stored_as_integers_are_not_compared_unscaled). On a float or double column != never rules a row group or page out: writers leave NaN out of min / max, so a range of one value equal to the literal can still hold a NaN, which != keeps. pyarrow's read_table(filters=...) rules out a row group whose min and max both equal the literal, and so leaves out that group's NaN rows; ArrowMetal returns them (test_not_equal_keeps_a_nan_row_group_where_pyarrow_drops_it). ParquetFile.selectedRowGroups(_:) / ParquetFile.selected_row_groups(...) report what the row-group statistics keep without reading anything, and ParquetFile.column_null_count(column) (am_parquet_column_null_count) a top-level column's null count: 0 for a required column, else the sum of the row groups' null_count statistics, or None when a row group does not record it.

Page-level skipping

A writer may store, after the row groups, a column index (each data page's min, max and whether it holds only nulls) and an offset index (where each data page starts and its first row). pyarrow writes them with write_page_index=True; Polars writes them by default. With them a filter narrows each row group to candidate row ranges (ParquetPageIndex.swift):

1. For each filter on a flat column, a page whose [min, max] cannot satisfy it, or that holds only nulls, rules out its rows. The ranges the filters leave are intersected; a row group left with none is not read at all, even when its row-group statistics let it through. A page counts as all-null only when the index's null count for it covers every row: Polars (1.44) flags each page that holds a NaN as a null page with a null count of 0, and such a page is kept whole. A NaN min or max rules nothing out, as the format asks. Polars also leaves the flagged pages out of the row group's min / max, so when a chunk's column index shows such a page, that column's row-group statistics do not drop the row group; pyarrow.parquet.read_table(filters=...) trusts them and returns no rows for a filter outside the unflagged pages' range, where ArrowMetal returns every match (test_polars_row_group_statistics_leave_nan_pages_out).
2. Every flat column with an offset index decodes only the data pages that overlap a candidate range. The page list comes from the offset index, so a skipped page is never decompressed, never decoded, and its header is never read.
3. Pages have different boundaries in different columns, so every column — nested ones and those without an offset index included, which decode their row groups whole — is trimmed to exactly the candidate rows with one filter. All columns of the result cover the same rows.

The rows that match a filter are the same with and without the index; the index only makes the superset smaller. ParquetFile.usePageIndex (f.use_page_index in Python, am_parquet_set_page_index in C) turns it off, and lastReadStatistics (f.last_read_stats, am_parquet_last_read_stats) reports what the last read did: row groups read and skipped by statistics, by the page index and by bloom filters (below), data pages decoded and skipped, rows returned. test_parquet_nested.py compares the exact matches with and without the index against pyarrow over fifteen filter sets on five files (pyarrow in three page layouts and once without an index, and Polars), and six filter sets on a float column with a NaN every 97th row, written by Polars and by pyarrow, and != on pages of 64 rows where a NaN hides in a page whose min and max both equal the literal; ParquetPageIndexTests also checks that the skipped and decoded pages of flat columns add up to the pages of the row groups read. A filter on a column inside a list does not narrow pages; the row-group statistics still apply to it.

Bloom filters

A writer may also store a split-block bloom filter per column chunk (pyarrow with bloom_filter_options, DuckDB for its dictionary-encoded columns). An == filter hashes its literal the way the writer hashed the column's values — xxHash64 of the PLAIN encoding — and a row group whose filter does not have all 8 bits of that hash set is dropped before any page is read, even when its min/max statistics let the value through (ParquetBloomFilter.swift). A set of bits is only a maybe, so the row group is then read as usual; a bloom filter never drops a row group that holds the value. Only literals with one exact encoding are looked up — integers stored as INT32 / INT64, strings and binaries, and floating-point values exactly representable in the column's type other than zero and NaN (which have two encodings each) — and anything else keeps the row group. ParquetFile.useBloomFilters (f.use_bloom_filters, am_parquet_set_bloom_filters) turns it off, and the read statistics count the row groups it dropped. ParquetBloomFilterTests checks xxHash64 against its published vectors and every value of the fixture against its row group's filter; test_parquet_nested.py looks up values in seven columns of a pyarrow file and one of a DuckDB file and requires pyarrow's exact matches with the filters on and off.

The open-file cache

Opening a file maps it, and the first read of a column hands the mapped pages it covers to Metal; a ParquetFile keeps both for as long as it lives (above, "The file's bytes are the GPU's bytes"). A caller that cannot hold the handle itself — one read_parquet call per query, or the Polars engine, which sees a file path in every plan — reads through a process-wide cache of open files instead:

cols = am.read_parquet("trades.parquet", columns=["price"], cache=True)   # opens and maps
cols = am.read_parquet("trades.parquet", columns=["price"], cache=True)   # the same open file

am.parquet_cache_info()     # {"entries", "bytes", "max_entries", "max_bytes", "hits", "misses",
                            #  "invalidations", "files"}
am.parquet_cache_limit(max_entries=4, max_bytes=8 << 30)
am.clear_parquet_cache()

• Keyed by the file's identity and state: its real path, device, inode, modification time in nanoseconds and size. A file rewritten in place (new size or new modification time) or replaced by another under the same name (new inode) is opened afresh on its next read, and its stale entry is dropped then (invalidations counts those).
• Bounded: least recently used entries are evicted above max_entries (16 by default) or above max_bytes, the sum of the cached files' sizes (a quarter of physical memory by default). A file larger than max_bytes is read without being cached; max_entries=0 turns the cache off.
• Shared: reads of one cached file are serialised by a lock per file, so last_read_stats belongs to the read that took it. Evicting or clearing an entry closes the file once the last read holding it returns.
• read_parquet without cache=True opens and closes the file on every call, as before.

python/tests/test_polars_engine.py checks reuse, invalidation on a new size, a new modification time with the same size, a touched file and a replaced file, LRU order, both bounds and clear.

Cold is the first read in the process (the cache cleared before each run; the file itself stays in the OS page cache), warm is the next read of the same file through the cache. 50,000,000 rows x 8 columns, the files of Benchmarks/parquet_bench.py, best of 5, Benchmarks/results/parquet_cache_2026-09-26-split.csv (run conditions in Benchmarks/results/bench_conditions_2026-09-26-split.txt):

Opening the file (the footer and the mapping) is 0.2-0.3 ms. A cold one-column read is 7.2-12.3 ms against 2.7-7.0 ms warm: the cold read also parses the column's page headers, a minor fault each on a fresh mapping, and makes the column's own chunks resident for the GPU (above, "The file's bytes are the GPU's bytes"). A cold read of all 8 columns is 38.3-66.7 ms, 16.0-53.9 ms warm.

PYTHONPATH=python python Benchmarks/parquet_bench.py --rows 50000000 --codecs snappy,lz4,none \
    --cache --skip-main --keep --repeat 5 --cache-out parquet_cache.csv

Benchmarks

Measured on an Apple M4 Max (Mac16,6, 64 GB), macOS 26.x, release build. 50,000,000 rows x 8 columns (int64, int64, int32, float64, float64, dictionary-encoded string, timestamp[us], bool), 1 MB data pages. Best of 5 in-process runs, caches warm. The tables below are from Benchmarks/results/parquet_bench_2026-09-26-split.txt (run conditions in Benchmarks/results/bench_conditions_2026-09-26-split.txt).

wall ms is elapsed time for the whole read; CPU ms is process CPU time over the same interval, so GPU work does not appear in it and ArrowMetal's host share of the decompression does; ttfc is time to first compute — read one float64 column and sum it, which is the smallest query anyone actually runs.

Whole-table read, 50 M rows x 8 columns

On wall time ArrowMetal is ahead of pyarrow on all three files (2.5x Snappy, 2.9x LZ4, 4.3x uncompressed) and ahead of Polars on all three (1.5x Snappy: 65 ms against 95; 1.4x LZ4: 54 against 73; 1.7x uncompressed: 38 against 63). It takes 1.7-2.0x less CPU time than pyarrow and Polars on the two compressed files (674 and 508 ms against 987-1,283), where the host decodes the token-dense pages, and 9.0-9.5x less on the uncompressed one (83 ms against 746-790), where the decode is work the host never does. Time to first compute, which opens the file afresh for every query, is 9-14 ms against Polars' 16-20 ms and pyarrow's 80-85 ms. With the handle kept, which is what a query engine does:

One float64 column (400 MB of values), file handle kept open

That is the shape a query actually has — open once, project a column, compute — and ArrowMetal is 6.9-14.0x ahead on the read (6.9x Snappy, 6.9x LZ4, 14.0x uncompressed). The reduction takes 2 ms against 6 on all three files, because the values are already in GPU memory when it starts. 400 MB decoded in 2.7 ms (the warm price read of the open-file cache table above) is 150 GB/s.

Decompression on its own

One int64 column of 20 M rows, deliberately structured data (an LCG sequence, so Snappy finds many short matches and emits many tokens — the hard case for a GPU), 1 MB pages:

Page size, the same column, MB/s of decoded output:

These Snappy pages compress 1.05x and are short tokens throughout, so they decode on the host; the LZ4 pages (under 1.0x at 1 MB and 256 KB, one long literal each) decode on the GPU, and at 64 KB (1.12x) on the host.

Reproduce with:

PYTHONPATH=python python Benchmarks/parquet_bench.py --rows 50000000 --codecs snappy,lz4,none --codec-scan --repeat 5

Nested reads — a struct, a list, a list of lists, a map and a list of structs, at 1 M and 10 M rows, against pyarrow, Polars and DuckDB — have their own script. Its numbers here are from Benchmarks/results/parquet_nested_2026-09-24.csv. The run conditions at its start are recorded in Benchmarks/results/bench_conditions_2026-09-24.txt; this run started at a load average of 6.10, busier than the other benchmarks of that day. Wall time in that file, with the fastest CPU reader of each row:

ArrowMetal is ahead on the struct, list of list and map reads at both sizes and on the list of struct read at 10 M rows. To improve: the list read (6.1 ms against pyarrow's 4.96 at 1 M rows, 17.54 ms against Polars' 14.46 at 10 M), the list of struct read at 1 M rows (10.76 ms against 6.09), and the read of all five columns together (42.81 ms against pyarrow's 19.8 at 1 M rows, 115.24 ms against 91.38 at 10 M). The host CPU time is lower on every row: all five columns at 10 M rows cost 34.99 ms of CPU against pyarrow's 593.14 and Polars' 1269.84 (cpu_ms). Every ArrowMetal row matches pyarrow's table (match). An earlier smoke run, recorded while other work shared the GPU, is kept as history in Benchmarks/results/parquet_nested_2026-09-23_provisional.csv.

PYTHONPATH=python python Benchmarks/parquet_nested_bench.py --rows 1000000,10000000 --repeat 3 --out results.csv

What the numbers say

• Both processors decode, and the CPU still does less. A whole-file read of the compressed tables spends 508-674 ms of CPU, the host's share of the decompression, against 987-1,346 ms for the CPU readers; the uncompressed one 83 ms against 746-910. Everything but the decompression of token-dense pages is compute the process never does.
• The arrays land in GPU memory already. Summing the column ArrowMetal just decoded takes 2 ms; the CPU reader pays 6 ms and had to materialise the array first. There is no import step, because the decode wrote into Metal shared memory in the first place.
• Opening the file costs little; the first read of a column costs a little more. A fresh open is 0.2-0.3 ms, and a cold one-column read 7.2-12.3 ms against 2.7-7.0 ms with the handle kept: the cold read parses the column's page headers and makes its chunks resident for the GPU. Holding the ParquetFile across queries, which is what a query engine does, keeps both.
• Snappy and LZ4 decode on whichever side is faster for the page. An LZ77 token stream is serial, so a page is one worker's job and its cost scales with the number of tokens, not with bytes. Incompressible pages are one huge literal and the GPU moves them at memory speed — the price column, 400 MB of random doubles, comes back in 7 ms with the handle kept. Token-dense pages go to the host's cores, which walk a token stream about a thousand times as fast as a GPU thread, while the GPU decodes the rest (Decompression, above): the whole Snappy table in 65 ms against Polars' 95, the LZ4 one in 54 against 73.
• Where the GPU is unambiguously ahead is the uncompressed and dictionary paths, which is also where a GPU-resident analytics stack wants to be: 50-59 GB/s for a plain int64 column, the whole uncompressed table 1.7x Polars, and a dictionary column that comes back as an Arrow dictionary array without materialising a single string.

The writer

ParquetWriter (and am.write_parquet, am_parquet_write) is deliberately small: it exists so ArrowMetal can round-trip its own output, not to compete with pyarrow's writer. It runs on the host — writing is not the part of Parquet that needed a GPU.

• PLAIN for fixed-width columns, PLAIN or RLE_DICTIONARY for byte arrays.
• RLE definition levels; every column is written as optional, so nulls round-trip.
• Uncompressed or Snappy (a compact hash-table Snappy encoder in ParquetWriter.swift).
• Data page v1, one data page per column chunk, rowGroupSize rows per row group.
• Types: all integers, float32, float64, bool, utf8, binary, fixed_size_binary, date32, time32, time64, timestamp. Dictionary columns are materialised first. Decimals, lists, structs and the other nested types are not written.

Files it writes read back with identical values, nulls and types under ArrowMetal and pyarrow — over int64, float64, string, bool and timestamp columns, uncompressed and Snappy; ParquetWriterTests and python/tests/test_parquet.py::test_writer_round_trip check both directions.

Correctness

• Tests/Fixtures/generate_parquet.py writes 44 small fixtures (1.8 MB total, committed): every logical dataset once per encoding/codec/page-version variant, plus long strings (up to 64 KB), lists of int64 and of string, several row groups, decimals stored both as FIXED_LEN_BYTE_ARRAY and as INT32 / INT64, a struct column, all-null columns, an empty file, a single-row file and INT96 timestamps.
• Tests/ArrowMetalTests/ParquetTests.swift reads every variant of a dataset and compares them element for element, which checks the encodings against each other, plus known-value, type, list, statistics and projection tests.
• python/tests/test_parquet.py reads every fixture twice — once on the GPU, once with pyarrow.parquet — and asserts the values, nulls and types are identical: 88 checks over the fixture set, plus projection, row-group selection, statistics pushdown, dictionary output, struct leaves and a 50 M-row round trip behind ARROWMETAL_PARQUET_BIG=1.
• Every Snappy and LZ4 decoder reads the same bytes. ParquetTests.testEveryDecoderAgrees reads every Snappy and LZ4 fixture, whole and one row group at a time, with every page sent to the host, to the SIMD-group kernel and to the page-per-thread kernel, against the routed split; test_every_fixture_reads_like_pyarrow_on_every_decoder reads every fixture and a Snappy, an LZ4 and a ZSTD file of small pages on each decoder against pyarrow; testDamagedPagesOnEveryDecoderRaiseOrReturn reads 240 damaged Snappy and LZ4 files on each decoder and on the split, requiring every read to raise or return. DecompressSplitTests checks the host decoders against byte-at-a-time reference decoders (overlapping copies at every offset from 1 to 16, runs ending exactly at the end of the slot, the Hadoop framing) and on damaged blocks placed between inaccessible pages, so a read or write outside a block would end the process; it also checks the router's decisions and that the staging layout never puts a host page and a GPU page in the same page of memory.
• python/tests/test_parquet_robustness.py damages a file two hundred ways — truncation, a broken magic, a footer length larger than the file, single-byte damage in the footer and in the pages, a hand-built footer nesting Thrift structs 60,000 deep, a 2^64 length, a num_children past the schema — and requires every one of them to raise rather than crash, hang or read outside the mapping.
• ParquetWriterTests and the Python writer test close the round trip.
• Tests/Fixtures/generate_parquet_nested.py writes the nested fixtures under Tests/Fixtures/nested (committed): structs three levels deep, structs of strings and binaries, map<string, int64>, map<int32, string>, maps of structs and of lists, list<list<T>>, list<list<list<int32>>>, list<struct<...>>, struct<list<...>> and list<map<...>>, with nulls and empty lists at every level — written by pyarrow in four encoding / codec / page-version variants with 512-byte pages (so levels cross hundreds of page boundaries), by DuckDB (COPY ... TO ... (FORMAT parquet)) and by Polars wherever it can express the shape (it has no map type).
• Tests/ArrowMetalTests/ParquetNestedTests.swift checks known values against the generator's formulas and every writer's and variant's file against every other's, row by row.
• python/tests/test_parquet_nested.py reads every nested fixture with ArrowMetal and with pyarrow.parquet.read_table and requires the same values and the same types, apart from the three differences listed under Limits (32-bit offsets and no view layouts, nullable struct members, the index type and ordered flag of a restored dictionary), each of which has its own test showing the difference is exactly that; it also damages nested files 200 ways and requires every read to raise or return rather than crash.
• The same generator writes the ARROW:schema fixtures: time zones on every unit (named zones, a fixed offset, UTC, naive), the null type (also below lists, maps and structs), durations, decimal32 / decimal64, a fixed-size list, categoricals over strings, binaries, integers, timestamps and dates (and a pandas categorical, a Polars Categorical and a Polars Enum), field metadata and a field_id, a registered (arrow.uuid) and two unregistered extension types, zones and durations inside a struct, a list and a map, the view and 64-bit-offset layouts, and the same columns with no ARROW:schema (from pyarrow and DuckDB). DuckDB's KV_METADATA writes the crafted ones: an ARROW:schema that is not base64, one that is a truncated message, stored schemas with fewer and more fields than the file and one whose first field is renamed, and a dictionary claim over a struct. ParquetArrowSchemaTests and the ARROW:schema half of test_parquet_nested.py check types, values, field metadata and schema metadata against pyarrow.
• It also writes the page-index fixtures (pyarrow with write_page_index=True in three page layouts and once without, and Polars; plus a float column with NaN from Polars and pyarrow, a NaN inside a page whose other values all equal the filter literal, and uint64 values past the int64 range) and the bloom-filter fixtures (pyarrow with bloom_filter_options, and DuckDB), which ParquetPageIndexTests, ParquetFilterEdgeTests, ParquetBloomFilterTests and the second half of test_parquet_nested.py read; that file also damages the page indexes 180 ways and reads them with filters, requiring every read to raise or return, and reads 40,000 rows of repeated columns three times over.

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test --filter "Parquet"
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test -c release --filter "Parquet"
swift build -c release --product ArrowMetalC
PYTHONPATH=python python -m pytest python/tests/test_parquet.py python/tests/test_parquet_nested.py python/tests/test_parquet_robustness.py -q
python Tests/Fixtures/generate_parquet_nested.py      # regenerate the nested, schema, page-index and bloom fixtures
ARROWMETAL_PARQUET_BIG=1 PYTHONPATH=python python -m pytest python/tests/test_parquet.py -q -k fifty

Limits

• Duplicate column names are kept, not merged. read hands back a ColumnSet — a dict that is positional underneath — so read_parquet(path, columns=["a", "a"]) returns both columns and a file with two columns of the same name reads both of them. Indexing by name gives the first; names, columns, items() and iteration walk all of them in order.
• BIT_PACKED (the deprecated level encoding) and LZO are rejected.
• Encrypted files are not supported.
• A column whose chunks in one read total 4 GiB or more is rejected (the file itself has no size limit).
• Statistics pushdown returns a superset of the matching rows: row-group granular, or page granular when the file has a page index. The deprecated min/max fields are only used when min_value/max_value are absent (they use a signed byte order that is wrong for strings, which is why Parquet deprecated them).
• ZSTD needs libzstd installed; see above.
• The members of a struct and the element of a list are exported as nullable. A writer that declares a struct member required gets not null on that member from pyarrow.parquet.read_table; the values are the same, and the member reads as nullable here.
• A restored dictionary type has int32 indices and no ordered flag. The engine's dictionary arrays carry int32 codes only, so a stored dictionary<int8 | uint8 | uint32, T> (pandas stores a categorical with int8 indices, Polars a Categorical with uint32 and an Enum with uint8, ordered) reads here as dictionary<int32, T>, unordered, where pyarrow.parquet.read_table keeps the stored index type and flag. The values are the same (test_restored_dictionaries_have_int32_indices_and_no_ordered_flag).
• Custom metadata on a nested field is not carried. A top-level column's field metadata comes back (above); metadata on a struct member, a list element or a map value does not, because the engine's nested arrays have no per-field metadata. The types still compare equal.
• large_string, large_binary and large_list come back 32-bit, and the view types as their non-view twins. ArrowMetal narrows 64-bit offsets everywhere and has no view layouts, so a column whose stored Arrow type is large_string or string_view reads here as string, large_list or list_view as list, with the same values. Polars records large_string / large_list for every string and list column it writes.
• decimal256 (precision above 38) is rejected, as is any Arrow type ArrowMetal does not carry.
• A dictionary-encoded column decodes itself where the kernels cannot use the codes. read_parquet returns dictionary-encoded columns encoded (dictionary=False, which read_parquet_table passes, materialises them instead), and the entry points that need the values rather than the codes decode once on the way in: the reductions and statistical aggregates, arithmetic, comparison, cast, the sorts and top_k, and the element-wise maths kernels. The decode is a GPU take of the values by the codes, so it costs one pass; dictionary=False at read time is still the cheaper way to run many operations over the same column. The one thing that does not decode for you is the fused expression compiler behind am.query / am.scan(...), which reads flat columns only: hand it a column read with dictionary=False.
• PLAIN BYTE_ARRAY pages are walked by one thread each to find the value boundaries — the format gives no other option — so a byte-array column with a handful of very large pages has less parallelism than a wide one.
• Arrays are capped at 2^32 elements, as everywhere else in ArrowMetal.

CSV on the GPU

ArrowMetal reads CSV with Metal compute kernels: the file's bytes go into one shared-memory buffer, a quote-aware scan finds every field and record boundary on the GPU, and each column is typed with pyarrow.csv.read_csv's inference rules and converted by a kernel straight into Arrow arrays. The host parses bytes only for a byte order mark, skip_rows and skip_rows_after_names, the header names, error messages and the rare float the GPU parser hands back (below).

The oracle is pyarrow.csv.read_csv. For every file and option set in python/tests/test_csv.py the two readers return the same table — names, types, validity and values, floating point bit for bit — or fail with the same message.

• Sources/ArrowMetal/CSV/CSVReader.swift — CSVReader, getting the file to the GPU, the header and the projection.
• Sources/ArrowMetal/CSV/CSVScan.swift — the structure pass and the row check.
• Sources/ArrowMetal/CSV/CSVColumns.swift — inference and conversion.
• Sources/ArrowMetal/CSV/CSVOptions.swift — CSVReadOptions, CSVColumnType, CSVError.
• Sources/ArrowMetal/Kernels/CSVSource.swift — every CSV kernel.
• Sources/ArrowMetal/Kernels/CSVFloatSource.swift, CSVFloatTable.swift — decimal text to float64 / float32 on the GPU, shared with MetalStringArray.parse(Double.self).
• Sources/ArrowMetalC/ArrowMetalC_CSV.swift — the C ABI (am_csv_* in include/arrowmetal.h).

Using it

var o = CSVReadOptions()
o.includeColumns = ["price", "qty"]
o.columnTypes = ["qty": .int32]
let batch = try CSVReader(path: "trades.csv", options: o).read()     // MetalRecordBatch

import arrowmetal as am
import pyarrow.csv as pc

cols = am.read_csv("trades.csv", include_columns=["price", "qty"])   # ColumnSet of MetalArrays
total = cols["price"].sum()                                          # already on the GPU

# pyarrow's own option objects work too, and read_csv_table returns a pyarrow.Table (zero copy)
t = am.read_csv_table("trades.csv", convert_options=pc.ConvertOptions(column_types={"qty": "int32"}))

am_csv_options o;
am_csv_options_init(&o);
const char* keep[] = {"price", "qty"};
o.include_columns = keep; o.n_include_columns = 2;
am_csv_reader* r; am_csv_batch* b;
am_csv_open("trades.csv", &o, &r);
am_csv_read(r, &b);          /* am_csv_batch_rows / _columns / _column_name / _column, as for Parquet */

The pipeline

  the file -> one Metal buffer          parallel pread into a pooled, page-aligned shared buffer
  csv_summarize                         each thread runs its block of bytes from every parser state
  csv_scan_local, csv_scan_groups       the blocks' transfer functions, prefix-composed
  csv_emit                              every field and record boundary, as a byte position
  csv_check_fields                      every record has the header's width; complex fields counted
  csv_classify on a sample              Arrow's inference over the first 8192 rows
  csv_convert_rows                      every column converted in one row-major pass, each row checked
                                        against the sampled type
  (strings) offsets scan, csv_str_copy  the utf8 / binary columns' bytes

Getting the bytes to the GPU

By default the file is read with pread — up to eight ranges in parallel — into a page-aligned buffer from the context's pool, which is already a Metal shared buffer. fileAccess = .map (file_access="map") maps the file instead and wraps the mapping with makeBuffer(bytesNoCopy:), as the Parquet reader does. No copy is made, but on a warm page cache the mapped read measured slower than the copy overall (105.39 ms against 51.8 ms at 10 M rows in Benchmarks/results/csv_bench_2026-09-24.csv); Benchmarks/csv_bench.py records both (arrowmetal and arrowmetal_map), and ARROWMETAL_CSV_TRACE=1 shows where the time goes phase by phase.

Finding every field: the parser as a table, scanned

RFC 4180 as Arrow reads it is a five-state machine over four byte classes:

A quote opens a quoted section only at the start of a field. Anywhere else it is an ordinary byte, and after a closing quote the rest of the field is literal: ab"c reads as ab"c and "ab"cd as abcd, which is what pyarrow does. A quote-parity scan cannot express that, so the scan runs the exact generalisation of it. Each thread takes a block of bytes (1024 by default, scanBlockBytes) and runs it from all five start states at once, recording for each the end state and the number of boundaries it passed — the block's transfer function. In practice the five runs collapse onto two states (inside quotes or not) within the first line, and the thread continues with just those two. Transfer functions compose associatively, so a prefix scan over them (256-wide in threadgroup memory, then once across the threadgroup totals) gives every block its true start state and the index of its first boundary, and a second pass over the bytes writes each boundary's position. A boundary is one uint32: the byte position, with bit 31 set when it ends a record.

The table is data passed to the kernels, so the delimiter, the quote character, quoting off and double_quote are all the same code. \r\n, a lone \r and \n all end a record: the second byte of a \r\n pair lands in state 0 as an empty line, and empty lines are skipped, as pyarrow's ignore_empty_lines=True does. A last record with no newline after it is closed by one boundary the host appends at the end of the data.

csv_check_fields then checks the shape in one pass over the boundaries: every record has the header's width exactly when the record ends fall at k = w - 1, 2w - 1, ..., so the first boundary that breaks the pattern names the offending row. The same pass counts each column's complex fields — a doubled quote, or bytes after a closing quote — whose value is not one contiguous span of the file.

Spans without materialising them

A field's value is found from two neighbouring boundaries: the bytes between them, or between the quotes of a quoted field. Column kernels derive it on the fly, so a column costs no per-row span storage. A column that has complex fields (usually none) materialises its spans, and its complex values are unescaped into a side buffer by csv_side_len, a prefix sum and csv_unescape.

Type inference: a sample, then a checked guess

Arrow infers a column's type as the first kind in a fixed order that every value converts as (null values convert as anything, except strings when strings_can_be_null is off). csv_classify computes, for each row, the set of kinds it converts as and ANDs them over a column (simd_and, one atomic per SIMD group). It runs over the first 8192 rows only. Every kind ahead of the sampled answer already fails on some sampled row, so the sampled type is Arrow's answer for the whole column exactly when every remaining row also converts as it — and that is checked by the conversion itself, which flags any row that does not convert. A column whose guess fails (a float after 8192 integers, say) is inferred over every row (csv_kind_range, then csv_classify when rows disagree) and converted again. The file's bytes are therefore read once per column set in the common case, not twice. test_type_decided_after_the_sample places the deciding value past the sample for thirteen type transitions.

Conversion: one pass over the rows

csv_convert_rows converts every column that has no complex fields in a single kernel: one thread per row walks the row's fields left to right and writes each column's value, with validity (and booleans) packed 32 rows at a time by simd_ballot. One kernel per column would read every cache line of the file and of the boundaries once per column; this reads them once. Each column's values are a sub-range of one shared buffer that stays alive as long as any of them does. Columns with complex fields use one kernel per column over their materialised spans.

Integers follow Arrow's CSV integer grammar, which is not the grammar of Arrow's cast: spaces and tabs around the value are trimmed, 0x / 0X with one to twice-the-byte-width hex digits is accepted (the digits are the value's bit pattern, so 0xFF is -1 as int8), and + is rejected. The package's string → integer cast (MetalStringArray.parse(Int64.self), str_parse_int) implements the cast grammar — + accepted, no trimming, no hex — so the reader has its own kernel rather than reusing that one; CSVReaderTests.testIntegerGrammarIsTheCSVOneNotTheCasts pins the difference. The kernel accumulates up to 19 significant digits without a range check and checks only the final value, so no 64-bit division runs on the GPU.

Floats parse on the GPU with no floating-point arithmetic (Metal has no double): see below.

Dates, times and timestamps follow Arrow's ParseYYYY_MM_DD, its time-of-day parser and ParseTimestampISO8601, transcribed into the kernel: YYYY-MM-DD validated as a calendar date, hh:mm / hh:mm:ss / hh:mm:ss.fraction, the T or space separator, and a zone of Z, ±HH, ±HHMM or ±HH:MM that is applied to the value. The package's GPU strptime takes one format per call and ISO-8601 as Arrow accepts it is several (a date alone, hh, hh:mm, hh:mm:ss, a fraction, four zone spellings), so it is parsed by its own grammar rather than by strptime.

Strings are a length pass, the GPU offsets scan, and a copy of each row's bytes (from the file, or from the side buffer for an unescaped value).

Float parsing on the GPU

MetalStringArray.parse(Double.self) and parse(Float.self) were Swift's Double(_:) / Float(_:) on the CPU. They now run on the GPU and return the same bits for every row, and the CSV reader uses the same core (CSVFloatSource).

The digits go into a 64-bit integer w (the first 19 significant digits) and a power of ten q; the Eisel-Lemire algorithm (the one in fast_float, which Arrow's CSV reader uses) multiplies w by a 128-bit approximation of 5^q from a table of 651 entries, using mulhi on ulong for the high half, and rounds to nearest-even from the bits below the mantissa. For up to 19 significant digits that is always correctly rounded, the same value a correct strtod returns. Anything it cannot settle exactly goes back to the CPU for that row: more than 19 significant digits where the 19-digit prefix and the prefix plus one round differently; the one product shape the original algorithm leaves to a fallback; and, for the parse API, every spelling outside the plain decimal form and inf / infinity / nan (hex floats, nan(payload), snan, embedded NUL, a malformed exponent). The CPU parses those with the initialiser the API always used. ARROWMETAL_FLOAT_PARSE_HOST=1 keeps parse on the CPU path.

CSVFloatParseTests compares the GPU path with the CPU path bit for bit, validity included, for float64 and float32 over: a hand-written adversarial list (exact halfway points such as 9007199254740993, the subnormal boundary 2.4703282292062327e-324 / ...328e-324, the overflow boundary around 1.7976931348623158e308, an 808-digit mantissa, 1e-400, 1e99999999999999999999, the float32 boundaries, every inf / nan spelling, hex floats); 60,000 random strings over the float alphabet; 120,000 random decimals of 1 to 30 digits with exponents from -360 to 359; and 100,000 random doubles printed shortest, as %.17g and as %.25e, plus odd integers above 2^53 (exact halfway cases) and random float32 values. It also checks that the GPU decided most rows itself, and that ordinary numbers of up to 19 significant digits never need the CPU. In the CSV reader the same kernel is compared with pyarrow's parse, bit for bit, by every float case in test_csv.py.

Matching pyarrow

pyarrow's rules were established by probing pyarrow 25.0.1 (the probe below), and every row of this table is a case in python/tests/test_csv.py.

The probe that established the table, abridged (probe(text, **options) prints pyarrow's types and values or its error):

import io, pyarrow as pa, pyarrow.csv as pc
def probe(text, **kw):
    try:
        t = pc.read_csv(io.BytesIO(text.encode()), **kw)
        print([(f.name, str(f.type), t.column(i).to_pylist()) for i, f in enumerate(t.schema)])
    except Exception as e:
        print(type(e).__name__, e)
print(pc.ConvertOptions().null_values, pc.ConvertOptions().true_values, pc.ConvertOptions().false_values)
probe("a\n1\n+3\n")                            # double: '+' is not an integer
probe("a\n0x1F\n")                             # int64 31
probe("a\n 1\n2 \n")                           # int64: trimmed
probe("a\n true\n")                            # string: booleans are not trimmed
probe("a\n1\ntrue\n")                          # bool
probe("a\n12:34:56.123\n")                     # string
probe("a\n2020-01-01 12:34:56 \n")             # string: timestamps are not trimmed
probe("a\n 2020-01-01\n")                      # date32: dates are
probe("a\n2020-01-01 12:34:56+01:00\n")        # timestamp[s, tz=UTC], 11:34:56
probe("a\n2020-01-01 12:34:56Z\n2020-01-01 12:34:56\n")   # string
probe("a\nNAN\n-NAN\nnan(123)\n")              # double NaN (nan alone is a null value)
probe("a,b,c\n\n1,2,3\n4,5\n", read_options=pc.ReadOptions(use_threads=False))   # Row #3
probe('"x\ny"\na,b\n1,2\n', read_options=pc.ReadOptions(skip_rows=1))           # skip_rows ignores quotes
probe("a,b\n1,2,3\n4,5\n", read_options=pc.ReadOptions(skip_rows_after_names=1))  # a=[4] b=[5]: not width-checked
probe("a\n1000-01-01 00:00:00.5\n")             # string: outside int64 nanoseconds
probe("a\n1000-01-01 00:00:00\n")               # timestamp[s]
probe('a"b\n1"2\n', parse_options=pc.ParseOptions(delimiter='"'))                  # a=[1] b=[2]
probe("a\n1\n" + "x" * 300 + ",1\n", read_options=pc.ReadOptions(use_threads=False))  # row text cut: 96 bytes + " ..."

Options

column_types takes null, bool, the eight integer types, float32, float64, utf8, binary, date32, time32[s|ms], time64[us|ns] and timestamp[s|ms|us|ns] with or without a timezone. In Python the keywords have pyarrow's names and win over the option objects.

Column names and error messages can hold NUL bytes (a header field a\0, an error quoting the value 1\0). The C ABI hands both out as NUL-terminated strings and also gives their byte length, am_csv_batch_column_name_length and am_csv_last_error(&length); Python reads them by length, so they arrive whole (test_nul_bytes_in_names_and_messages). Option strings go in the same way: each string array in am_csv_options has a *_lengths array beside it (NULL means NUL-terminated strings), and Python fills them, so include_columns=["a\0b"], a column_types key, column_names, null_values, true_values and false_values holding a NUL byte match as they do in pyarrow (test_nul_bytes_in_option_strings, test_c_option_lengths). A NUL delimiter, quote_char or decimal_point is refused, as pyarrow refuses it (test_nul_option_characters_are_refused).

Differences from pyarrow

• Quoted newlines are always parsed, which is pyarrow's newlines_in_values=True; the tests run pyarrow that way. pyarrow's default (False) reads a quoted newline correctly while it sits inside one of its blocks and raises when one straddles a block boundary; test_quoted_newline_across_pyarrow_blocks shows both.
• The ragged-row and conversion errors always carry Row #N, which is the message pyarrow's serial reader (use_threads=False) gives; its threaded reader leaves the row number out.
• skip_rows_after_names with no complete row after the header: when nothing after the header ends with a line terminator outside quotes (a,b\n, or a,b\n1,2 with no final newline), pyarrow raises straddling object straddles two block boundaries (try to increase block size?); this reader skips what is there and returns an empty table (test_skip_rows_after_names_with_no_complete_row).
• A header name that is not valid UTF-8 has each invalid sequence replaced by U+FFFD (\xef\xbba becomes \ufffda); pyarrow keeps the name's bytes as they are, and its schema.names then fails to decode in Python (test_header_name_not_utf8).
• The table comes back with one chunk per column, where pyarrow's has one chunk per block it read; the tests compare values, not chunking.
• Not supported yet, each raising NotImplementedError from Python rather than reading differently: escape_char, ignore_empty_lines=False, timestamp_parsers, auto_dict_encode, an encoding other than UTF-8, invalid_row_handler, and every column_types type outside the list above, among them decimal, dictionary, date64, duration, float16, large_string and large_binary (pyarrow reads all of these except float16, for which it raises its own ArrowNotImplementedError). The input is a file path given as str or os.PathLike (a bytes path is refused, as pyarrow refuses it: test_bytes_path_is_refused); pyarrow also takes file objects, and decompresses a path ending in .gz, .bz2, .lz4, .zst or .br, which am.read_csv refuses (test_compressed_extensions_are_refused).

Tests

• Tests/ArrowMetalTests/CSVReaderTests.swift — the structure scan against an independent byte-at-a-time CPU parser over 24 random files full of structural hazards (quoted delimiters and newlines, \r\n, \r, doubled quotes, text after a closing quote, mid-field quotes, empty lines, BOM, missing final newline) at scan block sizes 1, 2, 3, 5, 16, 64, 257 and 1024 bytes, in both file-access modes; inference, forced types, the error messages, projection, skip options, and a float column against Swift's parse.
• Tests/ArrowMetalTests/CSVFloatParseTests.swift — the GPU float parse against the CPU path, above.
• python/tests/test_csv.py — the differential against pyarrow.csv.read_csv: one case per rule in the table above, every option, 60 random files with a fixed seed (every inferred type, nulls, quoting, embedded delimiters, quotes and newlines, mixed line endings, BOM, block sizes 1 to 100), 3000-row files dense with quotes, doubled quotes and \r\n read at scan block sizes 1, 2, 7, 64 and 1024 (so they land across block boundaries throughout), a 300-column file, the default null and boolean spellings read from pyarrow itself, a quoted newline across one of pyarrow's blocks, a 20,000-row file, the type decided after the inference sample, and files written by Polars and DuckDB.

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test -c release --filter "CSVReaderTests|CSVFloatParseTests"
swift build -c release --product ArrowMetalC
PYTHONPATH=python python -m pytest python/tests/test_csv.py -q
PYTHONPATH=python python Benchmarks/csv_bench.py --rows 1000000,10000000

Benchmarks

Benchmarks/csv_bench.py reads a mixed-type file (int64, float64, string, bool, date, timestamp and a quoted text column) at 1M and 10M rows with ArrowMetal, pyarrow.csv.read_csv, polars.read_csv, pandas.read_csv (pyarrow engine and default engine) and DuckDB's read_csv, and writes the median, min and max of each. The numbers here are from a quiet run, Benchmarks/results/csv_bench_2026-09-24.csv; the run conditions at its start are recorded in Benchmarks/results/bench_conditions_2026-09-24.txt. In that file, median wall time:

The default read and read_csv_table are ahead of every CPU reader at both sizes; at 10 M rows arrowmetal reads 16580.9 MB/s against Polars' 8697.9. The mapped read is to improve: at 10 M rows it takes 105.39 ms against Polars' 98.75 ms, and at 1 M rows 14.56 ms against 10.37 ms.

An earlier run, taken while other work shared the GPU, is kept as history in Benchmarks/results/csv_bench_2026-09-23_provisional.csv (named for the day it was started; the run itself went past midnight, so its date column reads 2026-09-24).

ARROWMETAL_CSV_TRACE=1 prints the wall time of each phase of a read to stderr, and =2 runs every kernel in its own command buffer and prints its GPU time.

Limits

• Files of 2 GiB and more are rejected (boundaries are 31-bit positions), and so is a single field of 1 GiB or more.
• Per-field work is one thread per field: a quoted field of many megabytes is walked and copied by a single GPU thread.
• Arrays are capped at 2^32 elements, and a utf8 column at the 2 GB its 32-bit offsets address, as everywhere else in ArrowMetal.
• The options listed under Differences are not implemented.

JSON on the GPU

ArrowMetal reads newline-delimited JSON with Metal compute kernels and returns Arrow columns with the semantics of pyarrow.json.read_json: the same inferred types, the same field order, the same handling of missing keys and nulls, the same explicit-schema options and the same error texts. Finding the records, validating the JSON grammar, matching keys to fields, unescaping strings and parsing ISO-8601 timestamps all run on the GPU; number text is gathered on the GPU and handed to MetalStringArray.parse.

import arrowmetal as am
cols = am.read_json("events.jsonl")               # ColumnSet of MetalArrays, already on the GPU
cols["latency_ms"].mean()
table = am.read_json_table("events.jsonl")        # the same columns as a pyarrow.Table (zero copy)

import pyarrow as pa, pyarrow.json as pj
am.read_json_table("events.jsonl", parse_options=pj.ParseOptions(
    explicit_schema=pa.schema([("id", pa.int32())]), unexpected_field_behavior="ignore"))
am.read_json_table("events.jsonl", explicit_schema=pa.schema([("id", pa.int32())]))  # same, as keywords

let table = try JSONReader.read(path: "events.jsonl")            // JSONTable: names, columns, rowCount
let typed = try JSONReader(path: "events.jsonl").read(JSONReadOptions(
    explicitSchema: [JSONField("id", .int32)], unexpectedFieldBehavior: .ignore))

The C ABI is am_json_open / am_json_open_buffer / am_json_read and the am_json_batch_* accessors in include/arrowmetal.h, shaped like am_parquet_batch; the explicit schema crosses as a struct-typed ArrowSchema. am_json_batch_column_names hands out a column's name and its nested struct field names with their lengths, so a key holding \u0000 arrives whole where a C string and the C Data Interface's field names stop at the NUL; am.read_json_table restores such nested names on the pyarrow table. am.read_json keeps the full top-level names, but a struct column's to_arrow() goes through the C Data Interface and its nested names stop at the NUL (test_key_holding_nul_keeps_its_full_name). The same holds going in and for errors: am_json_read_named takes the explicit schema's field names with their lengths, which am.read_json / am.read_json_table always pass, so a schema field named "a\x00b" matches the key "a\u0000b" (the nul_in_*schema* explicit cases); am_json_last_error hands out the last error with its length, so a message quoting a key or value that holds a NUL arrives whole (test_messages_holding_a_nul_arrive_whole).

• Sources/ArrowMetal/JSON/JSONReader.swift — the public API, the options and the read pipeline.
• Sources/ArrowMetal/JSON/JSONColumns.swift — keys to fields, type inference, the column builders and the recursion into nested values.
• Sources/ArrowMetal/Kernels/JSONSource.swift — every kernel; JSONKernels.swift dispatches them.
• Sources/ArrowMetalC/ArrowMetalC_JSON.swift — the C ABI.

The pipeline

  file bytes (parallel pread into a pooled Metal buffer)
    -> structure    one thread per 256-byte block: escapes, strings, depth, record boundaries
    -> walk         one thread per record: full grammar check, one entry per field (key span,
                    value span, kind), counted then written at scanned offsets
    -> keys         positional byte compare against the first record's keys; the rest through the
                    GPU dictionary encoder
    -> slot matrix  entry index per (field, row), -1 for a missing key; kinds OR-reduced per field
    -> columns      one validity pass, one text gather and one parse per scalar type; nested values
                    walk again one level down

Records

A backslash escapes the byte after it, and whether a block of the file starts inside an escape depends only on the parity of the backslash run that ends the last block before it that is not all backslashes. jb_escape writes that as a key per block and a max-scan gives every block its escape carry. With the carry known, a block's unescaped quotes are counted, and a sum-scan of their parity gives the in-string state at every block start; brackets outside strings are counted the same way, and a sum-scan gives the nesting depth at every block start.

Inside a block the work is bit masks over 64-byte chunks: SWAR compares classify eight bytes at a time into backslashes, quotes and brackets ('[' and '{' differ in one bit, as do ']' and '}'), the escaped bytes come from the carry (with a byte loop only for chunks that hold a backslash), and the bytes inside strings are the prefix XOR of the unescaped-quote mask. The record pass then visits only the brackets, in order, and reads one at a time only the bytes that sit between them at depth 0 — normally a newline. A { at depth 0 opens a record and the bracket that brings the depth back to 0 closes it; records need not be one per line ({"a":1}{"a":2} is two records, and an object may span lines), which is what pyarrow reads too. A null at depth 0 is a record whose fields are all null, as pyarrow reads it after the first object. Anything else at depth 0 that is not whitespace is an error, reported with the text a sequential parser gives (Column() changed from object to array, The document is empty., a string's own error when the string is malformed). RapidJSON reads a NUL byte as the end of its input, so a NUL where a record would start is The document is empty. too.

The walk

Each record is walked by one thread with an explicit container stack (1024 levels). The walk checks the whole grammar as RapidJSON — the parser inside pyarrow — checks it: literals, the number grammar including NaN, Inf and Infinity (which end the number: a . or e after one is the missing-comma error, as in RapidJSON; the special_* probes), RapidJSON's "Number too big to be stored in double" rule for positive exponents, string escapes, \u surrogate pairs, control characters in strings, and every structural position. Errors carry RapidJSON's texts. For every immediate child of the record it emits an entry: the key span, the value span and a kind (null, false, true, integer that fits int64, other number, string, object, array) with flags for escaped keys and strings and for NaN/Inf. The walk runs once to count and once to write at the scanned offsets.

When a record holds a syntax error, the walk still reports the children it reached — including a key whose value never came and the kind of a container it had opened — so the columns see what a sequential parser had seen before the error. That is how a type conflict or a repeated key earlier in the same record is reported ahead of the syntax error, as pyarrow reports it.

A column whose classes conflict is not built, but its objects or arrays are still walked one level down as the class the column started with (values of other classes read as null there), so a conflict or a repeated key inside them that comes earlier in the file than the column's own conflict is found and reported first (nested_*_before_outer probes).

Keys and fields

Field order is first appearance. Entry k of every record is compared, byte for byte, with the key of entry k of the first record that has fields; when every entry matches, the field ids are the positions and nothing else is needed. Keys that do not match are unescaped, prefixed by the reference keys, and dictionary-encoded on the GPU (dictionaryEncode, first-seen order), which gives new fields their ids in order of first appearance. The entries are then scattered into a slot matrix — one column of row slots per field, -1 where a record lacks the key — with an atomic minimum, so a key named twice in one object leaves the later entry without its slot: that is the "was specified twice" check. For groups of up to 64 fields the same scatter OR-reduces each field's kinds through threadgroup atomics; wider groups take a separate pass. Matrices above 512 MB are built a group of fields at a time.

Columns

Every field's kinds decide its type (the table below) or, against an explicit type, whether the column conflicts. Columns that come out as the same scalar type are built together — one validity pass, one gather and one parse for all of them — so a file with hundreds of fields costs a few dispatches per type rather than per column.

• Numbers: the value text is gathered into one utf8 array and parsed by MetalStringArray.parse, integers and floats both on the GPU. Floats use Eisel-Lemire in integer arithmetic, bit-identical to Swift's Double / Float initialiser; the rare values the GPU cannot decide exactly (more than 19 significant digits on a rounding boundary, for instance) are parsed on the CPU with that initialiser (testFloatColumnsParseOnTheGPU checks that a JSON read builds the GPU float-parse kernel). NaN, Inf and Infinity are set by the reader rather than left to the text parser.
• Strings: a length pass and a write pass; strings without escapes are copied, escaped ones decoded (\uXXXX and surrogate pairs to UTF-8) on the GPU.
• Timestamps: a string column whose every value is an ISO-8601 timestamp becomes timestamp[s], as in pyarrow. The kernel (jt_parse) follows the rules of Arrow's ParseTimestampISO8601: the date alone or with an hour, hour and minute, or hour, minute and second after a space or a T; then Z or one of three offset forms, converted to UTC; fractions only where the unit holds them. It is its own kernel rather than a call to the strptime kernel in TemporalFormat.swift because one strptime format describes one layout, and inference has to accept all of them, with their zones, in one pass.
• Booleans, validity: bitmaps written 32 rows per thread.
• Nested values: a struct or list column gathers the spans of its objects or arrays and walks them one level down with the same kernel; a struct's fields go through the same key matching and column building, and a list's elements become its child column with the walk's offsets as the list offsets.

Type inference

The ISO-8601 forms that infer as timestamps, all probed against pyarrow: YYYY-MM-DD, YYYY-MM-DD[ T]hh, ...hh:mm, ...hh:mm:ss, each followed optionally by Z, +hh, +hhmm or +hh:mm (not after a bare date). Fractions (.123), a lower-case t, hour 24, second 60, an invalid calendar day, a five-digit or negative year and YYYY-MM-DDZ keep the column a string.

pyarrow's behaviour, as probed

Every behaviour below is an input in PROBES or EXPLICIT in python/tests/test_json.py, read by both readers and compared: the schema, the values and the nulls, or the error text.

With an explicit schema, the schema's fields come first in schema order and the others follow in order of first appearance (infer), are dropped (ignore) or fail with JSON parse error: unexpected field (error); ignore and error have no effect without a schema, and a name given twice keeps its first type. A value of the wrong class is a conflict against the schema's class (Column(/b) changed from string to number in row 0); a value of the right class that does not convert fails with Failed to convert JSON to int8, couldn't parse:300. Nested struct and list types apply the same rules one level down.

Explicit-schema types

bool, int8 to int64, uint8 to uint64, float, double, string, timestamp in any unit and timezone (fractions up to the unit's precision, zone offsets converted to UTC), null, list and struct of those. Any other type is rejected with an error naming the field (date32, decimal128, binary, large_string, float16, dictionary; test_explicit_types_outside_the_supported_set_are_rejected).

timestamp[ns] holds 1677-09-21T00:12:43.145224192 to 2262-04-11T23:47:16.854775807; a value outside fails with Failed to convert JSON to timestamp[ns], couldn't parse:..., as in pyarrow. As in Arrow, the whole seconds (after the zone offset) must fit before the fraction is added, so 1677-09-21T00:12:43.145224192 fails although its nanosecond count would fit (timestamp_ns_* explicit cases). The coarser units hold every four-digit year.

null follows pyarrow, which infers the column and then converts it: values that are all null (or absent) read as a null column, including as a list item or struct field; values of two classes are the inferred column's conflict; any other value fails with Failed to convert JSON to null from <type>, the type as pyarrow holds it before conversion (bool; numbers and strings as dictionary<values=string, indices=int32, ordered=0>; struct<...> and list<item: ...> of those; the null_* explicit cases, as top-level, list-item and struct-field types).

Differences from pyarrow 25.0.1

Each has a test of its own in python/tests/test_json.py that checks the difference is exactly this.

• Row numbers in error messages count from the start of the file. pyarrow parses in 1 MiB blocks and counts rows from the start of the block; the two agree for files within one block and whenever pyarrow is given a block as large as the file (test_error_row_counts_from_the_start_of_the_file).
• Field order is always first appearance. pyarrow's threaded read of a file larger than one block can order fields by a later block; with use_threads=False its order is first appearance, which is ArrowMetal's (test_field_order_is_first_appearance).
• An object may span a line anywhere in the file. With newlines_in_values=False pyarrow splits blocks at newlines and fails on an object that crosses a block boundary; ArrowMetal has no blocks and reads it, as pyarrow does when the file fits in one block (test_multiline_object_across_a_block_boundary). newlines_in_values, block_size and use_threads are accepted and do not change the result.
• Leading nulls in a list whose item type is not yet known are kept. pyarrow 25.0.1 drops them and shifts the values that follow ([null,1] then [2] reads as [1,2] then [0]); with a struct item its process can end. ArrowMetal returns the nulls, which is what pyarrow returns once the type is given explicitly (test_pyarrow_list_leading_null_defect).
• A file may start with a null record. It reads as a row of nulls, as pyarrow reads a null after the first object; pyarrow's process ends on a leading one (test_leading_null_record).
• Nesting deeper than 1024 levels is an error. pyarrow's process does not survive 2000 levels (test_nesting_limit). Columns nested deeper than 64 levels read on the GPU but cannot cross into pyarrow, whose C Data importer stops at 64 (test_nesting_deeper_than_pyarrows_import_limit).
• Explicit decimal128, binary and large_string are rejected, where pyarrow converts to them (test_pyarrow_converts_to_types_this_reader_rejects).
• When several explicit-schema values fail to convert, the one earliest in the file is named. pyarrow names one of them, depending on its conversion order (test_conversion_error_names_the_first_failing_value).
• Invalid UTF-8 in a key becomes U+FFFD. Field names are Swift strings, so {"\xff":1} reads as a field named "\ufffd"; pyarrow keeps the raw byte, which its Schema.names then cannot decode (test_invalid_utf8_in_a_key_is_replaced). Values are passed through unchanged.
• Errors are ArrowMetalError, with pyarrow's text; pyarrow raises ArrowInvalid.

Benchmarks

Benchmarks/json_bench.py reads the same generated event-log file with am.read_json, am.read_json_table, pyarrow.json.read_json, polars.read_ndjson, pandas.read_json(lines=True) and DuckDB's read_json (each producing an in-memory table), at 1 M and 10 M rows, in a flat shape and a nested one that adds a struct and a list per record, after checking ArrowMetal's table against pyarrow's. The numbers here are from a quiet run, Benchmarks/results/json_bench_2026-09-24.csv; the run conditions at its start are recorded in Benchmarks/results/bench_conditions_2026-09-24.txt. In that file, median wall time:

ArrowMetal is ahead of every CPU reader in all four cases. The closest is DuckDB on the nested file at 10 M rows (320.60 ms against 287.12 ms). The host CPU time differs more than the wall time: at 10 M flat rows am.read_json used 113.28 ms of CPU, Polars 2906.46 ms and pyarrow 8024.27 ms (cpu_ms). The first results, a smoke run taken while other work shared the GPU and the cores, are kept as history in Benchmarks/results/json_bench_2026-09-23_provisional.csv.

PYTHONPATH=python python Benchmarks/json_bench.py --rows 1000000,10000000 --shapes flat,nested
PYTHONPATH=python python Benchmarks/json_bench.py --rows 1000000 --readers arrowmetal,pyarrow,polars

Tests

• python/tests/test_json.py — every probe and explicit-schema case against pyarrow; 60 seeded random files mixing every type, nulls, missing keys, shuffled key order, unicode and escapes; every prefix of eight small files (truncation at each byte); 400 files with one or two bytes replaced; escapes and backslash runs straddling structure blocks at 43 alignments; wide files, a one-record 5000-field file, a file where every record has a new key, long escaped strings and a file past pyarrow's block size; the API; and one test per documented difference.
• Tests/ArrowMetalTests/JSONReaderTests.swift — the max-scan against a CPU loop; record boundaries of random documents (strings holding brackets, escaped quotes, backslash runs across blocks, multi-line objects, null records) against a sequential CPU scanner; top-level error positions and codes; the walk's key spans, value spans, kinds and flags; RapidJSON's error texts; conflicts and repeated keys; timestamps against a day-counting reference and the edges of timestamp[ns]; explicit schemas, explicit null; nested structs and lists; nested conflicts ahead of a column's own; NUL bytes; the sign of -NaN; NaN / Inf / Infinity ending the number; partial byte order marks at the start; keys holding \u0000; random flat files against Foundation's JSONSerialization; a 400-field file; files on disk and bytes in memory.

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test -c release --filter JSONReaderTests
swift build -c release --product ArrowMetalC
PYTHONPATH=python python -m pytest python/tests/test_json.py -q

Limits and what is to improve

• Inputs of 4 GiB or more are rejected: byte positions are 32-bit on the GPU.
• The whole input is read into memory and parsed in one pass; there is no streaming or chunked read yet.
• One thread walks each record, so a single very large record — one line of many megabytes, or one long string — is walked serially, which is to improve.
• Each nested level costs its own set of dispatches, so deeply nested documents take time proportional to their depth.
• Explicit-schema types are the set above.
• Arrays are capped at 2^32 elements, as everywhere else in ArrowMetal, and a column set's text at 2 GiB (32-bit utf8 offsets).

Delta Lake and Apache Iceberg tables

ArrowMetal reads Delta Lake and Apache Iceberg tables from the local filesystem. The table metadata is resolved on the CPU, the data files are pruned by partition values and column statistics, and the Parquet files that remain are decoded by the GPU Parquet reader (Parquet on the GPU) with a projection. The result is the same thing a Parquet read returns: GPU-resident columns (a ColumnSet in Python, a MetalRecordBatch in Swift), or a pyarrow.Table exported from them without a copy.

Source: Sources/ArrowMetal/Lakehouse/ (DeltaTable.swift, IcebergTable.swift, Avro.swift, LakehouseCommon.swift, LakehouseParquet.swift), the C ABI in Sources/ArrowMetalC/ArrowMetalC_Lakehouse.swift and the last section of include/arrowmetal.h, the Python functions in the "Lakehouse tables" section of python/arrowmetal/__init__.py.

Using it

import datetime as dt
import arrowmetal as am

cols = am.read_delta("events", version=3, columns=["user", "amount"],
                     filters=[("day", ">=", dt.date(2024, 1, 1)), ("amount", ">", 100)])
total = cols["amount"].sum()                                # on the GPU, no import step

t = am.read_iceberg_table("warehouse/db/orders", snapshot_id=5593686759590374981)   # a pyarrow.Table
cols, stats = am.read_iceberg("warehouse/db/orders/metadata/00004-....metadata.json",
                              filters=[("region", "==", "west")], with_stats=True)
stats["files_pruned_by_partition"], stats["manifests_pruned"]
am.delta_latest_version("events"), am.iceberg_current_snapshot("warehouse/db/orders")

Swift: DeltaTable(path:) with read(version:columns:filters:), scan(...) (rows plus LakehouseScanStats), snapshot(version:) and latestVersion(); IcebergTable(path:) with read(snapshotId:columns:filters:), scan(...), plan(snapshotId:filters:), schema(forSnapshot:) and the parsed metadata (schemas, specs, snapshots). Filters are ParquetFilter values, the type the Parquet reader takes. C: am_delta_read, am_iceberg_read, am_delta_latest_version, am_iceberg_current_snapshot and the am_lakehouse_batch_* accessors.

Filters

filters is a list of (column, op, value) triples with op one of == != < <= > >=; all of them must hold. The column is named as the table's schema names it (the snapshot's schema when time-travelling in Iceberg). They are used three times: to skip files (partition values; Delta's per-file minValues / maxValues / nullCount; Iceberg's manifest partition summaries, partition tuples and column bounds), to skip row groups (Parquet footer statistics), and then on the rows. The result holds exactly the matching rows, as deltalake's to_pyarrow_table(filters=...) and pyiceberg's scan(row_filter=...) return them. This differs from read_parquet, whose filters only skip row groups.

A null never matches, including under != (SQL and Arrow semantics); NaN is unequal to everything, so it matches != and nothing else, as in pyarrow. Literals are read in the column's type and compared as exact values, as pyarrow compares them:

• an integer column compared with 2.5 behaves as >= 3 for > 2.5; a literal outside the column's range, including doubles beyond the Int64 range such as 9.25e18, matches everything or nothing as the comparison says (testFilterLiteralsAtTypeEdges, test_filter_literals_at_type_edges);
• a float32 column compared with a double literal is the exact comparison: 0.1 as a float32 is 0.10000000149..., so it is > 0.1 and not == 0.1 (testFloat32ComparesExactly, test_float32_filters_compare_exactly);
• a date literal is a datetime.date or "YYYY-MM-DD" (ASCII digits, a year from 0 to 5,000,000, a value that fits date32); a timestamp literal is a datetime.datetime (converted to UTC when it carries a zone, taken as UTC when it does not) or an ISO 8601 string with two-digit hour, minute and second fields that fits the column's unit;
• a decimal literal is an integer, a decimal.Decimal or a string of ASCII digits that is exact at the column's scale and has at most 38 significant digits;
• a string literal is written into the filter text double-quoted, with " and \ escaped as \" and \\ (the grammar of the Parquet filter text, include/arrowmetal.h), so it may hold ;, ", \ and operator characters (("s", "<", "a==b")); a bytes literal (for a binary column) is passed the same way as its UTF-8 text, so it must be valid UTF-8. A NUL byte ends the C string and is refused in both, and so are bytes that are not UTF-8, each with an error saying so. Data columns, partition columns and binary columns filter as deltalake and pyiceberg filter them (test_delta_string_literals_holding_quotes_and_semicolons, test_iceberg_string_literals_holding_quotes_and_semicolons). A column name cannot hold =, !, <, >, ; or ", since the filter text's name ends at the first operator character; Python refuses one that does (test_operators_in_a_literal_are_kept_and_in_a_column_name_refused).

A literal that does not fit its column is an error naming both, never a crash (test_filter_literals, testRowFilterSemantics, testDeltaUnknownColumnAndBadLiteral, testFilterLiteralsAtTypeEdges). Integer, float, temporal and decimal columns compare on the GPU; string, binary and boolean columns compare on the CPU over the unified-memory buffers, strings and binary byte-wise, and a comparison whose answer is known without the data (a literal outside the column's range, a NaN literal) is a constant mask built on the CPU. Row groups are pruned by the same order: the readers decide string and binary row-group pruning themselves, byte-wise on the footer's min_value / max_value, so a decomposed "é" (which byte-wise sorts below "f") is kept for < "f" (testStringRowGroupPruningIsByteWise, test_delta_decomposed_strings_survive_row_group_pruning). Numeric, date, timestamp and boolean filters use the Parquet reader's row-group filter, only where its comparison is exact (an integer column against a double literal of any magnitude, compared without rounding the integer; a timestamp stored in the table's unit; floats never under !=).

Malformed metadata is an error naming the file, never a crash or a hang: Avro blocks whose sizes or counts run past the data, records that contain themselves with nothing optional in between, a Delta partitionValues that is not an object of strings and nulls, a Delta reader protocol 3 whose readerFeatures is missing or not a list of strings, an Iceberg snapshot with neither a manifest-list nor manifests, and a data file that holds none of the table's columns (a Delta table without column mapping, or an Iceberg file without field ids in a table without a name mapping) instead of rows of nulls (testAvroMalformedContainersAreErrors, test_malformed_manifest_list_is_an_error, testDeltaMalformedPartitionValues, testDeltaReaderFeaturesMustBeAListOfStrings, testIcebergSnapshotWithoutManifestsIsAnError, testDataFilesWithoutTheTableColumnsAreErrors). version is 0 or more (None reads the latest; in C, -1).

Delta Lake

File pruning by the per-file statistics uses integer, float, date and string columns. Timestamp statistics are written at millisecond precision and string statistics may be truncated, so timestamp columns and strings of 32 characters or more are not used; boolean, decimal and binary statistics are not used either. The row filter still applies to every column.

Apache Iceberg

Where the results differ from the reference readers

Checked by python/tests/test_lakehouse.py:

• Types match deltalake column for column. pyiceberg returns large_string and large_binary for some string and binary columns; ArrowMetal returns string and binary (test_iceberg_fixtures_match_pyiceberg, test_iceberg_partition_values_that_need_escaping, which has a binary column).
• A projection comes back in the order the columns were asked for; pyiceberg's selected_fields returns them in schema order (test_iceberg_partition_values_that_need_escaping).
• Float comparisons follow pyarrow. deltalake 1.6.5 agrees on every number but also returns the NaN row for some ordering comparisons on a float32 column (> 0.1, < 1e300); pyiceberg 0.12.0 rounds a double literal to float32 before comparing, so its f > 0.1 leaves out the rows holding 0.1 as a float32, which ArrowMetal and pyarrow return (test_float32_filters_compare_exactly, test_float32_literal_rounding_differs_from_pyiceberg).
• pyiceberg binds a time-travel scan's row filter against the current schema, so a column renamed after the snapshot is not found under its old name; ArrowMetal resolves filters in the snapshot's schema, like its projection (test_time_travel_filters_use_the_snapshot_schema). pyiceberg also refuses a float literal against an integer column; ArrowMetal compares exactly, as pyarrow does (test_generated_iceberg_filters).
• pyiceberg 0.12.0 returns every row, nulls included, for a comparison it can decide without the data: != on a column added after some files were written returns those files' null-filled rows, and a literal beyond the column's range (i32 < 2**63 - 1, f32 < 1e308) returns the null and NaN rows. ArrowMetal keeps its rule (a null never matches; NaN matches only !=) and returns the rows pyarrow's filter returns (test_null_rows_under_always_true_filters_differ_from_pyiceberg).
• deltalake 1.6.5 and polars 1.44.1 return nulls for every mapped column of the column-mapping fixture (Parquet files with physical names and field ids, partition values keyed by physical name, as the Delta protocol specifies). DuckDB 1.5.5's delta_scan reads the mapped data columns (id, total) exactly as ArrowMetal does, which checks the file-level mapping, and returns null for the mapped partition column region, which ArrowMetal fills from partitionValues keyed by the physical name. No independent reader confirms that partition keying; it follows the protocol text, and the reference for that column is the pyarrow data the table was written from (test_column_mapping_reference_readers_return_nulls, test_column_mapping_duckdb_reads_data_columns_not_the_partition, test_column_mapping_reads_the_source_data). The fixture is written by hand: pyarrow data files and JSON commits.
• Row order is not defined by either format; every comparison sorts.

Tests and fixtures

Tests/Fixtures/lakehouse/generate_lakehouse.py writes the fixture tables with deltalake 1.6.5 and pyiceberg 0.12.0 (the SQL catalog on SQLite, pyarrow 25.0.1): several commits, appends, deletes, predicate overwrites, a checkpoint, a multi-part checkpoint, partitions of several types, schema evolution, a hand-written column-mapping table, three tables with reader features the reader refuses, an Iceberg table whose last snapshot adds a position delete file, and parquet/nfd_strings.parquet (decomposed strings in three row groups, for the byte-wise row-group pruning test). It also writes expected.json: the rows the reference readers return for 62 reads, which LakehouseTests.swift replays. python/tests/test_lakehouse.py replays the same file, compares the committed tables with the reference readers live, and generates larger tables (thousands of rows, many files, a checkpoint, deletes, schema evolution) to compare every filter comparison over every column type. See Testing.

Performance

Benchmarks/lakehouse_bench.py times full, projected and filtered reads of one Delta and one Iceberg table against deltalake, pyiceberg, polars (scan_delta, scan_iceberg) and DuckDB's delta_scan / iceberg_scan (used only when the extensions load with auto-install switched off); the first run used deltalake 1.6.5, pyiceberg 0.12.0, polars 1.44.1 and DuckDB 1.5.5 with its extensions from the local extension cache. The numbers here are from a quiet run over a 2,000,000-row table, Benchmarks/results/lakehouse_2026-09-24.csv; the run conditions at its start are recorded in Benchmarks/results/bench_conditions_2026-09-24.txt. Median wall time in that file:

Every CPU reader is ahead of ArrowMetal on every read, so table reads are to improve. The gap is largest on Delta: a full read takes 1788.18 ms against Polars' 7.16 ms. On Iceberg it is narrower: a projected read takes 46.29 ms against 36.04 ms for DuckDB and 10.81 ms for Polars. The file also times arrowmetal_table, the same read handed to pyarrow. An earlier run over a 1,000,000-row table, taken while other work shared the machine and the GPU, is kept as history in Benchmarks/results/lakehouse_2026-09-23_provisional.csv. The benchmark times whole reads; data files are decoded several at a time by the GPU Parquet reader (Parquet on the GPU), each thread with its own command buffer.

Out-of-core streaming execution

Datasets larger than memory, answered on the Apple GPU. The rows flow from disk through the GPU one record batch at a time; the CPU touches value columns only where §5 and §10 say it does: a utf8 sort key and the external sort's k-way merge. Only the answer grows with the input, so RAM tracks the answer rather than the scan: measured to 30 GB on a 64 GB machine, where the private running state ranges from a few scalars to a 940 MB group table (§4.1) and peak RSS is dominated by mapped file pages (§9).

let rows = try StreamQuery(ipc: "/data/events")
    .filter(col("amount") > 100)
    .groupBy(["region"], [StreamAggregate(.sum, "amount")])

import arrowmetal as am
am.scan_ipc("/data/events").filter(am.col("amount") > 100) \
  .group_by("region").agg([("sum", "amount", "total")])

Source files: Sources/ArrowMetal/Stream/, C ABI in Sources/ArrowMetalC/ArrowMetalC_Stream.swift and the block at the end of include/arrowmetal.h, Python in python/arrowmetal/stream.py.

---

1. The pipeline

Three stages run at the same time on three threads, connected by bounded queues:

   ┌──────────────┐   batch i+1   ┌─────────────┐   result i   ┌──────────────┐
   │  stage 1     │ ────────────► │  stage 2    │ ───────────► │  stage 3     │
   │  read        │   queue(3)    │  GPU        │   queue(2)   │  merge       │
   │              │               │             │              │              │
   │ mmap + import│               │ one batch   │              │ fold a small │
   │ next batch   │               │ of kernels  │              │ result into  │
   │ (own thread) │               │ (this one)  │              │ running state│
   └──────────────┘               └─────────────┘              └──────────────┘
        SSD                        Metal                        host memory

• Stage 1 is PrefetchingSource. It runs the wrapped source on its own thread and keeps up to depth batches ready (3 by default). Backpressure is two-sided: the reader blocks when depth batches are queued or when the queued batches exceed budgetBytes (2 GB by default), whichever comes first, so a stream of very wide batches cannot pin more memory than the budget.
• Stage 2 is the caller's thread inside StreamingExecutor.run. It pulls one batch, records the operator's kernels into a single Metal command buffer via MetalContext.batch { }, and hands the small result on. It never waits for the merge unless the merge queue is full.
• Stage 3 is the merge thread. It folds each batch's result into the running state, one at a time, in batch order.

Every stage records its own busy time. StreamStats.overlap is (read + gpu + merge) / wall: 1.0 means the stages ran one after another, and it approaches 3.0 when all three are saturated at once. readStallNanos is how long stage 2 waited for a batch the reader had not finished; mergeStallNanos is how long it waited for the merge to drain.

Why the merge stage is cheap

The GPU stage always produces a small result — a handful of scalars, one row per group, k rows for a top-k, a 16 KB sketch. That is the whole reason streaming works: the merge's cost is a function of the answer's size, not the dataset's. The one exception is documented below (the external sort's k-way merge, which is O(rows) on the host by construction).

Buffer budget

MetalContext.pool recycles page-aligned MTLBuffers by exact byte length. Batches of the same shape hit that cache on every read after the first, so an out-of-core scan does not pay mmap and page-fault costs per batch. Its cap defaults to a quarter of the device's recommended working set, at most 8 GB (MetalContext(poolLimitBytes:)).

---

2. Sources

Sources/ArrowMetal/Stream/BatchSource.swift.

One reader thread is not enough

PrefetchingSource hides the read behind the GPU, but it is still one thread doing the mapping and the import. One core copies at roughly 10 GB/s; Polars and DuckDB scan the same directory at roughly 50-60 GB/s because they read it with every core. ParallelIPCSource closes that gap: it hands each of readers threads its own files, so the read stage scales with cores.

The cost is batch order. Batches arrive interleaved across files, in whatever order the threads finish. Every operator here is order independent in its answer — aggregates, the HLL sketch, group-by, top-k, the external sort's run generation, both joins — though a float64 sum's last bits depend on batch order (§10). So the only thing that needs readers: 1 is a filter -> sink_ipc (or to_reader) that must preserve the source's row order.

am.scan_ipc("/data/events", readers=8).group_by("region").agg([("sum", "amount", "t")])

try StreamQuery(ipc: "/data/events", readers: 8)

Parquet files reach the streaming executor through scan_arrow(pyarrow.dataset(...)); am.read_parquet is the GPU reader for in-memory reads (Parquet on the GPU).

3. Sinks

Sources/ArrowMetal/Stream/StreamingSink.swift.

• IPCStreamSink writes the Arrow IPC stream encapsulation incrementally: a schema message, one message per record batch, then the 8-byte end-of-stream marker. That encapsulation is a pure concatenation, which is what makes it appendable — the bytes for each message come from ArrowIPCWriter.encode, with the schema prologue (measured once by encoding the schema with no batches) and the end-of-stream marker trimmed off each call. pyarrow.ipc.open_stream, polars.scan_ipc_stream and this package's own ArrowIPCReader all read the result. Dictionary-encoded columns are decoded before writing, because an incremental sink cannot promise one dictionary per column across batches.
• CallbackSink hands each batch to a closure.
• CollectingSink keeps the batches (only for results known to be small).
• ArrowStreamExporter exports a query as an ArrowArrayStream. get_next pulls exactly one source batch through the whole pipeline, so pyarrow or Polars can consume an out-of-core ArrowMetal query lazily and never materialise more than one batch — Stream.to_reader() in Python.

---

4. Operators

Null handling follows Arrow throughout: a null key forms its own group, count(expr) counts non-null values, sum/min/max skip nulls and return null for an all-null input, and a filter drops rows where the predicate is null.

Filter and project: two paths

The fused Expr compiler evaluates every expression into a fixed-width register, so it can read a utf8 column but cannot write one. streamFilterProject picks between:

• Fused — every output is numeric or boolean: predicate and projections compile into one kernel, the batch is read once, nothing intermediate is materialised.
• Two-step — some output is a plain utf8 / binary / temporal / decimal column. The predicate still compiles into one fused kernel, producing a boolean mask; that mask drives the GPU filter, which carries any column type, and only the computed outputs go back through the compiler.

Streaming group-by: three global tables

• Dense integer key (denseKeyCount given, one key column already inside [0, K)): the global table lives on the GPU as one accumulator array per aggregate, K entries wide. Each batch runs GroupBy over the batch's keys and the per-batch result is folded into the global arrays with element-wise add / min / max kernels — no host round trip at all. When K * aggregates * 16 bytes exceeds gpuStateBudgetBytes the state spills to the host table, and partials recorded before the spill are folded into it, so a 10-million-key group-by still runs inside a fixed GPU budget.
• One integer key with sum / count / mean: the resident hash table of §4.1, on the GPU for the whole scan. A dense key encodes per batch and folds one row per group into it; a sparse one skips the encoding and puts the rows in directly.
• Anything else (several key columns, a string or float key, min / max / variance): GroupByKeys (which turns any key columns into dense ids on the GPU) plus the aggregates give one row per group — typically thousands of rows for a batch of millions. The merge folds those rows into a host dictionary keyed by the key value itself. Exact, order independent, and its size is the number of distinct groups.

Every path returns the groups in ascending key order, so the result is deterministic across batch layouts and runs. (This differs from pyarrow, which returns first-seen order; sort both sides before comparing.)

4.1 The resident group table

Sources/ArrowMetal/Stream/StreamGroupTable.swift. For one integer key with sum / count / mean aggregates, the global table is an open-addressing hash table in device memory that lives for the whole scan: one slot per distinct key, and per aggregate a 64-bit sum and a 64-bit count. Ten million groups with one aggregate is about 940 MB at a load factor of one third. Nothing crosses to the host until finish().

Publishing a 64-bit key with 32-bit atomics

Metal has no 64-bit atomic add or compare-exchange — checked on this M4 Max: atomic_ulong admits only fetch_min and fetch_max on device memory — and only guarantees memory_order_relaxed on the 32-bit ones. So the usual "claim the slot, then publish the key beside it" handshake is not safe: nothing orders the key's store against the claim, and a reader that sees a claimed slot may read a key that is not there yet. Kernels/HashTable.swift avoids that by storing row + 1 in the atomic and reading the key out of an array nothing writes, which a persistent table cannot do.

This table publishes the key through the atomics instead. A slot is three 32-bit words holding the key's 64 bits split 22 / 21 / 21, each stored as field + 1, so 0 means "not written" and every one of the 2^64 keys maps to three non-zero words. There is no reserved sentinel value, and so no key a caller may not use — 0, -1, Int64.min and Int64.max are ordinary keys, and the tests say so. Each word is written with compare_exchange(0 -> field + 1), which gives three properties:

• a word only ever goes from 0 to its final value, and never changes again;
• a thread accepts a slot only when all three words equal its own fields, and because those words are immutable once set, that decision is permanent and every thread agrees with it;
• a thread that mismatches any word walks on, which is what linear probing wants it to do.

The case a naive scheme gets wrong — two threads with the same key racing on one empty slot — resolves without a spin (which can deadlock a divergent SIMD group) and without a duplicate slot: the loser's compare-exchange fails with the winner's value, which is its own field, so it reads the failure as a match and stops on the same slot. Two threads with different keys resolve the same way, because the loser sees a value that is not its field and walks on. A thread that wins word 0 and then loses word 1 to another key leaves a slot some other thread completes — the thread that won a word is still inside its own probe step and goes on to write the rest — so no slot is left half written when a dispatch ends.

Nothing in that argument needs an ordering Metal does not give, and nothing needs the keys in a dispatch to be distinct. So one thread per row is safe, which is what removes the per-batch dense encoding from a sparse, high-cardinality scan.

Three ways in

The chooser is one min/max over the key column per batch: a span small enough for GroupByKeys.rangeIsWorthIt keeps the per-batch encoding, because a thousand groups fit in threadgroup memory and beat a million atomic adds landing on a thousand slots; anything sparser goes row-level. Both give the same answer, so it is only ever a choice of plan.

The atomic accumulate builds a 64-bit add out of two 32-bit ones: add the low word, take the carry out of it into the high word. That is exactly two's-complement 64-bit addition, so it wraps like Int64 does, and being plain integer addition it is independent of the order the rows reach a slot in. Counts and integer sums are therefore bit-identical to the per-batch path and to the host table, run after run, whatever the batch layout — which the tests assert at cardinalities 1, 1k, 100k and 2M.

A float64 sum cannot take that path. There is no 64-bit atomic to compare-exchange a double into, and no emulation of one rounds a binary64 addition correctly or reproducibly. It takes the dense-id branch instead, which still never sorts the key column — the resident table is the dictionary — and keeps the correctly-rounded software adder d_add. The batch-local id is assigned by whichever thread first swaps this batch's stamp into a slot, so exactly one thread per slot per batch assigns one, and the ids are read back in a later dispatch, after the barrier that makes them visible. The cost is two uint arrays the size of the table (about 270 MB at ten million groups), allocated only when that branch is used.

The per-group float64 sum is one thread per group, not the one threadgroup per group the segmented reduction gives it: at a million groups of one row that was 256 threads per row, and three quarters of the batch. Its answer has to match the old one bit for bit, because a binary64 sum depends on the order it was added in and the host table is checked against it — and it does, because the shape of that reduction is fixed. Every butterfly step whose stride is at or above the run length does nothing, so a run of L rows only uses the first P lanes (P the power of two at or above L); and the butterfly over P lanes is the balanced tree whose leaves, left to right, are the lanes in bit-reversed order. Walking the lanes in that order and merging equal-rank partial results on a stack of at most nine rebuilds the same tree, including the rule that an empty accumulator is replaced rather than added to. O(L) per group, same additions, same order. Runs long enough that one thread each is the wrong shape keep the threadgroup version; the mean run length decides, and since both give the same bits it is only a choice of dispatch.

Growth

Growth doubles the table and rehashes on the GPU under the same claim rule, and the row path checks the load factor after its insert, when the number of new keys is known: the pre-emptive growth is sized by what the last batch added, and a batch that outruns it grows the table and re-runs the insert, which finds the keys that did land already present and claims the rest. Being wrong about the estimate costs one extra pass, never an answer.

min / max / variance, multi-column keys and non-integer keys keep the host table, and the tests check every resident path against it bit for bit — a null-key group, all-null groups, duplicate keys per batch, empty and ragged batches, growth across the rehash threshold, keys at both ends of the int64 range, and three integer key widths.

Both fold the same per-batch partials in the same order, one with Swift's + and one with d_add, so the float64 sums agree to the last bit.

4.2 Threshold pruning

Once a top-k (or an ORDER BY ... LIMIT n) has k rows resident, the k-th value is a lower bound on anything that can still enter the answer. Every later batch therefore starts with one comparison kernel over the key column; the rows that pass are gathered — usually none, or a handful — and only they reach the selection and the fold. The expected number of survivors in batch i of a scan for the top k is k / i, so the work per batch collapses to a single pass over one column.

Two details make it exact rather than merely close:

• The comparison is >=, not >. A row that ties the running k-th value can still be the row the total order picks (topK breaks ties by row index), and for a multi-key sort it can still win on the second key. Keeping ties is what makes the pruned answer the answer.
• A threshold is only taken when the k-th row's key is non-null and not NaN. Nulls sort last, so once k non-null rows are resident no null row can displace one; while fewer than k are, nothing is pruned. With a sort key's nulls first (§5), a batch that holds nulls is not pruned, and where the key's float order puts NaN ahead of the values the NaN rows pass the prune as well.

The fold runs on the GPU thread rather than the merge thread, which leaves the merge stage idle and means the threshold the next batch prunes with is always the newest one.

---

5. External sort

Each batch is sorted on the GPU (the existing radix argsort) and written to its own Arrow IPC stream file — one run per batch. finish() merges the runs.

The merge is on the CPU, deliberately. A k-way merge is a branch-per-row loop with no parallelism to give a GPU: at 20 runs the whole merge is one comparison of 20 heap entries per output row. What is on the GPU is the gather. For each output chunk the merge decides the order, then:

1. the rows needed from each run are gathered with one take per run, giving rows grouped by run;
2. one permutation take puts them back in merged order.

So no column data is ever rebuilt value by value on the host, and memory during the merge is one batch per run plus one output batch.

ORDER BY ... LIMIT n never gets here. With a limit the answer is n rows, so nothing has to spill: the running n rows stay Metal-resident and each batch folds into them on the GPU thread, under the threshold prune of §4.2. That is the same argument the bounded fan-in below already used — the global first n are inside the union of each part's first n — applied one batch earlier, where it is worth a thousand times more: a run used to be a whole million-row batch written to disk and read back, and now there are no runs at all. A single-key head uses the GPU top-k selection, which returns exactly the first n indices argsort would (same order-preserving key, same tie-break by row index), so it is a cheaper way to compute the same prefix and not a different answer.

Runs are merged with a bounded fan-in. 570 batches means 570 runs, and opening them all at once would want 570 file descriptors and 570 resident batches; instead the merge runs in passes of at most mergeFanIn (32 by default), writing intermediate runs and deleting their inputs as it goes, so both stay constant however many runs there are. A limit is applied to every pass — the global first n rows are always inside the union of each group's first n — which is what makes ORDER BY ... LIMIT cheap over hundreds of runs.

Each sort key carries the options of a plan sort key (Query engine, "Sort key options"): its direction, its null placement (nullsFirst, last by default) and, for a float key, its float order (ieee by default, total, nan_largest). The per-batch GPU sort and the CPU merge order by the same rules, so the runs and the merged output agree, and every NaN is placed as the order says (ieee: next to the nulls). ExternalSortOperator.Key(_:descending:nullsFirst:floatOrder:) in Swift, am_stream_sort_ex in the C ABI and Stream.sort(..., null_placement=, float_order=) in Python.

Sort keys use the GPU radix sort for numeric, boolean and temporal columns. utf8 and binary columns have no order-preserving GPU key in the streaming sort, so they fall back to a host sort of the string values (correct, but the one place a sort touches every row on the CPU).

6. Streaming quantiles

An exact quantile needs the whole column ordered, which an out-of-core query cannot afford. Instead a mergeable digest keeps a bounded set of weighted centroids: each says "about w values sit near v". Two digests merge by concatenating their centroids, ordering them and compressing back to the budget, which is associative and order independent.

Per batch the GPU does the work: sort the non-null values with the radix argsort, then gather compression sample points with one take. The sample positions are not uniform in rank — they follow the inverse of the t-digest k1 scale function k(q) = (C / 2π) · asin(2q − 1), which packs samples into the tails, where a uniform sample is least accurate. The merge re-compresses under the same scale function, so a centroid near q = 0.99 keeps far less weight than one near q = 0.5.

A centroid covering rank interval [r0, r1) answers any quantile inside it with rank error at most (r1 − r0) / 2: about n / C at the median and n·π / (2C²) near the extremes, so p99 is roughly C times more accurate than the median. The tests assert a rank error under 1 % at C = 1000 across p1 … p99.

7. HyperLogLog on the GPU

Sources/ArrowMetal/Stream/HyperLogLog.swift. Every row is hashed to 64 bits (splitmix64 for fixed-width values, FNV-1a then splitmix64 for bytes); the top p bits pick a register and the position of the first 1 in the rest updates that register with a max. One thread per row, atomic_fetch_max_explicit on a 2^p-entry uint array in device memory (64 KB at p = 14). Contention is low: the hash spreads rows across 16,384 registers and a max only ever moves a register upward, so most atomics are no-ops the hardware resolves without a retry.

Two sketches merge by element-wise max, which makes the result independent of batch boundaries and batch order — the tests assert bit-identical registers for a whole-array sketch and the same data split into 17 chunks. The estimator is the standard one with linear counting below 2.5·m.

-0.0 and 0.0 hash the same; nulls are skipped, as Arrow's count_distinct does.

8. Streaming joins

• Broadcast — the build side is read once into one Metal-resident batch and every probe batch runs the existing GPU hash join against it, straight to the sink. Memory is the build side plus one batch. The right plan whenever one side fits, however large the other one is.
• Grace hash — both sides are streamed once and partitioned by ((key · 0x9E3779B97F4A7C15) >>> (63 − log₂P)) & (P − 1) into P Arrow IPC files per side. The hash takes the high bits of the product so an id column that is a multiple of P does not land entirely in one partition. Equal keys always land in the same partition, so the join of the whole is the union of the per-partition joins, and each partition is small enough for the in-memory GPU hash join. Two passes over each input plus one write and one read of each; memory is one partition pair. Row order differs from an in-memory join, as it does for any hash join; the test compares the result as a set of pairs.

Join keys are int32 or int64 on both sides, which is what the GPU hash join takes. Null keys never match.

8.1 A join followed by an aggregate is one kernel

Sources/ArrowMetal/Stream/StreamJoinAggregate.swift.

A join -> sum used to be two operators with a whole joined table between them: every probe batch ran the hash join, gathered every column of both sides into a new record batch, wrote it to a sink, and the caller summed the rows it got back. The joined rows are the largest thing in that pipeline and none of them is the answer.

try StreamQuery(ipc: path).filter(col("region") < 50)
    .join(dim, on: "region", buildKey: "region")
    .sum("amount")

am.scan_ipc(path).filter(am.col("region") < 50).join(dim, on="region") \
  .agg([("sum", "amount", "total"), ("sum", "weight", "w")])

The terminal picks the plan. .sink / .collect still stream the joined rows out; .sum, .aggregate and .groupBy fuse:

1. The build table is built once. MetalRecordBatch.join builds its hash table per call, which for a streamed join is per batch. The build side does not change across a scan, so BroadcastBuildTable builds it at construction with the same hj_clear / hj_build kernels and every probe batch reads it.
2. The filter projects down to what the kernel reads — the join key plus the probe-side value columns. A filtered join used to gather every column of the surviving rows, including a utf8 one.
3. jfa_probe probes, gathers and accumulates in one kernel. One thread per probe row walks its bucket chain and, for each match, reads the aggregate's value — from the probe row it is standing on, or from the build row the chain points at — and folds it into thread registers. A threadgroup tree reduction turns 256 of those into one partial per accumulator, and jfa_fold folds the partials into a device-resident accumulator pair that lives for the whole scan. Nothing proportional to the number of matches is allocated, nothing is gathered, and nothing crosses to the host until finish() reads sixteen bytes per aggregate. The merge stage does nothing.

Every accumulator is a raw ulong — a signed sum, an unsigned sum, or an IEEE-754 binary64 bit pattern folded by the correctly-rounded software adder d_add, since Metal has neither a double type nor a 64-bit atomic add. The kernel is generated per (key type, aggregate list) and unrolled, so there is no per-aggregate branch in the probe loop. A slot's count doubles as its "is empty" flag, which is what makes min / max free of a per-type sentinel and an all-null input come out as null rather than as an infinity.

sum, count, min, max and mean fuse; up to seven of them at once (Metal binds 31 buffers and the probe itself needs nine). A column name is resolved against the probe side first and then the build side, so sum("weight") reaches a build-side column; a build column shadowed by a probe column of the same name is reachable as name_right, matching the materialising join's output names.

The grouped form (join(...).groupBy(keys, aggs)) keeps the join's index pairs, because a per-group accumulator needs a table probed once per matched pair, and the resident group table's race-free insert relies on every key in a dispatch being distinct (§4.1), which rows are not. It gathers only the key columns and the aggregated values of the matched pairs — two or three columns, never the joined batch's eight — and folds those into the ordinary streaming group-by, whose global table is already resident. A key may come from either side: a probe-side key is gathered with the left index array, a build-side key with the right one.

Semantics are the inner join's and match Polars and DuckDB: duplicate build keys multiply rows, a null key on either side never matches, and a probe row with no match contributes nothing. A left join followed by an aggregate raises, naming what to use instead, rather than returning a number that counts unmatched rows wrongly; so do variance, stddev, count_distinct_approx, a grace hash join, and a non-numeric value column.

Tests/ArrowMetalTests/StreamJoinFusionTests.swift checks the fused answer against the in-memory join followed by the in-memory aggregate — bit for bit for integer aggregates, within one ulp per element for float64 sums — over duplicate build keys, null keys on both sides, empty batches, a batch with no matches at all, a build side of one row and of a million, a group-by key from each side, and int32 and int64 keys.

---

9. Numbers

Measured on an M4 Max (16 cores, 64 GB unified memory, internal SSD), Swift release build, over a 30.21 GB Arrow IPC directory: 570,000,000 rows, 30 files, 570 record batches of 1M rows, 8 columns (id int64 unique, region int32 with 1000 values, bigkey int64 with 10M values, label utf8 with 50 values, amount float64, qty int32, ts int64, flag int8). Generated and run by Benchmarks/streaming_bench.py, which runs every (workload, engine) cell in its own subprocess so peak RSS is that engine's alone and an out-of-memory kill costs one cell rather than the run.

The machine has 64 GB of RAM and the dataset is 30 GB, so after the first pass it is in the page cache: the GB/s figures are memory bandwidth as much as SSD bandwidth, and they are the same for every engine.

The row-level group-by: 4 GB, before and after

Measured on a 4.03 GB run of the same generator (--size-gb 4: 76,000,000 rows, 4 files, 76 batches of 1M rows) with readers = 2, best of three interleaved A/B rounds — before, after, then the reference engines, three times round, one cell process at a time, same dataset and same page cache, and only the loaded library changes between "before" and "after". "before" is the per-batch dense encoding of the previous section; "after" is the row-level path of §4.1.

groupby_10m is 3.5x faster and is now ahead of DuckDB; it was 5.5x behind Polars and is 1.6x behind. groupby_1k is unchanged to the millisecond, which is the point of the chooser: a thousand contiguous values is exactly the shape the range encoding is for, the row path is never taken, and nothing about that workload moved.

Where the batch's time went, per stage, at 76 batches:

The work moved rather than only shrank: the resident table can only be touched from the merge stage, so the insert, the aggregates and the fold all run there now and the GPU stage is left with the read and the filter. Total GPU-side work went from 2.28 s to 0.77 s. What remains in the merge is the insert of a million rows into a 33-million-slot table, the batch's GroupBy, and the fold.

Two intermediate points, measured the same way on the same dataset, showing what each half bought:

The last two lines are the point of the atomic path: with an integer aggregate the batch never builds dense ids, never sorts anything and never runs a second kernel over its groups, and the whole query is read-bound. A float64 sum pays for the dense ids and the per-group reduction because Metal has no 64-bit atomic to fold a double with.

Peak RSS rises by 0.62 GB on groupby_10m, and that is the price of the dense-id branch: two uint arrays the size of the table (2 x 134 MB at 33.5 M slots) plus the batch's segment arrays. The atomic path allocates neither. Nothing else in the table moved.

The earlier measurement: 8 GB, before and after the resident-state work

The 30 GB table this section used to carry is kept below, because it is the size at which the read stops being the limit. The current numbers are from an 8.06 GB run of the same generator (--size-gb 8: 152,000,000 rows, 8 files, 152 batches of 1M rows) with readers = 2, best of three interleaved A/B rounds in which only the loaded library changes — same benchmark script, same dataset, same page cache, one cell process at a time. "before" is main; "after" is §4.1, §4.2 and §5. Wall-clock milliseconds, then peak RSS.

sort_limit is 4.4x faster and spills no runs at all (peak RSS falls by 1.3 GB with them), groupby_10m is 1.86x faster on a gigabyte less memory, topk is 1.4x faster. filter_sum and count_distinct are unchanged within noise, which is the point: they were already right.

The four cross-checked workloads (filter_sum, groupby_1k, topk, sort_limit) are compared against Polars' answer with a 1e-9 relative tolerance on float sums, and all four pass.

Where the pipeline's time goes now

overlap is 1.0 for a serial pipeline, and above that by however much the stages ran at the same time. With readers = 2 the read column is the sum across both reader threads, so it can exceed wall on its own. The stall columns are the GPU stage waiting: for a batch the readers had not finished, and for the merge queue to drain.

The merge stage is zero on every operator that had one, which was the point: nothing folds a batch's answer on the host any more except the host group-by table (§4), and that table is only reached by the keys and aggregates the resident one does not take. The merge stall is 0.00 s everywhere, so the bounded queue between the GPU stage and the merge never fills, and peak RSS stays flat at 2.3 to 2.5 GB over 8 GB of data.

Where the CPU engines are ahead, and why.

• filter_sum and count_distinct are read-bound at readers = 2. Their GPU stage is 0.09 s of a 0.32 s wall and the read stall is 0.21 s of it, so what is being measured there is two reader threads moving 8 GB against Polars reading the same directory with every core at 51 GB/s. The 30 GB numbers below, taken with readers = 8, are what the same operators look like once the read is not the limit.
• groupby_10m was still 5.5x behind Polars at this point, and the reason was precisely located: the merge was 0.34 s, but the GPU stage was 4.0 s, and essentially all of it was the per-batch GroupByKeys that turns a million int64 keys into dense ids. bigkey spans ten million values over a million-row batch, too sparse for the range-encoding path, so every batch paid a full radix sort of its key column plus an atomic group-by min to pick each group's representative row — with a host loop over that batch's ~950,000 groups inside it. That is what the row-level path of §4.1 removed; the 4 GB table above is the result.
• groupby_1k, topk, sort_limit and the join are within 2x to 3.2x of Polars and DuckDB, and are read- or fixed-cost-bound rather than dominated by any one stage.
• The broadcast join is unchanged because the benchmark streams its joined rows back to Python and sums them there. Fusing the aggregate into the probe, so the join's output never leaves the GPU, is measured separately below.

The fused join + aggregate, measured

A separate 4.03 GB run (--size-gb 4 --ipc-format file: 76,000,000 rows, 4 files, 76 batches of 1M rows) with readers = 2, best of three interleaved rounds, one cell process at a time. Both sides are the same binary over the same dataset in the same run: broadcast_join_unfused is the old plan (gather every matched pair into a record batch, collect it, sum it in Python) and broadcast_join is the fused one (§8.1). Both compute sum(amount) from the probe side and sum(weight) from the build side after region < 50, and both agree with Polars to the last two digits of a double (the benchmark's own cross-check passes on every round).

The GPU stage is 3.5x smaller — 0.14 s to 0.04 s — which is what the fusion did: the gather of every column of every matched pair is gone, and so is the joined batch. Peak RSS falls by 180 MB and host CPU by 170 ms (698 to 529 ms), because nothing is collected and nothing is summed in Python. (The fused GPU stage measures between 0.04 and 0.09 s across rounds; the spread is the one-off compile of the generated kernel on the first batch, which the unfused path does not pay because its kernels are already in the pipeline cache. The unfused stage is 0.14 to 0.15 s in every round.)

The wall clock only moves from 379 ms to 336 ms, and Polars and DuckDB are still ahead, because at readers = 2 this cell is read-bound. The read stage is 0.315 s of a 0.336 s wall and the GPU stage waits 0.13 s of it for a batch the two reader threads had not finished: what is being measured is two threads moving 4 GB against Polars reading the same directory with every core. One supplementary round at readers = 4 shows the operator with the read out of the way — 359 ms unfused to 264 ms fused, a 1.36x speed-up (GPU 0.147 s to 0.043 s), which brings it to within 15 % of DuckDB's 229 ms; the cost is peak RSS, 4.4 to 4.9 GB, because four part files are mapped at once.

The earlier 30 GB measurement

Kept because 30 GB with readers = 8 is the size at which the read stops being the limit and the per-batch work shows through. Same M4 Max, a 30.21 GB directory: 570,000,000 rows, 30 files, 570 batches of 1M rows.

The resident top-n and threshold pruning of §4.2 and §5 were measured once on that dataset before it was deleted, in a single interleaved A/B round with readers = 8, printed to a tenth of a second:

At that size top-k and ORDER BY ... LIMIT come within 1.4x-1.9x of Polars and DuckDB — 0.8 s against 0.587 and 0.544, 1.1 s against 0.642 and 0.567 — down from 4.7 s and 11.2 s, because 30 GB is where the per-batch work, not the read, was the limit. The 30 GB run predates the resident group table (§4.1); groupby_10m with that table is measured at 8 GB and 4 GB in the tables above.

pyarrow.dataset (measured in a separate single-reader run, so not comparable cell by cell) finished groupby_10m in 557 s and timed out at 900 s on count_distinct.

• count_distinct_approx is 7x Polars (532 ms against 3,825 ms) at 8.8 GB of peak RSS against Polars' 17.0 GB, and its own running state is a 16 KB sketch. DuckDB's sketch is faster still, but its answer is 12.4 % off against ArrowMetal's 0.007 % — DuckDB's approx_count_distinct is tuned for a much smaller sketch. Against an exact count this is the operator where streaming on a GPU clearly pays.
• filter_sum ties Polars at 57 GB/s — a whole-dataset filtered aggregate over 570 million rows in half a second, with the CPU doing nothing but reading bytes.

Accuracy of the two approximate operators

DuckDB's approx_count_distinct on the same column: 640,524,923, an error of 12.37 %.

The overlap, measured

prefetch=0 makes the read stage run inline on the GPU thread — the same work, serialised. Same process, same page-cache state, one reader thread on both sides, best of two alternating runs each, on a 30 GB dataset:

filter + sum gains least because its GPU stage is only 0.5 s against a 3 s read: there is barely anything to hide. top 100 gains most because all three stages are busy. Going from one reader thread to eight is a second, larger step on top of this: filter_sum went from 2.83 s to 0.53 s and count_distinct from 2.36 s to 0.53 s once the read stopped being the bottleneck.

What the memory figures mean

ArrowMetal's peak RSS at readers = 8 is around 9 GB against Polars' 2 GB, and almost all of the difference is mapped file pages: eight readers hold eight ~1 GB part files mapped at once, and ru_maxrss counts those. They are page cache, not anonymous memory — the kernel reclaims them under pressure — but the number is real and it is the price of the parallel read. With readers = 1 the same queries run at 1.2 GB of RSS (and take two to five times as long). The private state is what §4 lists: a few scalars, a 16 KB sketch, k rows, or the group table.

10. Limits

• readers > 1 does not preserve batch order, so a row-order-preserving sink_ipc or to_reader needs readers: 1 (the default).
• Threshold pruning is exact but not free of assumptions: it compares against the running n-th value with >=, which needs a comparison kernel for the key column's type. Types without one (utf8, binary, boolean, decimals) simply run unpruned — correct, just not faster.
• Sort keys on utf8 / binary columns fall back to a host sort per batch. Numeric, boolean and temporal keys use the GPU radix sort.
• count_distinct_approx is a whole-dataset aggregate, not a per-group one; a per-group HLL would need one sketch per group and is not implemented.
• variance / stddev are not available in the dense-key GPU group-by path (they need a third accumulator array); the arbitrary-key path has them.
• Grace join keys must be integers.
• The IPC sink writes the stream encapsulation, not the random-access file format: the file footer's block index cannot be built incrementally without reaching into IPCWriter. Every reader that matters (pyarrow, Polars, this package) reads it.
• Dictionary-encoded columns are decoded by the sink and by the merge helpers. One dictionary per column for a whole stream is the only form the writer emits, which an incremental sink cannot promise.
• A Stream is single use: a terminal consumes the source. Open a new scan for a second question.
• A group-by over ten million groups is 1.6x behind Polars (§9), down from 5.5x. The rows go straight into the resident table now, so there is no per-batch encoding left to remove; what is left is the insert itself, the batch's GroupBy and the fold, all in the merge stage, which the GPU stage then waits 0.22 s on.
• A float64 sum cannot use the row-level atomic accumulate, because Metal has no 64-bit atomic add or compare-exchange (atomic_ulong offers only fetch_min/fetch_max) and no emulation of one rounds a binary64 addition correctly. It takes the dense-id branch of §4.1 instead, which costs a second pass over the batch's groups and two uint arrays the size of the table. Counts and integer sums skip all of that; on the same query and dataset that is 330 ms against 614 ms.
• The row-level path's float64 sum is deterministic but not associative-order-free: a group's rows are added in a fixed order that depends on how the batches fell, so the same dataset read with a different batch_rows can differ in the last bits, exactly as the per-batch path always could. Integer sums and counts have no such dependence — they are 64-bit integer adds and commute — so they are bit-identical across paths, batch layouts and runs. There is no non-deterministic option to turn on.
• The resident group table takes one integer key and sum / count / mean. min / max (they would need a per-slot atomic minimum), variance (a third accumulator), multi-column keys and non-integer keys all keep the host table, which is exact but folds one row per group per batch on the CPU.
• A fused join + aggregate is an inner broadcast join only. join(...).sum(...) and join(...).groupBy(...) fuse (§8.1); a left join followed by an aggregate raises, because it would have to count unmatched probe rows and that is not implemented, and so does a grace hash join, which writes its partitions to disk and has no fused form. Both name the plan to use instead. variance / stddev / count_distinct_approx and non-numeric value columns raise too.
• A fused aggregate takes at most seven aggregates. Metal binds 31 buffers and the probe needs nine of them, three per aggregate after that.
• The dense-id branch costs about 270 MB at ten million groups — two uint arrays the size of the table — which is why groupby_10m's peak RSS rises by 0.62 GB.
• Each batch is one Metal command buffer, not several. The fixed cost per batch is paid once per batch; a larger batch_rows amortises it, at the cost of memory per batch.
• readers > 1 raises peak RSS by roughly readers x the part-file size, because that many files are mapped at once. Those pages are reclaimable page cache, not anonymous memory, but they do show up in ru_maxrss.
• am_stream_query supports filter + project and filter + aggregate; a grouped query goes through am_stream_group_by, because the Expr compiler's group_by takes a dense integer key and the streaming group-by takes arbitrary key columns. A joined query has no s-expression form either: it goes through am_stream_join_aggregate / am_stream_join_group_by, which take the build side as an Arrow C Stream.
• Arrays above 2³² rows are still refused (Dispatch.checkLength), which bounds one batch, not a dataset.

Polars is the reason most people on an Apple silicon Mac have Arrow-shaped data in memory at all. This document is how you point that data at the GPU.

There are four tiers, the first three in 0.1.0 and the engine (tier 4) in 0.2.0, and they differ in where the GPU sits relative to the Polars plan:

All four move data over the Arrow C Data Interface. For a single-chunk numeric Polars column that is no copy at all -- the GPU reads the buffer Polars already owns. A String column crosses the same way in tiers 1, 3 and 4: in Polars' own Utf8View layout, which the string kernels read directly (tier 2's plugin still asks Polars for large_string). Categoricals and multi-chunk Series each cost one conversion pass -- see Limits. The evidence is below.

---

Install

From the wheel

scripts/build_wheel.sh                                   # swift build, cargo build, then the wheel
pip install python/dist/arrowmetal-*.whl polars

A wheel built from this tree carries both native libraries in arrowmetal/_lib/: libArrowMetalC.dylib (the GPU library) and libarrowmetal_polars.dylib (the tier-2 expression plugin, built by cargo during the wheel build, against that same libArrowMetalC.dylib). All four tiers work from that install, with no Xcode, no cargo and no DYLD_LIBRARY_PATH. The packaged plugin has one rpath, @loader_path, so its @rpath/libArrowMetalC.dylib resolves to the copy beside it, the same file Python loads. The 0.2.0 wheel on PyPI carries libArrowMetalC.dylib only; with it, tier 2 needs the cargo build below.

scripts/check_wheel.sh checks a built wheel: it installs the wheel with its polars extra into a fresh virtualenv outside the repository, with a scrubbed environment and no cargo on PATH, runs one expression or plan per tier, checks that the process loaded exactly one libArrowMetalC.dylib, the packaged one, and runs python -m arrowmetal.bench with and without --parquet. On an M4 Max (macOS 26.6.2, Homebrew Python 3.13.9) it passed with polars 1.44.2 and pyarrow 25.0.1 as pip resolved them, NumPy not installed: the packaged plugin, built against polars 0.55 crates, loads in py-polars 1.44.2. The plugin adds 5.0 MB to the compressed wheel (3.2 MB before, 8.2 MB after) and is 21 MB on disk after strip -x.

Which Polars

pip install 'arrowmetal[polars]' installs polars>=1.44,<1.45, the range every tier works in. Per tier:

A Polars outside 1.44 installed without the extra keeps tiers 1 and 3.

From source

# 1. The GPU library (every tier needs it)
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
    swift build -c release --product ArrowMetalC

# 2. The Python package
pip install polars pyarrow
export PYTHONPATH=python          # or: pip install ./python

# 3. Tier 2 only: the Polars expression plugin
cd polars-plugin && cargo build --release && cd ..

That is the whole build, and both halves are checked: on an M4 Max the Swift product and the cargo release build both go through from an empty target/, with no other flags and no DYLD_LIBRARY_PATH. cargo build --release is enough for the plugin -- it is a plain cdylib that Polars dlopens, not a Python extension module, so maturin is optional, and arrowmetal.polars_plugin.plugin_path() finds polars-plugin/target/release/ on its own.

plugin_path() looks, first hit wins, at $ARROWMETAL_POLARS_PLUGIN (the full path to a plugin dylib); the packaged arrowmetal/_lib/libarrowmetal_polars.dylib when Python loaded the packaged libArrowMetalC.dylib beside it, which is what a wheel install has; polars-plugin/target/release/ and target/debug/ in a source checkout; the packaged copy in any other case; and a maturin install on sys.path. When $ARROWMETAL_LIB pins a development build, a cargo build linked against it therefore wins over a packaged plugin left in _lib/ by a wheel build in the same checkout.

maturin develop --release is the untested path: polars-plugin/ has no pyproject.toml, and plugin_path() looks for a maturin install at <sys.path>/arrowmetal_polars/libarrowmetal_polars.dylib, which is not the layout maturin produces for a cdylib crate. Use cargo build --release, or set ARROWMETAL_POLARS_PLUGIN to whatever you did build.

The plugin's build.rs links libArrowMetalC.dylib by rpath. The dylib's install name is @rpath/libArrowMetalC.dylib, so polars-plugin/arrowmetal-sys/build.rs finds the directory holding it -- $ARROWMETAL_LIB_DIR, $ARROWMETAL_LIB, <repo>/.build/release, <repo>/.build/debug, /usr/local/lib, /opt/homebrew/lib, first hit wins -- and bakes it in as an LC_RPATH entry. No DYLD_LIBRARY_PATH is needed at run time. It republishes that directory to the plugin crate through its links = "ArrowMetalC" key (DEP_ARROWMETALC_LIB_DIR), because Cargo passes a build script's link arguments only to the crate that owns it.

Version pinning

polars-plugin/Cargo.toml pins polars 0.55.1 and pyo3-polars 0.28.0. Those are the Rust crates py-polars 1.44.x is built from: the polars workspace at tag py-1.44.1 carries version = "0.55.1", and pyo3-polars 0.28 is the release that depends on polars ^0.55.1. The requirement is a caret, so Cargo.lock currently resolves polars 0.55.2 and polars-ffi 0.55.2; the ABI check is on polars_ffi's major/minor, so that patch bump loads against py-polars 1.44.1 without complaint.

Polars checks the plugin ABI when it loads the library (_polars_plugin_get_version, which returns polars_ffi's major/minor packed into a u32) and refuses a mismatched pair with this Polars engine doesn't support plugin version: .... Moving to a different Polars means re-pinning both crates to whatever that release's workspace version is -- read it from https://raw.githubusercontent.com/pola-rs/polars/refs/tags/py-<version>/Cargo.toml, do not guess. Tiers 1 and 3 have no such constraint: they are pure Python over the C Data Interface.

---

import polars as pl
import arrowmetal as am

Importing arrowmetal does not import Polars -- the package's only hard dependency is pyarrow. The Polars half arrives on first use. Any of these arms it:

import arrowmetal.polars_bridge          # explicit
am.from_polars(df)                       # first touch of a bridge function
import polars as pl; import arrowmetal   # Polars already loaded -> registered at import

Two functions

am.from_polars(series)      # -> MetalArray
am.from_polars(dataframe)   # -> dict[str, MetalArray]
am.to_polars(metal_array)   # -> pl.Series
am.to_polars({"a": ...})    # -> pl.DataFrame

Three namespaces

# Series
s.arrowmetal.sum() / .min() / .max() / .mean() / .count()
s.arrowmetal.top_k(100) / .bottom_k(100) / .sort(descending=True) / .arg_sort()
s.arrowmetal.filter(mask) / .unique() / .cum_sum() / .hash64()
s.arrowmetal.contains("x") / .starts_with("x") / .ends_with("x") / .upper() / .lower()
s.arrowmetal.to_metal()                     # keep the column on the GPU

# DataFrame
df.arrowmetal.group_by("k").sum("v")
df.arrowmetal.group_by("k", "region").agg(total=("v", "sum"), avg=("f", "mean"), n=(None, "len"))
df.arrowmetal.query(am.filter(am.col("k") == 2).sum(am.col("v")))
df.arrowmetal.sort(["k", "v"], descending=[False, True])
df.arrowmetal.top_k(10, by="v")
df.arrowmetal.filter(df["k"] > 8)
df.arrowmetal.join(other, on="k", how="inner")
df.arrowmetal.to_metal() / .device()

# LazyFrame
lf.arrowmetal.collect_gpu(query_or_callable)

Results come back as Polars objects. Scalar reductions come back as Python scalars, as pl.Series.sum() does -- with one difference, in both tier 1 and tier 2: the sum of an empty or all-null column is None, where pl.Series.sum() answers 0. am_reduce has nothing to add up and says so; min, max and mean are None on both sides.

Grouped aggregates available through .agg: sum, mean, min, max, count, len, n_unique, first, last, median, std, var, product, any, all, plus gb.quantile(column, q). One am_group_by_keys pass builds the dense group ids and every aggregate after that reuses it, so .agg(...) with six outputs costs one group-by.

What each tier-1 method runs

The join

The bridge does not call the C ABI's am_join hash join; df.arrowmetal.join is built out of two other kernels. am_index_in finds, for every left key, the row of the right key column it matches; am_take and am_filter then gather both sides. That is a complete inner or left join whenever the right key is unique -- the usual dimension-table shape -- and the whole thing, uniqueness check included (one am_group_by_keys: as many groups as rows means every key is distinct), runs on the GPU with no row-by-row work on the host.

Everything else falls back to pl.DataFrame.join: a duplicated right key (which changes the row count in a way index_in cannot express), a multi-column key, or an outer/semi/anti join. Pass allow_cpu_fallback=False to get an error instead, so a benchmark can be sure what it measured. Null keys never match, which is Polars' default join_nulls=False.

Column pruning in df.arrowmetal.query

query imports only the columns the query names -- it reads them out of the serialised query's (col "name") nodes. This matters more than it sounds: on the benchmark frame, importing all four columns (one of them 50M strings) cost more than the entire query did.

---

Tier 2 -- the expression plugin

import polars as pl
import arrowmetal.polars_plugin        # registers the namespace

lf.select(pl.col("amount").arrowmetal.sum())
lf.with_columns(pl.col("name").arrowmetal.upper())
lf.select(pl.col("amount").arrowmetal.filter_sum(pl.col("region") == 2))
lf.group_by("k").agg(pl.col("v").arrowmetal.sum())

These are real Polars expressions: they compose with select, with_columns, filter, group_by, over, and they take part in projection and predicate pushdown. That is the whole point of the tier -- tier 1 needs a materialised frame, tier 2 does not.

.filter_sum is the shape that pays for itself: one GPU compaction plus one reduction, with the filtered column never crossing back into Polars.

What tier 2 accepts

Narrower than the bridge, and not the same list as the "Types" paragraph under Limits -- that one is about what round-trips, which is a tier-1 and tier-3 question.

Everything else -- Boolean, the temporal types, Binary, Categorical, Enum, Decimal, List, Struct, Null -- raises a Polars ComputeError whose message starts arrowmetal:. Nothing panics through pyo3. There are two places tier 2 is behind tier 1. Categorical and Enum: the Python bridge recodes Polars' dictionary<uint32> index buffer to int32 for the GPU, and polars-plugin/src/bridge.rs does not, so a Categorical column reaches the kernel as-is and is refused with "dictionary indices must be int32 or int64". And Decimal: am_decimal_op backs s.arrowmetal.sum() in tier 1, and the plugin does not reach for it.

The scalar in .add / .sub / .mul / .truediv

"Integers wrap" is about the arithmetic: 127 + 1 is -128 on an Int8 column, and integer division by zero is 0. It is not about the operand. A scalar the column's type cannot hold exactly is an error, the same call the tier-1 bridge makes (it packs the scalar with struct.pack at the column's own width, and struct.pack raises):

pl.col("i8").arrowmetal.add(1000)      # raises: 1000 is out of range for an Int8 column
pl.col("u8").arrowmetal.add(-1)        # raises: -1 is out of range for a UInt8 column
pl.col("i64").arrowmetal.add(1.5)      # raises: an integer column takes an integer scalar
pl.col("i64").arrowmetal.add(2**60+1)  # exact -- the scalar does not go through an f64

A float column takes either an integer or a float, and an operand too large for Float32 becomes an infinity, which is what struct.pack("f", 1e300) gives tier 1.

group_by_sum, and what the plugin API cannot do

pl.col("k").arrowmetal.group_by_sum(pl.col("v")) returns a struct column of n_groups rows:

df.select(pl.col("k").arrowmetal.group_by_sum(pl.col("v")).alias("g")).unnest("g")

It has to be a struct because a plugin function answers with a single Series. And it runs as a projection over the whole frame, not inside df.group_by(...).agg(...): Polars' expression plugin API (polars.plugins.register_plugin_function) registers expressions, and has no hook for contributing a hash aggregate to the group-by engine itself. The flags it accepts are is_elementwise, changes_length, returns_scalar, cast_to_supertype, input_wildcard_expansion and pass_name_to_apply -- none of them says "I am an aggregation the group-by engine should call per group". df.arrowmetal.group_by(...) (tier 1) is the ergonomic spelling; pl.col("v").arrowmetal.sum() inside .agg(...) also works and is called once per group, which is the wrong granularity for a GPU on small groups.

How the plugin gets the data across

polars-plugin/src/bridge.rs, in both directions:

Series --rechunk--> polars_arrow::ffi::ArrowArray --am_import--> am_array (Metal-resident)
am_array --am_export--> ArrowArray --import_array_from_c--> Series

polars_arrow::ffi::ArrowArray and arrowmetal_sys::ArrowArray are both #[repr(C)] transcriptions of the same C struct, so the hand-off is a pointer cast. Ownership follows the C Data Interface's consumer-releases rule: am_import takes over the exported array (so the Rust side mem::forgets its copy, exactly as the Python binding does after _export_to_c), and import_array_from_c takes over the exported one coming back.

Strings go through CompatLevel::oldest() -- Arrow LargeUtf8, not the Utf8View layout Polars uses natively. That conversion is the plugin's only copy, and it is why the plugin's string row below is 2.3x where tier 1's, which takes the column as views, is 17.6x. Tiers 1, 3 and 4 hand String columns over as Utf8View, which the kernels read directly (the Strings paragraphs under Numbers and Limits).

arrowmetal-sys is a hand-written transcription of the header, not bindgen output: the surface is small, the header is stable, and a checked-in file needs no libclang on the build machine. cargo test in polars-plugin/arrowmetal-sys runs 10 tests against the real dylib (import, export, reductions, compare + filter, scalar arithmetic, top-k + take, hash64, group-by, and the error path).

---

Tier 3 -- the streaming hand-off, and Polars' engine hook

lf = pl.scan_parquet("trades/*.parquet").filter(pl.col("day") == "2026-09-01")

lf.arrowmetal.collect_gpu(am.filter(am.col("region") == 2).sum(am.col("amount")))
lf.arrowmetal.collect_gpu(lambda df: df.arrowmetal.group_by("k").sum("v"))
lf.arrowmetal.collect_gpu()                     # just lf.collect()

collect_gpu collects the Polars plan and then runs one ArrowMetal pass over the result. Its engine= and any other keyword go straight to LazyFrame.collect, so the Polars half can still use the streaming engine: projection pushdown, predicate pushdown and slice all happen before a single byte reaches the GPU. It is an explicit hand-off. Polars' own engine= hook is the other way in, and tier 4 uses it; this section is what the hook is, read from the installed package.

The engine= hook, as it stands in polars 1.44.1

Read from the installed package, not from memory:

• polars/_typing.py:443 -- EngineTypeName: TypeAlias = Literal["auto", "in-memory", "streaming", "gpu"], and EngineType: TypeAlias = Union[EngineTypeName, "Engine"]. So collect(engine=...) does accept an Engine object, not only a name.
• polars/lazyframe/engine.py:77 -- class Engine(ABC), documented as "Subclass this to plug a new backend into Polars", with an abstract name property, a plan_engine property, and collect / execute / collect_async / collect_batches / the sink_* family.
• polars/lazyframe/engine.py:330 -- class _LocalEngine(Engine), "Base for in-process engines backed by PyLazyFrame". Its collect ends in wrap_df(ldf.collect(self.name, callback)).
• The callback comes from _LocalEngine._post_opt_callback(*, background, eager), typed PostOptCallback | None where PostOptCallback: TypeAlias = Callable[[Any, int | None], None] (polars/_typing.py:450). The base returns None.
• polars/lazyframe/engine.py:884 -- class GPUEngine(_LocalEngine) with _name = "gpu". Its _post_opt_callback imports cudf_polars and returns partial(cudf_polars.execute_with_cudf, config=self). It refuses background collection and opts out in eager mode.
• polars/lazyframe/engine_config.py:28 -- SUPPORTED_ENGINE_NAMES = ("auto", "in-memory", "streaming", "gpu"), and _engine_from_name maps the string "gpu" to GPUEngine().
• The callback's first argument is a NodeTraverser (polars/_plr.pyi:2595), whose surface is: get_exprs(), get_inputs(), version(), get_schema(), get_dtype(expr_node), set_node(node), get_node(), set_udf(function, is_pure=False), view_current_node(), view_expression(node), add_expressions(expressions), set_expr_mapping(mapping), unset_expr_mapping().

So a Metal backend is not blocked on Polars adding an API -- the API is there, and tier 4 below is built on exactly this surface. One fact about names: the engine's name is passed to Rust as ldf.collect(self.name, callback), and Rust only knows the four in SUPPORTED_ENGINE_NAMES (a fifth string raises ValueError). When a callback is supplied, Rust invokes it for any known name, "in-memory" and "streaming" included (checked on 1.44.1), and an Engine object passed to collect(engine=...) bypasses the Python-side name check. So a third-party engine runs by passing "in-memory" to Rust and reporting itself through plan_engine; what it cannot do is carry its own name through Rust, so explain and the callback's error message ('cuda' conversion failed) name the wrong engine.

collect_gpu stays the explicit form of the same idea: Polars owns the plan, ArrowMetal owns one pass over the result.

---

import polars as pl
import arrowmetal as am

engine = am.MetalEngine()
df = lf.collect(engine=engine)      # the same frame lf.collect() returns
print(engine.last_report)           # which nodes ran on Metal, and why the rest did not

MetalEngine is python/arrowmetal/polars_engine.py. Polars optimises the plan as it always does and hands the optimised IR to the engine's post-optimisation callback. The callback walks every node and expression, translates the subtrees it can into an ArrowMetal plan (the grammar in Query engine), and replaces each one with a function that runs that plan on the GPU and returns a Polars DataFrame. Everything it does not take stays with Polars' in-memory engine, which also runs whatever sits above a replaced subtree. Every plan collects; the most that can happen is that nothing moves, and then the answer is plain Polars'.

import arrowmetal still does not import Polars: am.MetalEngine loads the module on first touch, like the other three tiers.

How it plugs into Polars 1.44.1

Read from the installed package and checked by python/tests/test_polars_engine.py:

• lf.collect(engine=<an Engine object>) passes the object through unchanged, so no Polars change is needed. The name the engine gives Rust is "in-memory": Rust accepts only its four engine names, and with a callback supplied it runs the callback for any of them; the in-memory engine is also what runs every node the callback leaves. engine.name is "in-memory", engine.plan_engine and repr(engine) say metal.
• The callback receives the NodeTraverser and a second argument that is None under collect and an integer under profile (the time since the query started, which the engine uses to place its rows in the profile).
• set_udf turns the current node into a PythonScan whose function Polars calls as f(with_columns, predicate, n_rows, should_time). That function takes no input, so a replaced subtree is a leaf: the only subtrees that can move are ones whose leaves are in-memory frames (DataFrameScan) or Parquet files the engine reads itself (Scan, below). Any other scan (PythonScan, a CSV or IPC Scan) stays with Polars, and so does everything above it.
• view_current_node raises NotImplementedError: ipc scan for a scan_ipc node. The engine leaves such a node, and everything above it, to Polars and names it in the report.
• Polars does not check the replacement's output. The engine does: a frame whose schema is not the one get_schema() promised raises ArrowMetalError inside the query.
• An exception from the callback reaches the user as ComputeError: 'cuda' conversion failed: <Type>: <message>; the 'cuda' is hardcoded in Polars. The engine's own messages start with ArrowMetal MetalEngine: so they read correctly inside it, name the node (Sort#3) and the reason, and end with the way to run the plan on Polars instead (FORCE_POLARS: collect without engine=, or set ARROWMETAL_METAL_ENGINE=off).
• LazyFrame.profile(engine=...) passes the callback only for a GPUEngine, so lf.profile(engine=MetalEngine()) profiles plain Polars. engine.profile(lf) passes the callback through profile's own keyword, and each replaced subtree appears as a metal:<Node>#<id> row.
• Only collect and the paths built on it run the callback; the next section lists every path and what the engine does on it.
• The IR version the engine was written against, (14, 7), is pinned by a test, as is every Polars surface it touches (_LocalEngine, _post_opt_callback, the NodeTraverser methods, the node classes), so a Polars upgrade that moves one fails a named test instead of changing an answer. A different IR major makes the engine leave the whole plan to Polars. So does a node kind outside KNOWN_NODE_KINDS (the 20 node classes of polars 1.44.1) or a node Polars fails to show to the engine: the report line is The plan holds an unknown node <kind> in polars <version>, so the whole plan stays with Polars., and the query is not an error.
• ARROWMETAL_METAL_ENGINE=off (or 0, false, no, polars) makes every MetalEngine leave every plan to Polars, raise_on_fail=True included; the report says so.

Collect paths

What each Polars 1.44.1 entry point does with a MetalEngine (eng below), read from the installed package and checked by test_every_collect_path_is_explicit and test_polars_side_entry_points_that_never_call_the_engine. last_report.path names the path; a path on which the whole plan runs on Polars says why in last_report.fallbacks and issues a MetalEngineFallbackWarning (a UserWarning) once per process.

collect(background=False) with Polars' eager optimisation flag set asks for no callback either; the report says eager and there is no warning, as for GPUEngine. For code that cannot change its collect calls, ARROWMETAL_METAL_ENGINE=off sends every path to Polars.

What it translates

Polars engine capabilities is the same boundary measured: 29 plan shapes (filters, projections, with_columns, slice, sorts, top-k, group-by with x as the key and as the value of each aggregate, whole-frame aggregates, the four joins, unique, and Parquet scans) over 29 input dtypes and three null patterns, each run through the engine and marked Metal or Polars with the engine's reason. It is generated by python/tests/engine_capabilities.py, and test_the_capability_table_is_current fails when the committed file differs from a fresh run.

Everything else -- %, //, eq_missing, str.ends_with, regex, windows (over), rank, median, an expression over an aggregate, a String-valued output, a narrowing or fallible cast, true division by a scalar that is not a plain literal, a column name holding a NUL byte (Polars' own Arrow export panics on one) -- falls back, and the report says which one. Categorical, Enum, Decimal, List, Struct, Null, Binary and Object columns in a subtree's input keep the whole subtree on Polars.

Where the answers would differ, and what the engine emits instead

Each line is a differential case in test_polars_engine.py or test_engine_conformance.py, run against Polars itself.

• Float comparisons. Polars compares floats in a total order: NaN equals NaN and is greater than every number, and -0.0 equals 0.0. The engine adds the NaN terms ((ne x x) is "x is NaN") so the fused comparison gives Polars' answer, nulls included. is_in matches the same way: a NaN in the value list becomes an (ne x x) term, since ArrowMetal's is_in compares with IEEE equality.
• Float32 arithmetic. The GPU's float adds, multiplies and divides flush subnormals to zero, which Polars does not. The engine computes Float32 + - * / in ArrowMetal's correctly rounded software binary64 and rounds once back to Float32, which is the correctly rounded Float32 result for these four operations, subnormals included. Float64 + - * / is correctly rounded in both.
• Division by a scalar. Polars divides a column by a scalar as x * (1 / c), with the reciprocal rounded in the result type; that differs from the correctly rounded x / c by at most one ulp, in a share of rows that depends on the divisor (about a third of Float64 rows for / 3.0, none for a power of two). The engine emits the same multiply, so the bits match Polars' (test_true_division_by_a_literal_is_polars_reciprocal_multiply, which also checks zero, infinite, NaN, subnormal and null divisors). A column divisor is a true division in both. Over a column of one row Polars divides element-wise instead (the scalar and the column have the same length), so there the answer is the correctly rounded x / c. The engine emits whichever of the two Polars computes for the row count of the node's input: from the in-memory frame when nothing between it and the division changes the row count by an unknown amount, and otherwise by counting that input when the plan runs, both forms in the plan and the count choosing between them (test_scalar_division_and_minus_one_follow_polars_at_every_length).
• Multiplying by -1. Polars multiplies a float column by a scalar -1 (on either side, and divides by -1) as a negation, which flips a NaN's sign bit where a multiply keeps the input NaN. The engine emits ArrowMetal's negate there, so NaN rows carry Polars' bits too (test_multiply_by_minus_one_is_a_negation_like_polars, which compares the raw bits). Over a column of one row Polars multiplies, and the NaN keeps its sign; the engine chooses by the row count as for a division.
• Aggregates. A sum over no values is 0 in Polars (ArrowMetal: null) and gets a fill_null; a min/max over only NaN is NaN in Polars (ArrowMetal: null over a whole frame, an infinity per group), so the engine counts the non-null and non-NaN values and decides from the two; a mean of an Int64/UInt64 column is taken over the values cast to Float64, because ArrowMetal's integer mean sums in 64-bit integers and wraps on extreme values where Polars does not; every result is cast to Polars' dtype (UInt32 counts, the Int32 sum of an Int32 column, Float32 of a Float32, UInt32 for the sum of a Boolean). A per-group count of a Float64 or Boolean column is the sum of its validity bits, because ArrowMetal's group-by will not read those values even to count them, and min/max of a Float64 column per group stays with Polars for the same reason. Polars' min and max order -0.0 below 0.0, so a min over both zeros is -0.0 and a max 0.0; ArrowMetal treats the two as equal and returns whichever it met first over a whole frame, and 0.0 per group. The engine counts the zeros of the sign Polars prefers and takes the sign from that count (test_min_and_max_over_both_zeros_are_polars_signed_zeros).
• Float sums and means. The one place the answers differ, and the engine leaves it: a Float32 or Float64 sum, and a Float64 mean, add the same values in the GPU's order where Polars adds in its own, so the two can differ in the last bits. Both are sums of the same values, so they differ by at most twice the rounding bound of a sum, 2(n - 1) u sum(|x|) for a sum of n values and 2 u sum(|x|) for a mean (u the unit roundoff of the type the sum accumulates in: 2^-24 for a Float32 sum, 2^-53 otherwise), and the engine conformance grid holds each such case to that bound. In the run recorded in Benchmarks/results/engine_conformance_2026-10-02_report.txt (totals in engine_conformance_2026-10-02.csv) the largest difference was 0.199 u sum(|x|) for a Float32 sum (3.55e-5 of the answer), 0.372 u sum(|x|) for a Float64 sum (2.00e-14 of the answer) and 0.0033 u sum(|x|) for a Float64 mean (7.05e-14 of the answer). A mean of an integer or Float32 column is computed in Float64 by both (the engine casts the column, as Polars does); in that run it matched bit for bit in every case but one int64 group mean over the integer extremes (1,000 rows), where the engine returns the correctly rounded mean of the cast values and the two differ by 8.76e-8 u sum(|x|), as much as the answer itself. Every other output of the grid is compared bit for bit. test_polars_engine.py compares these aggregates to a relative tolerance.
• Sort order. Polars (1.44 and the 2.0 release candidate alike, pinned by test_polars_sort_order.py) places the nulls per key, first unless nulls_last, in either direction; orders a float key with every NaN (of either sign) one value above every number, +inf included, in both directions, and -0.0 equal to 0.0; keeps tied rows in input order in both directions under maintain_order=True, a tie falling through to the next key; orders String keys byte-wise and Boolean keys False first. top_k is a descending sort with nulls_last=True and a slice, bottom_k the ascending one. The engine writes each Polars key as one ArrowMetal key whose options give that order: "nulls": "first" where the nulls go first, and "float_order": "nan_largest" on a float key (docs/ENGINE.md, "Sort key options"). No key column is added, and a single-key sort with a slice stays a GPU top-k.
• Integer overflow wraps in both (checked on Int8 and Int64 extremes), and an integer true division by zero is IEEE in both.

ArrowMetal behaviours this suite found, now fixed in the engine

Each was first worked around here and pinned by a strict xfail; each is now fixed in ArrowMetal, its test in test_polars_engine.py passes, and the workaround is gone (test_engine_takes_the_plans_it_once_worked_around runs the plans the engine once changed or declined):

1. A null String slot with bytes under it. Polars exports a null String value with the bytes the slot held (valid Arrow), and ArrowMetal's String gather copied those bytes over the next kept value (test_core_string_filter_null_slot). The engine now hands such a column over as exported.
2. A Boolean column through the plan's sort came back with the right validity bitmap and a null count of 0 (test_core_bool_sort_null_count). Result columns are now used as ArrowMetal returns them.
3. A filter over a scan that carries a date32 column was rejected by the expression compiler (test_core_filter_carrying_a_date). Temporal columns now go to ArrowMetal as their own types.
4. Sorting by a String column that holds a null and a value of 8 bytes or more returned wrong rows (test_core_string_sort_with_nulls, run in a child process because one run ended in a bus error). The engine now sorts by String columns.
5. A finite float literal of magnitude 2^63 or more trapped the process inside the expression compiler, which converted every float literal to Int64 as well. The engine now runs those plans on Metal (test_a_float_literal_of_magnitude_2_63_or_more_runs_on_metal).
6. A UInt64 literal above 2^63 - 1 (found by the conformance grid below, not worked around first: is_in, a comparison or fill_null against a large unsigned value) was rejected by the expression parser, which read every integer literal as an Int64, and the plan stayed with Polars. The parser now keeps such a literal as its bit pattern, the way the code generator writes unsigned literals (test_a_u64_literal_above_int64_max_reaches_the_gpu, and testWideUnsignedLiteralsParsePrintAndStayUnfolded in Swift).

Which translatable subtrees it runs: the defaults

A subtree the engine can translate still has to be one where the GPU is ahead, because getting a Polars column onto the GPU is not free: a single-chunk numeric column is imported without a copy, but mapping its pages into Metal and releasing them costs time on every query. String columns are handed over in Polars' own view layout (see "Strings" under Limits), in the crossover sweep and in the benchmark of the defaults below alike. MetalEngine() (shapes="measured") decides per subtree from measured crossovers (python/arrowmetal/_engine_policy.py). Each translated subtree has shape classes:

• rowwise -- filters and projections only;
• aggregate:<family> (a whole-frame aggregate), group_by:<family> (one key) and group_by_multi:<family> (two or more keys), the families being sum, count (count and len), mean and minmax;
• sort (a full sort) and top_k (a sort with a slice);
• join:inner, join:left, join:semi, join:anti, and distinct (unique).

It also has a dtype class, string when a String column is among the columns it reads and numeric otherwise, and an input, in-memory frames or a Parquet file. A subtree runs on Metal when its input rows (the rows of its in-memory frames, or of its Parquet file as the footer states them; what a predicate will keep is not estimated) are at or above the crossover of every class in it for its dtype class and input; a group-by is judged at the bucket of its estimated number of groups ("Group-bys: the number of groups", below). A class whose row says not taken stays with Polars at every size. The decision reads nothing but the subtree's classes, dtypes, input rows and input, a group-by's estimate and the tables, and the estimate comes from a fixed-seed sample, so the same subtree over the same frame always gets the same answer (python/tests/test_engine_policy.py checks both across 200 calls and across processes). Where the policy leaves a node, the placement moves down to its inputs, so a smaller subtree below it can still be taken: under a whole-frame sum over an inner join, the join runs on Metal and Polars adds up its output.

The crossovers. Benchmarks/polars_engine_bench.py --crossover ran 93 LazyFrames over in-memory frames, the 45 cases below and a group-by grid of 48 (each aggregate family over one int32 key and over two, the keys drawn from 200, 1,000, 10,000, 100,000 or 1,000,000 values or from half the frame's rows, (g1sum200) to (g2minmaxR2)), at 250,000, 500,000, 1,000,000, 2,000,000, 5,000,000, 10,000,000, 20,000,000 and 50,000,000 rows (the probe side of the joins grows with the size, the build side is 1,000,000 rows), and the four Parquet scan cases over files of 1,000,000, 2,000,000, 5,000,000, 10,000,000, 20,000,000 and 50,000,000 rows, snappy and uncompressed, each through Polars' in-memory and streaming engines and through MetalEngine(shapes="all", min_rows=0) cold, best of 7, every MetalEngine result equal to Polars': Benchmarks/results/polars_engine_crossover_2026-09-26-final.csv, run conditions (the load average, sampled each minute) in Benchmarks/results/bench_conditions_2026-09-26-final.txt. Each group-by row of the results file also records the number of groups the case's data holds at that size (groups). The in-memory sort and top-k cases ((b), (m), (n), (o), (p), (q), (x2), (x3), (y6), and (x4), a top 100 by a nullable Float64 key descending, added then) were swept again on 2026-09-29, after each Polars sort key became one plan key with Polars' null placement and float order as its options (see "Sort order" above), at the same eight sizes, best of 7: Benchmarks/results/polars_engine_crossover_2026-09-29-sort.csv holds those rows and the 2026-09-26 rows of every other case, run conditions in Benchmarks/results/polars_engine_crossover_2026-09-29-sort_conditions.txt. The group-by cases that run the grouped Float64 sum and mean or the grouped min/max ((c), (l), (t3), (t4), (t6), (t7), (v3), (v4) and the grid's mean and min + max cases; a mean of an int64 column runs as a Float64 mean, the engine casting it first) were swept again on 2026-09-30, after both were rebuilt, at the same eight sizes, best of 7, and the grid gained a Float64 sum over each of its key sets and group counts ((g1fsum200) to (g2fsumR2), a sum of q / 1e9): Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv holds those rows and the earlier rows of every other case, run conditions in Benchmarks/results/polars_engine_crossover_2026-09-30-groupby_conditions.txt, and the table is fitted from it. Benchmarks/polars_engine_crossover.py fits it the way Benchmarks/router_table.py fits the router table (python/arrowmetal/_router_fit.py), with the MetalEngine as the GPU side and the faster Polars engine as the CPU side, over each case's input rows. A case's fit is the first size from which it is ahead at every larger size, placed between that size and the one below it where the two straight lines meet; a case ahead at the largest size alone has none, and a size where the MetalEngine's answer differed from Polars' counts as behind.

Four rules on the fit. The table keeps each case's and each bucket's fit as fit, and the rows the default uses as rows:

• The margin (MARGIN). A case is ahead at a size when its MetalEngine time, raised by a margin, is at most the faster Polars engine's time: 15% for a numeric shape, 35% for a shape with a String column. A String shape's advantage grows slowly with size: the String sort (o) is 0.98x the faster Polars engine at 1,000,000 rows in the 2026-09-26 sweep (Benchmarks/results/polars_engine_crossover_2026-09-26-final.csv, vs_fastest_polars), 1.2x at 2,000,000, 1.37x at 5,000,000 and 1.5x at 10,000,000, so a fit at 15% would land where the two engines are within noise of each other.
• The String floor (STRING_FLOOR). A shape with a String column is taken from 5,000,000 rows at the earliest, whatever its fit says. The floor is set from the 2026-09-26 runs, where the String sorts below it are close to Polars: (o) 1.2x and (p) 1.38x at 2,000,000 rows in the 2026-09-26 sweep, and 1.21x and 1.31x under shapes="all" at that size in the 2026-09-26 default benchmark (Benchmarks/results/polars_engine_bench_2026-09-26-final3.csv, cold), inside that benchmark's noise band (0.62x to 1.31x); from 5,000,000 rows that sweep has them at 1.37x and up. In the 0.4.0 benchmark below (Benchmarks/results/polars_engine_bench_2026-10-02.csv, cold) they are 1.39x and 1.84x under shapes="all" at 2,000,000 rows, against a noise band of 0.86x to 1.18x.
• The headroom (HEADROOM). The default takes a shape from 1.5 times its fit (a String shape from the larger of that and the floor), and a shape whose 1.5 times passes the largest size measured is not taken. The fit interpolates between sizes measured 2 to 2.5 times apart (250,000, 500,000, 1,000,000, 2,000,000, 5,000,000 and so on), and shapes just past their crossover were within run-to-run noise of Polars: two runs of the same Polars plan differ by 0.62x to 1.31x in the 2026-09-26 default benchmark and by 0.86x to 1.18x in the 0.4.0 benchmark below (MetalEngine default, cold where it took nothing, against polars in-memory, rows of 5 ms or more in Benchmarks/results/polars_engine_bench_2026-09-26-final3.csv and Benchmarks/results/polars_engine_bench_2026-10-02.csv).
• The default benchmark (BENCH_RATIO, BENCH). The sweep times each case back to back; the default is also held to Benchmarks/polars_engine_bench.py --idle (a 100 ms warm-up, a first run after 500 ms of idle on its own, another warm-up, then best of 5 and their median), over the files BENCH names (Benchmarks/results/polars_engine_default_groupby_raw_2026-09-30.csv, "Group-bys after the grouped Float64 sum, mean and min/max were rebuilt" below). A class, or a group-count bucket for a case judged by its estimate, of which a case the default took there is behind the faster Polars engine on the best run or on the median is taken only from the smallest benchmarked size above the largest size it was behind at, and not at all when it is behind at the largest. A case of several classes (a mean and a max in one group-by) that is behind counts against those of its classes whose one-class cases at that bucket and size are not all ahead, and against all of them only when every one of its classes has one-class cases there that are ahead, so a class that is ahead on its own keeps its rows and the case itself is still left. The table records each benchmarked row's cases and their ratios as benchmark.

A class's crossover is the largest over the cases whose classes all belong to its node, so every case that measures a class has to be ahead; if one of them has no fit, the class has no crossover. A case whose classes span two nodes, (b) (a group-by under a top-k) and (e) (a whole-frame sum over a join), measures no class. The fitted table is python/arrowmetal/_engine_crossovers.py, and polars_engine_crossover.py --check fails when it and the results file disagree. The policy then takes the larger of that crossover and the kernels' own: the router table in force (am.router_table(), The crossover) for the kernels it routes, and for the sort classes the crossover of argsort int64, argsort float64 and lexsort (2 int32 keys) against the fastest CPU library in Benchmarks/results/router_2026-09-24.json, 1,000,000 rows. The fits below are from Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv.

A class with no row (a String column in any other class, and every Parquet class but these) has no measurement and is not taken. arrowmetal.polars_engine.placement_rules() returns the table under the router table in force, and group_placement_rules() the group-count buckets below.

Group-bys: the number of groups. Where the GPU group-by is ahead depends on the number of groups more than on the rows. In the sweep (Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv, vs_fastest_polars of MetalEngine all, cold) the one-key sum over 200 groups is behind at every size ((t1) at best 0.58x, (g1sum200) 0.51x); over 10,000, 100,000 and 1,000,000 groups it is ahead at 50,000,000 rows by 2.65x, 2.97x and 2.74x ((g1sum10k), (g1sum100k), (g1sum1M)); over about 0.43 times as many groups as rows it is at 1.02x there ((g1sumR2)), inside the margin. So a group-by class is judged at the bucket of its subtree's number of groups:

A count between the 1,000,000 bucket and a quarter of the rows has no bucket and stays with Polars. Each bucket is fitted like a class, from the points of the group-by cases whose number of groups at that size falls in it (the grid, (c), (i), (j), (k), (l), (t1) to (t7), (v1) to (v4), (y1), (s1) and (s2)); at each size the bucket's point is the worst of its cases there, so its step is the largest of the cases' own, and a bucket ahead at its largest size alone has no crossover. The grid's 1,000,000-group cases hold a quarter of the rows or more below 5,000,000 rows, where they count in the rows/2 bucket, so the 1,000,000 bucket was measured from 5,000,000 rows, and that is where its fits start. The rows the default uses, with the fit each came from in brackets (the router table's group_by_sum row is below every one of them):

An empty cell has no measurement and is not taken. * Raised by the default benchmark (BENCH_RATIO; Benchmarks/results/polars_engine_default_groupby_raw_2026-09-30.csv, the faster Polars engine ÷ the default over every run in which the default took the case): at 2,000,000 rows the default took (c) (a mean and a max over two keys, 10,000 groups) at 0.82x of the faster Polars engine on the best run and 0.75x on the median, (g2mean10k) at 0.87x and 0.75x and (g2mean100k) at 0.65x and 0.57x (both tables' runs where the default took them), all ahead at 50,000,000 rows, so the two-key mean's 10,000 and 100,000 buckets are taken from 50,000,000 rows. (c)'s min + max class keeps its 10,000 bucket: (g2minmax10k), its one-class case there, was 1.18x and 1.09x. The rows/2 bucket is never taken (UNTAKEN_BUCKETS), whatever its fit, because the sweep (Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv) is not monotone there: the one-key count over 0.43 times as many groups as rows, (g1countR2), is 0.78x, 1.9x, 4.04x, 2.21x and 1.18x the faster Polars engine at 2,000,000, 5,000,000, 10,000,000, 20,000,000 and 50,000,000 rows, and the two-key rows/2 cases peak at 10,000,000 rows the same way ((g2countR2) 6.47x there, 2.38x at 50,000,000). (j), (v3) and (v4), whose keys range over 100,000 x 1,000 values and which hold 0.79 to 1.0 times as many groups as rows in the sweep, are 1.22x, 1.17x and 1.51x at 50,000,000 rows. The grid's Float64 sums set two sum buckets: the one-key 10,000 bucket ((g1fsum10k) 1.04x at 2,000,000 rows and 1.25x at 5,000,000, then 2.63x at 10,000,000) and the two-key 1,000 bucket ((g2fsum1k) 0.74x at 5,000,000 and 1.88x at 10,000,000). The one-key min + max over 200 and 1,000 groups is taken from about 20,000,000 rows: (g1minmax200) is 1.13x, 1.41x and 1.67x at 10,000,000, 20,000,000 and 50,000,000 rows, (g1minmax1k) 1.06x, 1.27x and 1.64x.

The probe. For a group-by whose keys are columns of one in-memory input frame, read as they are (through filters, projections, joins and unique below it, which can only drop key values, so the frame's count is an upper bound), the engine estimates the number of distinct key tuples in that frame at plan time. It reads the keys of a stratified sample of n of the frame's N rows, row i·(N / n) + h(i) with h a fixed-seed hash, so the same frame always gets the same sample; counts the distinct tuples d and those seen exactly once (f1) and twice (f2); and scales them with the bias-corrected Chao1 estimator (Chao, Biometrics 2005), D = d + f1 (f1 - 1) / (2 (f2 + 1)), clipped to [d, N]. Its range is D with f2 moved by two Poisson standard deviations (and one or two more) either way, the high end being the whole frame while fewer than about 6 pairs have been seen. The samples are 512, 2,048, 8,192 and so on, at most 65,536 rows and a quarter of the frame, and the probe stops at the first whose range the decision does not depend on: every bucket the range reaches is taken at these rows, or none is. So a few hundred groups or ten thousand settle on 512 rows, a hundred thousand on 2,048, and a count near a quarter of the rows takes more. A frame of at most 4,096 rows is counted exactly. One integer key is read as it is; several keys, and a String, Boolean or temporal key or one holding nulls, through Polars' hash. The samples' counts are cached per frame, key columns and sample size, keyed by the columns' buffers (the cache keeps those columns alive, so no other frame can take their addresses while it stands), so collecting the same frame again samples nothing and gets the same estimate; arrowmetal.polars_engine.clear_group_estimates() drops the cache. The probe runs only when some bucket of the class is taken at the input rows; below every bucket's crossover the subtree stays with Polars without one. A Parquet file's footer answers when it states a distinct count for every key column in every row group, and only when the largest row group's count and the sum over row groups fall in one bucket; the benchmark files state none. With no estimate (a key the plan computes, keys from an aggregate below, two group-bys in one subtree, a footer without distinct counts) a group-by class is judged by its row in the class table: group_by_multi:minmax from 11,675,406 rows, group_by:minmax from 19,740,807, group_by_multi:mean from 24,143,946, group_by_multi:count from 26,547,327, the (String, int32) sum from 9,744,372, and every other numeric group-by class at no size.

What the probe costs, and how close it gets. Benchmarks/group_probe_bench.py collects a group-by sum over the grid's frames (one int32 key, two, and one String key, at each group count) through MetalEngine() nine times with its caches cleared and reads the probe's time from the report: Benchmarks/results/group_probe_2026-09-26.csv, run conditions in Benchmarks/results/group_probe_2026-09-26_conditions.txt. Against the fastest of Polars' two engines and the default for the same group-by:

That file was measured under the earlier group-count table. Under the table in force, a group-by at 2,000,000 rows is probed only for the two-key count and min/max, the classes with a bucket taken at that size; a one-key, String or two-key sum group-by there is not probed, and neither is a String group-by at 50,000,000 (only the (String, int32) sum has buckets). In the default benchmark below, where the probe runs in a process that holds every case's frames (probe_us in Benchmarks/results/polars_engine_bench_2026-10-02.csv, the probes of one collect summed), its median is 150.5 µs at 2,000,000 rows (2.90% of the default's time for the case; 1 of the 14 probes under 1%) and 126 µs at 50,000,000 (0.75%; 51 of 77 under 1%). The seven probes over 5 ms, 6.9 to 12.7 ms, are all at 50,000,000 rows on frames with 0.43 times as many groups as rows or more ((g1minmaxR2) 6.9 ms, 1.41% of the default's time; (g2sumR2) 12.7 ms, 1.99%), and its largest shares are 15.96% at 2,000,000 rows ((g2minmax100k), 390 µs of 2.4 ms) and 14.60% at 50,000,000 ((g2sum1M), 3.8 ms of 26.2 ms). A second collect of the same frame reads its samples from the cache. Sampled until its denominator settles instead of until the decision does (free_estimate in Benchmarks/results/group_probe_2026-09-26.csv), the estimate falls in the bucket of the true count for every frame and key set of the grid at both sizes, within 29% of it up to 1,000,000 groups at 50,000,000 rows, and 24% to 31% above it at half the rows.

Each probe is listed in the report, and the rule of a group-by the policy judged by its estimate names it:

  metal:  GroupBy#1 [GroupBy > DataFrameScan] over 50,000,000 rows, ran in <t> ms -> 100,000 rows
          rule: 50,000,000 input rows is at or above the 2,961,043-row crossover for group_by:minmax at an estimated 102,981 groups over (k1), a Chao1 estimate from a 2,048-row sample, 67,758 to 198,392 (engine table, the 100,000-group bucket, ...)
  groups: GroupBy#1 over (k1): 102,981 groups, a Chao1 estimate from a 2,048-row sample, 67,758 to 198,392 (probed in 132 us)

  polars: GroupBy#1: rule: estimated 191 groups over (region), a Chao1 estimate from a 512-row sample, 189 to 196: below the measured band for group_by:sum at 50,000,000 input rows (taken at 3,163 to 3,162,277 groups; no count in the estimate's range is; ...)
  groups: GroupBy#1 over (region): 191 groups, a Chao1 estimate from a 512-row sample, 189 to 196 (probed in 76 us)

The other classes in the sweep (Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv, vs_fastest_polars of MetalEngine all, cold):

• Joins and unique on numeric keys: the left join is ahead from the smallest input measured, 1,250,000 rows, and taken from 1,875,000; the inner join is fitted at 1,353,218 and taken from 2,029,827, the anti join at 2,485,306 and from 3,727,959. The semi join is not taken: against a 1,000-row table, (f), it is behind at every size (at best 0.47x), though against a 1,000,000-row table, (w3), it is fitted at 2,464,828. unique is fitted at 1,252,378 rows over 10,000 groups (r) and at 3,662,727 over about as many groups as rows (x1), and taken from 5,494,090.
• Sorts over in-memory frames are fitted at 250,000 rows (x2, the smallest input measured; 1.42x there), 456,986 (m) and 631,891 (q). The sort kernels' crossover, 1,000,000 rows, is above (q)'s 947,836 and sets sort.
• Top-k is fitted at 6,600,767 rows by a Float64 key descending (n; 1.86x at 10,000,000 rows, 1.81x at 50,000,000) and at 6,210,387 by an int64 key (x3; 1.1x to 1.35x from 5,000,000), but the top 100 by a nullable Float64 key descending (x4), whose answer is 100 of the null rows, is ahead at no size (0.75x to 1.05x), so top_k is not taken.
• Whole-frame aggregates and row-wise filters and projections are behind at every size: (a) to (a4) at 0.09x to 0.24x, (d) at 0.14x to 0.42x.
• String shapes: the sorts (o) and (p) and unique (y5) are taken from the 5,000,000-row floor, and the (String, int32) group-by (k), about 1,000,000 groups from 5,000,000 rows, from 9,744,372. The filter by a String equality (y2, at best 0.13x), the prefix filter and sum (y3, 0.5x), the join on a String key (y4, 0.67x) and the top-k with a String column (y6, 0.88x) are behind at every size; the group-by on a String key with 1,000 values (y1) is ahead by the margin at no size (1.27x at 20,000,000 rows, 1.09x at 50,000,000).
• Over a Parquet file, cold (the open-file cache cleared before each run), the sort is ahead from the smallest file measured, 1,000,000 rows, with both codecs (4.37x uncompressed and 4.9x snappy there), and taken from 1,500,000. The filter and group-by (s1) has no fit with either codec (uncompressed at best 1.09x, at 50,000,000 rows; snappy 1.42x and 1.26x at 1,000,000 and 2,000,000, then 0.79x to 0.92x), nor has the aggregate (s3) (at best 0.84x); the filter that skips row groups, (s2), is ahead at every size snappy (1.24x to 2.48x) and at none uncompressed (at best 0.99x). So the group-by and aggregate classes over a file are not taken.

The default against Polars and against shapes="all". Benchmarks/polars_engine_bench.py --scan over the 106 in-memory cases at 2,000,000 and 50,000,000 rows and the four Parquet scan cases over the 50,000,000-row files, snappy and uncompressed, 220 case-size pairs, best of 5, every result equal to Polars' (the cold runs of MetalEngine() clear the group-count cache too, so each pays its probe): Benchmarks/results/polars_engine_bench_2026-10-02.csv, run conditions in Benchmarks/results/bench_conditions_2026-10-02.txt. The default takes a subtree in 75 pairs (11 at 2,000,000 rows and 64 at 50,000,000), 55 of them group-bys, and every one is ahead of the faster Polars engine, from 1.52x ((g1minmax200) at 50M) to 11.54x ((r) at 50M):

In the other 145 pairs the default takes nothing and runs Polars' in-memory plan; there its time is 0.40 to 1.42 times the polars in-memory row of the same case (wall_ms in Benchmarks/results/polars_engine_bench_2026-10-02.csv), and 0.86 to 1.18 times in the 112 of them where that row is 5 ms or more. That is the spread of two runs of the same Polars plan in this run, and the noise band the ratios above are read against. shapes="all" takes a subtree and is ahead of the faster Polars engine in 64 of those 145 (vs_fastest_polars of MetalEngine all, cold), 47 of them above 1.18x, each left by the default because its class or bucket has a crossover above that size or none:

• at 2,000,000 rows: two-key min + max and count over about 1,000,000 groups, estimated above every bucket taken for their class at that size, (g2minmaxR2) 3.07x, (g2minmax1M) 2.54x, (g2countR2) 1.68x and (g2count1M) 1.64x; group-bys below the lowest crossover of their class, (g2fsum100k) 2.47x, (g2sumR2) 1.80x, (g2fsumR2) 1.61x, (g2sum10k) 1.58x, (v1) 1.57x, (g2sum100k) 1.51x, (g2fsum1M) 1.48x, (g2sum1M) 1.27x, (g2fsum1k) 1.26x and (g2fsum10k) 1.22x (group_by_multi:sum from 2,288,101), (g2mean100k) 2.41x, (g2mean10k) 2.05x, (l) 2.00x, (c) 1.98x, (g2meanR2) 1.67x and (g2mean1M) 1.65x (group_by_multi:mean from 7,500,000), (g1minmaxR2) 1.80x and (g1minmax1M) 1.56x (group_by:minmax from 2,184,147), (g1fsum10k) 1.75x and (g1fsum100k) 1.35x (group_by:sum from 5,252,956), (g1mean10k) 1.57x and (t6) 1.31x (group_by:mean from 2,436,520) and (g1count10k) 1.35x (group_by:count from 3,123,562); unique (r) 2.07x and (x1) 1.24x (below 5,494,090); the String shapes (p) 1.84x, (o) 1.39x and (y5) 1.31x (below the 5,000,000-row floor);
• at 50,000,000 rows: group-bys estimated at 11,180,716 to 16,775,168 groups, above every bucket taken for their class, (g2minmaxR2) 2.26x, (g2countR2) 2.11x, (g2sumR2) 1.59x, (g2fsumR2) 1.52x, (g2meanR2) 1.51x, (v4) 1.45x and (g1minmaxR2) 1.34x; group-bys in buckets not taken for their class, (g2sum200) 3.08x and (g2fsum200) 2.70x (two-key sum, 200 groups), (t2) 1.61x and (g1count200) 1.58x (one-key count, 200 groups) and (g1count1k) 1.57x (1,000 groups); the semi join (w3) 2.60x (join:semi is not taken, (f)); the top-k (n) 1.85x and (x3) 1.23x (top_k is not taken, (x4)).

The other 17 are 1.01x to 1.18x, inside the noise band.

Sorts with one key per Polars key. The 2026-09-26 default benchmark (Benchmarks/results/polars_engine_bench_2026-09-26-final3.csv) ran when a nulls-first nullable key and a descending float key still carried an extra key column each. With Polars' order as options of the one key, the sort and top-k cases and four group-by controls were timed old against new, alternating, three rounds of best of 5 after a 100 ms warm-up (Benchmarks/results/polars_engine_sort_options_2026-09-30.csv, run conditions in …_conditions.txt); best and median of the rounds, in ms:

The faster Polars engine in the same runs (best of the rounds of the new build, new_best_ms of the Polars rows): (x4) 9.75 ms at 50M and 1.21 ms at 2M (streaming), (n) 18.43 ms and 2.23 ms (streaming), (x2) 542.42 ms and 17.54 ms, (p) 978.94 ms and 22.02 ms (in-memory). The top-k cases are still left to Polars by the default (top_k has no crossover, above), so its rows for them are Polars' in either build. The sorts whose keys carry no option ((m), (o), (q), the Parquet (s4)), the top-k by an int64 key (x3), (y6), (b) and the group-by controls (c), (i), (j), (t1) are unchanged: every row that was more than 5% slower in both best and median over the three rounds, all of them on paths this change does not touch, was timed again alone over eight alternating rounds of 100 runs (the rows flagged row re-timed alone in the same file), where before ÷ after is 0.95 to 1.25 in best and 0.96 to 1.22 in median. No row's median process CPU time rose by more than 1.5x.

Group-bys after the grouped Float64 sum, mean and min/max were rebuilt. The 75 group-by cases (the grid's 60, (c), (i), (j), (l), (t1) to (t7), (v1) to (v4)) at 2,000,000 and 50,000,000 rows, MetalEngine() with the table fitted before the 2026-09-30 re-sweep and with the table fitted from it (before the default-benchmark rule), the same build, three alternating rounds of best of 5 after a 100 ms warm-up, the first run after 500 ms of idle recorded on its own: Benchmarks/results/polars_engine_default_groupby_2026-09-30.csv (per row best of the rounds and median of the rounds' medians; every run in …_raw_2026-09-30.csv, run conditions in …_2026-09-30_conditions.txt). Every answer was equal to Polars'. The re-swept table took 57 of the 150 case-size pairs (8 at 2,000,000 rows, 49 at 50,000,000), the earlier one 48, and it left none that the earlier one took. What it took that the earlier one left, in ms (best of the rounds; the earlier table's default ran Polars' in-memory plan there):

Every other pair it took was ahead of the faster Polars engine on the best run and on the median (new_vs_fastest_polars_best and _median in the same file): at 50,000,000 rows from 1.24x ((g1minmax1k)) to 10.69x ((g2count1M)), at 2,000,000 rows from 1.12x to 1.63x ((g2count10k)). Taken and behind at 2,000,000 rows were the two-key means (c) at 0.82x, (g2mean100k) at 0.65x and (g2mean10k), which the earlier table took as well, at 0.87x (5.83 against 5.08 ms; 0.73x under the earlier table). The default-benchmark rule (above) raises the two-key mean's 10,000 and 100,000 buckets to 50,000,000 rows, so the table in force takes 54 of the 150 pairs (5 at 2,000,000 rows, 49 at 50,000,000), every one at 1.0x or more of the faster Polars engine on the best run and the median in this benchmark, and leaves (c), (g2mean10k) and (g2mean100k) at 2,000,000 rows to Polars, (g2mean10k) being the one pair the earlier table took that this one leaves. Those three and (g2minmax10k), which shares (c)'s 10,000-group bucket of the min + max class, re-timed, the table before the rule against the table in force, three alternating rounds (Benchmarks/results/polars_engine_default_groupby_retime_2026-10-01.csv, run conditions in …_retime_2026-10-01_conditions.txt), in ms, best / median, against the faster Polars engine's best of 5.49, 4.98, 5.61 and 5.44 ms:

Left to Polars, MetalEngine() runs Polars' in-memory plan, which uses the CPU cores (77 to 90 CPU-ms against 4 to 6 on the GPU path for these three, new_cpu_ms and old_cpu_ms in the retime file). In this benchmark each engine's timed runs follow Polars' runs of the same case and a 500 ms idle gap, and these three take 4.75 to 9.79 ms (best of 5) under shapes="all", cold or warm, as well (the reported rounds of …_raw_2026-09-30.csv); run back to back, 15 runs of each way of collecting in turn, three rounds and no idle gap (Benchmarks/results/polars_engine_groupby_backtoback_2026-10-01.csv), the default runs them in 2.89 ms (c), 2.41 ms (g2mean10k) and 3.03 ms (g2mean100k) against Polars' streaming engine's 5.48, 5.09 and 5.74 ms, which is where the crossover sweep, measured the same back-to-back way, put them (2.09x, 2.19x and 2.34x at 2,000,000 rows in Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv). In the 150-pair benchmark the pairs whose decision did not change ran the same plan under both tables; the rows of those that were more than 5% slower in best and median under one table than the other ((g1count10k) and (g1count1M) at 50,000,000 rows, taken by both; (g2count100k) at 2,000,000, taken by both; (g1minmax100k) and (t4) at 2,000,000, Polars' plan under both) differ between the two runs of one plan.

Float64 group sums and means at 2^24 groups. The (v3) case of the crossover sweep, a mean over two keys with about as many groups as rows, found ArrowMetal's group-by returning null for most groups at 16,777,216 groups and above (16,777,215 were right): the per-group kernels dispatched one threadgroup per group and the grid wrapped past 2^32 threads. Fixed in the core (Findings against pyarrow, round 13; python/tests/test_group_by_2_24.py), so no engine rule is needed and every group-by shape follows the policy above. The sweep and the benchmark above ran with the fix: under shapes="all" the cases the guard once kept with Polars, (c), (l), (t3), (t6), (v3) and the Parquet cases (s1) and (s2), run on Metal and equal Polars' answers. At 50M rows, under shapes="all", cold (Benchmarks/results/polars_engine_bench_2026-09-26-final3.csv), (c) is 1.78, (l) 2.03 and (t6) 1.21 times the faster Polars engine; (t3) is behind at 35.5 ms against 12.4 and (v3) at 612.0 ms against 421.7.

am.MetalEngine()                          # shapes="measured": the crossovers and buckets above
am.MetalEngine(shapes="all")              # every translatable subtree of at least 1,000,000 rows
am.MetalEngine(shapes="all", min_rows=0)  # everything it can translate (what the tests use)
am.MetalEngine(shapes={"group_by", "sort"})  # the classes named (or their prefix), any size
am.MetalEngine(min_rows=5_000_000)        # the crossovers, and at least 5,000,000 rows
am.MetalEngine(raise_on_fail=True)        # raise instead of leaving anything to Polars

Each subtree the report lists as taken carries the rule that took it, and each node the policy left has a rule: line:

  metal:  Sort#2 [Sort > DataFrameScan] over 2,000,000 rows, ran in <t> ms -> 2,000,000 rows
          rule: 2,000,000 input rows is at or above the 1,000,000-row crossover for sort (crossover sweep, ...)
  polars: Select#3: rule: aggregate:sum was not measured ahead of Polars up to 50,000,000 input rows (...)
  polars: Join#2: rule: 900,000 input rows is below the 2,029,827-row crossover for join:inner (...)

shapes="all" is there for plans the benchmark did not cover and for moving work off the CPU cores: the CPU-ms column of the results file is the process CPU time of each run.

The import cache

Each Polars column the engine imports is kept, keyed by its Arrow type, length, offset and the address and size of every buffer, and the next query that reads the same column reuses the import. That is safe because the cached import holds the exported array and, through it, Polars' own buffer: while an entry lives, Polars can neither free that memory (so no other column can appear at the same address) nor write to it in place (Polars copies a buffer it shares before writing). Only a column that was imported without a copy is cached; a String column (handed over as views, without a copy, but outside the cache's key) and a multi-chunk column (concatenated on every export) are not. Entries are evicted least recently used above a byte budget, a quarter of physical memory by default:

from arrowmetal import polars_engine
polars_engine.import_cache_info()       # {"entries", "bytes", "limit", "hits", "misses"}
polars_engine.import_cache_limit(2 << 30)
polars_engine.clear_import_cache()      # and with it the Polars buffers it kept alive

The "warm" column of the results file is a second collect over the same frame; the defaults above were read from the cold one.

Parquet scans

lf = (pl.scan_parquet("trades.parquet")
        .filter(pl.col("price") > 500.0)
        .group_by("qty").agg(pl.col("weight").sum()))
lf.collect(engine=am.MetalEngine(shapes="all"))

A Scan of one local Parquet file becomes a leaf the engine reads itself, so the subtree above it needs no import at all: the columns are decoded on the GPU straight into Metal memory (Parquet on the GPU) and the plan runs over them.

• Projection. The columns Polars' projection pushdown left on the Scan are the reader's column list; no other column chunk is touched. Dictionary-encoded columns are read materialised, which is what Polars reads them as.
• Predicate. Polars pushes a filter into the Scan as its predicate. The engine translates it like any Filter and runs it on the GPU over the rows the reader returns. The comparisons in it that the file's statistics can judge the way Polars compares go to the reader as well, which skips row groups (and, with a page index, pages) that cannot hold a matching row: a comparison of a column with a literal, joined by &, on an integer column (all six operators), a String column (==, !=) or a float column (<, <= and == only, against a literal exact in the column's type). Polars orders NaN above every number, so NaN rows pass >, >= and !=, and writers leave NaN out of min/max; those three never reach the reader for a float column, so != x keeps a NaN in a row group whose statistics say x .. x (the case of apache/arrow#51491). |, functions and comparisons of two columns stay on the GPU filter only. A Filter Polars left directly above the Scan (with its predicate pushdown off) is handed to the reader the same way. scan_parquet(use_statistics=False) hands it nothing.
• Nulls. A column is treated as nullable unless the footer's statistics say it holds no null (ParquetFile.column_null_count), so a sort key by a column without nulls carries no null placement. The read checks the footer's word, and a file whose data holds nulls its statistics deny fails the query with ArrowMetalError rather than answering differently.
• The open file is kept. The reader goes through the open-file cache (Parquet on the GPU, "The open-file cache"), keyed by the file's path, inode, modification time and size, so the second query over a file does not map it again, and a rewritten file is read afresh. am.clear_parquet_cache() and am.parquet_cache_limit() control it.
• Checked once per plan and file. As for in-memory inputs, the plan first runs over a prefix of the file (64 rows of its first row group) and its output schema is compared with Polars'. A file whose stored Arrow schema Polars applies and ArrowMetal's reader does not (one with a different number of fields than the file, which Arrow's own reader ignores) fails that check or names a column the file does not have, and stays with Polars.
• A bare scan stays with Polars. A Scan with nothing above it that does GPU work is Polars' to read.

What stays with Polars, with the reason in the report: several files (a list, or a glob or directory that matches more than one), hive partition columns, a URL or cloud path (file:// included), row_index_name, n_rows (and a head, tail or slice Polars pushed into the scan), include_file_paths, schema=, deletion files, column mapping, a column whose dtype the plan does not carry (Decimal, Categorical, Enum, List, Struct, Binary, ...), a predicate that does not translate, and CSV and NDJSON scans. polars 1.44.1 cannot show an IPC scan to an engine (NodeTraverser.view_current_node raises NotImplementedError: ipc scan), so scan_ipc stays with Polars as well.

Each taken subtree's report entry lists what the reader did (scans: the file, the filters it was given, row groups read and skipped, pages skipped):

  metal:  GroupBy#2 [GroupBy > SimpleProjection > Scan] over 50,000,000 rows, ran in <t> ms -> 1,000 rows
          parquet /data/bench-none-50000000.parquet: filters [('id', '<', 5000000)], 5 row groups read, 45 skipped, 0 pages skipped

MetalEngine() judges a scan subtree by the same rule as an in-memory one, with the Parquet rows of the crossover table and the file's row count from its footer: a sort of a file of at least 1,500,000 rows is taken, and the filter, group-by and aggregate shapes over a file are not taken at any size ("Which translatable subtrees it runs: the defaults" above). The scan cases below are in line with that: the sort is ahead of both Polars engines cold and warm, and the filter, group-by and aggregate cases are behind cold.

The report

MetalEngine report for collect (polars 1.44.1, IR (14, 7))
  metal:  Sort#3 [Sort > HStack > Filter > DataFrameScan] over 10,000,000 rows, ran in <t> ms -> <n> rows
  polars: Select#1: Function rank has no ArrowMetal translation.

engine.last_report is a MetalPlanReport: taken (one entry per subtree that ran on Metal: the node at its top, the node kinds inside it, the rows it read, the ArrowMetal plan text, and after the run its wall time and output rows), fallbacks (one Kind#id: reason line per node that stayed with Polars for a reason of its own; every translation reason is a full sentence, which test_every_unsupported_reason_is_a_full_sentence checks, and a node the placement policy left has a rule: ... line, as above), walked (every node of the optimised plan), nodes, and path (the collect path it is for, "Collect paths" above). With POLARS_VERBOSE=1 the fallback lines are also issued as a PerformanceWarning. The report belongs to the engine object, so two threads collecting through one MetalEngine overwrite each other's.

Tests

PYTHONPATH=python python -m pytest python/tests/test_polars_engine.py -q
DIFF_QUICK=1 PYTHONPATH=python python -m pytest python/tests/test_polars_engine.py -q   # without the 100,003-row datasets
PYTHONPATH=python python -m pytest python/tests/test_engine_policy.py -q

test_engine_policy.py checks the default policy: the same decision for the same subtree across 200 calls and in a second process, every row of the crossover table one row below its crossover (left, with the reason) and at it (taken), a subtree needing every class's crossover, the String and Parquet rows, the router-table and sort-kernel floors, the overrides, a Parquet scan judged by its footer's row count under a filter that keeps a handful of rows, the report's rule lines, that the committed table is the fit of the results file it names, and the Float64 group-by guard at 2^24 groups. For the group-count buckets it checks that the buckets tile the counts, every bucket one row below its crossover and at it, a losing bucket named below or above the band, an estimate's range taken only where every count in it is, the fallback to the class row without an estimate, where the probe finds a group-by's keys, the probe's estimate in the true count's bucket over one int key, two and a String key at 2,000,000 rows, the same estimate across 200 calls and in a second process, the cache, and the engine taking a group-by in a winning bucket and leaving one in a losing bucket at the same rows.

The differential cases collect each LazyFrame on Polars and through MetalEngine(raise_on_fail=True, min_rows=0, shapes="all") -- so nothing may fall back -- and compare the frames: schema first, then values, exactly for integers, Booleans, Strings, nulls and element-wise floats, with a relative tolerance for float aggregates. The inputs are test_differential.py's generators at 0, 1, 33, 4,097 and 100,003 rows, null ratios 0, 0.3 and 1, its "sliced" and "special" (extremes, NaN, infinities, subnormals) flavours, and two Polars-side shapes, a two-chunk frame and a frame sliced with DataFrame.slice. Every numeric dtype runs through the comparisons, arithmetic, null logic, is_in, casts, group-by and whole-frame aggregates, sorts in both directions with nulls at both ends, and top-k; Boolean logic, String predicates and carried temporal columns have their own cases, and so do the four join kinds on integer, String and two-column keys with null keys and duplicates on both sides (a two-chunk right side among the shapes), suffix collisions, different key names, a join inside a filter-join-aggregate plan, and unique with and without a subset. Each fallback in the tables above has a case that checks the result is still Polars' and the report names the reason. One case reruns test_polars.py and test_lazy.py with every LazyFrame.collect() also collected through the engine (python/tests/metal_engine_everywhere.py) and requires the two to agree.

test_every_collect_path_is_explicit runs every row of the collect-path table and checks where the plan ran, the report's path and reason, and that the warning comes once. test_the_capability_table_is_current regenerates Polars engine capabilities and compares it with the committed file; no cell of it may be a Metal answer that differs from Polars' or an engine exception. test_an_unknown_node_kind_keeps_the_whole_plan_on_polars and test_a_node_polars_cannot_describe_keeps_the_whole_plan_on_polars check the fail-closed walk, and test_the_check_command_prints_versions_and_the_capability_header the command below.

python -m arrowmetal.polars_engine check prints the installed Polars version and whether it is one of TESTED_POLARS, the IR version against TESTED_IR_VERSION, whether the callback API is present, the IR node kinds of this Polars the engine does not know, whether ARROWMETAL_METAL_ENGINE is set, and the header of the capability table (in a source checkout). It exits 1 when the callback API is missing or the IR major differs, since then every plan stays with Polars.

The Parquet scan cases (212 tests) run fourteen scan shapes -- filters over every numeric dtype, Polars' float total order, !=, String predicates, projections, group-by on one and two keys, whole-file aggregates, sorts with nulls at both ends, top-k, a join with an in-memory frame and unique -- over one 20,011-row dataset with nulls, NaN, -0.0 and infinities written by pyarrow (snappy; uncompressed with a page index and no dictionary; ZSTD with v2 pages), by Polars and by DuckDB; a filter, a sort and an aggregate over the flat columns of 30 nested fixtures from all three writers; seventeen predicate-pushdown cases on every writer, each checking the answer, the filters the reader was given and the row groups it skipped, including NaN under >, >= and != (the apache/arrow#51491 shape, also on the pageindexnan__pa_constpage fixture); every fallback reason above; the open-file cache's reuse, invalidation and bounds; and footer null counts, including a file whose statistics deny its nulls.

The conformance grid. python/tests/engine_polars_grid.py generates the cases instead of choosing them: every shape the engine translates (filters, select and with_columns expressions, slice, sorts in both directions with nulls at both ends and over two keys, top-k, group-by with each aggregate with the column as the key and as the value, whole-frame aggregates, the four join kinds, unique) over every dtype it carries (the eight integer widths, Float32, Float64, Boolean, String, Date, Datetime in ms, us and ns and with a time zone, Duration in ms, us and ns, Time), with no, 5%, 70% or all nulls, at 0, 1, 7, 1,000 and 100,000 rows, plus the special values (integer extremes, NaN, infinities, subnormals, -0.0). Each case collects through MetalEngine(shapes="all", min_rows=0) and through Polars and compares the frames bit for bit. The sorts cover each direction with the nulls first and last, a stable sort in each direction (maintain_order=True: tied rows, the two zeros and every NaN keep their input order, compared bit for bit), two and three keys with a per-key null placement, a tie on one key falling through to the next, and top-k as sort().head(), top_k, bottom_k and over two keys. In the run recorded in Benchmarks/results/engine_conformance_2026-10-02.csv (the per-shape table in engine_conformance_2026-10-02_report.txt): 15,376 cases, 15,096 identical on Metal, 33 within the float-summation bound above, none different otherwise, and 247 where Polars' optimised plan had nothing to run (all of them sorts over 0 or 1 row, which Polars leaves out). python/tests/engine_report.py --engine polars reruns it and prints the per-shape table.

To verify on your own machine

1. DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift build -c release --product ArrowMetalC, then PYTHONPATH=python python -m pytest python/tests/test_polars.py python/tests/test_lazy.py -q: tiers 1 to 3 still green.
2. PYTHONPATH=python python -m pytest python/tests/test_polars_engine.py -q, after that fresh build.
3. In a REPL: import sys, arrowmetal as am and check "polars" not in sys.modules; then collect one plan through am.MetalEngine(shapes="all", min_rows=0) and read engine.last_report.
4. With POLARS_VERBOSE=1, collect a plan with an unsupported node (pl.col("v").rank()) and see the PerformanceWarning listing it.
5. am.zero_copy_report(df["v"]) on a single-chunk numeric column of a million rows says the two addresses are the same.
6. out, timings = engine.profile(lf) shows a metal: row, and PYTHONPATH=python python -m arrowmetal.polars_engine check prints the versions it was tested with.
7. Run PYTHONPATH=python python Benchmarks/polars_engine_bench.py --sizes 2000000,50000000 --out a.csv twice on a quiet machine, compare the two files, and read the taken and rule columns of the MetalEngine default, cold rows against the table above. --crossover with more sizes, then python Benchmarks/polars_engine_crossover.py a.csv --print, fits the table from your own run.
8. Read this section against engine.last_report on your machine.

---

Numbers

These are tiers 1 and 2, and tier 4 over a Parquet file at the end; tier 4's measurements over in-memory frames are in Benchmarks/results/polars_engine_bench_2026-10-02.csv and Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv (see "Tier 4" above).

Apple M4 Max, macOS 26.6.2, polars 1.44.1 (16 threads), pyarrow 25.0.1, ArrowMetal 0.1.0. Best of 5 runs after a warm-up, one process, one data set. Every figure below is from Benchmarks/results/polars_bench_50000000_2026-09-07.txt, except the String rows: those are ArrowMetal 0.2.0 with String columns handed over in Polars' view layout, best of 5, Benchmarks/results/polars_bench_strings_view_2026-09-26-quiet.txt (run conditions in Benchmarks/results/bench_conditions_2026-09-26-quiet.txt). Reproduce with:

PYTHONPATH=python python Benchmarks/polars_bench.py 50000000 5

Columns: k Int32 with 1000 distinct values, v Int64, amount Float64, name String drawn from 4096 distinct values. The "Polars" column is Polars' eager idiom; the lazy engine and pyarrow's Acero are on the Compare tab and in the benchmark matrix.

50M rows

The String hand-off of that column, 50M rows, in the same run:

Reading the table

• The "Polars" column is Polars' eager idiom, which is what Benchmarks/polars_bench.py measures and what Benchmarks/results/polars_bench_50000000_2026-09-07.txt records. The published baseline is the parallel run, Benchmarks/results/full_matrix_2026-09-07-parallel.csv, which adds Polars' lazy engine (polars-lazy, a median of 11.5 of the 16 cores); ratios there are lower on the rows where the lazy engine parallelises. The full matrix has both.
• "GPU-resident" is the same kernel with the column already in Metal memory -- the import is outside the timed region, and, for the group-by rows, the am.group_by([k]) key-mapping pass as well: those rows time the aggregate only. It is what a pipeline that stays on the GPU sees, and it is the column that shows what the kernels are worth.
• The group-by row is not a like-for-like ratio against Polars, whose 79.3 ms includes its whole hash group-by. The comparable end-to-end figure is in the benchmark matrix: sum by int32 key (1000 groups) at 50M rows is 4.89 ms against Polars lazy's 81.93 ms, 16.8x (Benchmarks/results/full_matrix_2026-09-07-parallel.csv), and 1.73 ms against 20.24 ms, 11.7x, at 10M. The fastest CPU idiom on that row is pyarrow's Acero (pyarrow-threaded), 18.45 ms at 50M, which puts the ratio at 3.8x.
• The hand-off is the whole difference between the middle columns and the right one. At 50M rows the import is 5.25 ms and the export 0.01 ms, and the run records the import as a no-copy -- the Polars source buffer and the Metal buffer are the same address (zero copy: ... -> SAME). Every tier-1 and tier-2 row pays it once per call, so on a single sum Polars is ahead, and a group-by is 4x.
• CPU-ms is the other half of the story. The 50M group-by costs Polars 1121 CPU-ms across 16 threads; ArrowMetal costs 14.0 CPU-ms end to end and 0.4 CPU-ms resident. On a laptop that is battery, and on a shared box it is 16 cores left free for something else.
• Strings cross in Polars' own Utf8View layout: 50M rows hand over in 30.0 ms with no bytes copied, against 309.0 ms through large_string (offsets + bytes), which is a copy. Through the namespace contains is 17.6x with views and 2.1x with offsets; resident it is 112.4x (106.9x over offsets). The tier-2 plugin still asks Polars for large_string and is 2.3x. Keeping a string column resident (s.arrowmetal.to_metal()) takes the 38.1 ms call to 6.0 ms.

Where each tier is worth using

Tier 4 over a Parquet file, 50M rows

pl.scan_parquet(file) under a filter, a group-by, an aggregate or a sort, over the 50,000,000-row, 8-column files of Benchmarks/parquet_bench.py (snappy, 1.65 GB, and uncompressed, 2.23 GB), collected by Polars' in-memory and streaming engines and by MetalEngine(shapes="all", min_rows=0). "cold" clears the open-file cache before every run, so the engine opens and maps the file each time; "warm" keeps it open between runs. Polars reads the file on every run; the file stays in the OS page cache throughout. Best of 5, every engine result equal to Polars', Benchmarks/results/polars_engine_scan_2026-09-26-quiet.csv (run conditions in Benchmarks/results/bench_conditions_2026-09-26-quiet.txt).

• The sort is ahead cold and warm: 4.4x (snappy) and 7.6x (uncompressed) the faster Polars engine cold, 4.7x and 8.7x warm, and the default takes it (4.4x and 7.5x).
• Cold, the other three are behind and to improve, from 35.0 ms against 12.1 ((s2), snappy) to 13.3 ms against 13.1 ((s3), uncompressed) (vs_fastest_polars). The cold runs include what the open-file cache removes: opening the file and handing the pages the query reads to Metal; a one-column read of these files is 6.9-12.2 ms through a fresh open and 3.2-7.6 ms through the cache (Parquet on the GPU, "The open-file cache"). The default leaves them to Polars.
• Warm, over the uncompressed file (s3) is ahead of Polars' streaming engine (5.9 ms against 13.1, 2.2x) and so is (s1) (31.6 ms against 36.1, 1.1x); (s2) is behind at 6.3 ms against 5.1. Over the Snappy file, which the GPU decompresses first (Parquet on the GPU, "What the numbers say"), (s1) is behind at 91.3 ms against 45.6, (s2) at 33.4 against 12.1 and (s3) at 13.9 against 13.7.
• CPU time: the warm engine runs cost 2.9-7.8 ms of process CPU, against 37.5-6733.8 ms for Polars (cpu_ms).

PYTHONPATH=python python Benchmarks/polars_engine_bench.py --scan-only --scan-rows 50000000 \
    --scan-codecs snappy,none --out scan.csv

---

Limits

Types. This paragraph is about what the bridge round-trips, which is a tier-1 and tier-3 question; tier 2's expressions accept a shorter list, set out in "What tier 2 accepts" above. Every dtype Polars and Arrow share round-trips: all signed and unsigned integer widths, Float32/64, Boolean, Date, Datetime (all units), Time, Duration, String, Binary. Categorical and Enum also work, but Polars encodes them as dictionary<uint32> and dictionary<uint8> while ArrowMetal wants int32 or int64 indices, so the bridge recodes the index buffer -- 4 bytes a row, values untouched. An Enum comes back from to_polars as a Categorical: the dictionary crosses, the fact that its value set was closed does not. List and Struct cross and round-trip, but only the structural kernels operate on them -- you cannot sum or group by one. Object is not bridged.

Chunking. am.from_polars takes one Arrow array. A multi-chunk Series is rechunked once, which does copy; am.from_polars(s, rechunk=False) raises instead, so the copy is never silent. The MetalEngine hands a multi-chunk column of an in-memory frame over as its own chunks, which the chunked import (am_import_chunks) copies straight into the final Metal buffers without Polars concatenating them first; Categorical, Enum and nested columns are still concatenated by to_arrow.

Order. Group order is ArrowMetal's, not Polars': ascending by key for numeric, boolean, temporal and decimal keys, first-seen for utf8 and binary, lexicographic in column order for several keys. Polars' group_by promises no order at all, so sort both sides before comparing. s.arrowmetal.unique() is first-seen, the order Polars' unique(maintain_order=True) gives, and it keeps a null as one of the distinct values rather than dropping it. Sorts put nulls last in both directions, where Polars' ascending default is nulls first -- pass nulls_last=True when comparing.

Strings. A String or Binary column crosses in Polars' own Utf8View / BinaryView layout (to_arrow(compat_level=newest)): its 16-byte views and data buffers are mapped into Metal without a copy, and export hands them back to Polars the same way. s.arrowmetal.to_metal().string_layout says "view". The string kernels read the views directly (the list is am.STRING_VIEW_KERNELS); the rest (am.STRING_VIEW_CONVERTS: str_concat, the splits, match_like, strptime, the byte-counting pads, utf8_zero_fill, the byte slices and reversals, casts from strings, the file writers) convert the column to offsets + bytes once, on the GPU, and keep that form, which string_layout then reports as "view (converted)". am.from_polars(s, string_layout="offsets") asks Polars for large_string instead, the layout used before the kernels read views; the tier-2 plugin still uses that. upper/lower are Unicode's simple 1:1 case mapping over every script (the GPU table covers U+0000–U+017F exactly; a row holding anything above is mapped on the host), so Greek and Cyrillic come back mapped. Simple, not full: the multi-character expansions are not applied, so U+00DF becomes ẞ where Polars' str.to_uppercase() gives SS, and fi stays put. contains / starts_with / ends_with are literal, not regex.

Arithmetic. .add/.sub/.mul/.truediv in tier 2 keep the column's own type and follow Arrow's unchecked rules for the operation: integers wrap, and integer division by zero yields 0 where Polars raises. The operand is checked -- one the column's type cannot hold raises rather than being clamped or truncated, and an integer operand is exact past 2^53. See "The scalar in .add" above.

Joins. GPU path only for a single-column inner or left join against a unique right key; everything else falls back to Polars (or raises with allow_cpu_fallback=False).

Aggregation inside group_by. Tier 2's group_by_sum is a projection, not a hash aggregate Polars calls per group -- the plugin API has no hook for that. See the tier-2 section.

Threads. ArrowMetal serialises command-buffer commits behind its own lock and keeps its error state thread-local, so Polars is free to call the plugin from several worker threads. The plugin takes no lock of its own.

Version coupling. Tier 2: the plugin is pinned to polars 0.55.1 / pyo3-polars 0.28 for py-polars 1.44.x. Tiers 1 and 3 speak the C Data Interface and are not coupled to a Polars version. Tier 4 is pure Python but reads Polars' optimised IR through an API Polars calls unstable; it was written against IR version (14, 7) of polars 1.44.1, a test pins both, and a different IR major makes it leave every plan to Polars.

---

Tests

PYTHONPATH=python python -m pytest python/tests/test_polars.py -q     # 90 tests
cd polars-plugin/arrowmetal-sys && cargo test --release               # 10 tests
PYTHONPATH=python python -m pytest python/tests/test_polars_engine.py -q   # tier 4

test_polars_engine.py is described under "Tier 4", "Tests".

test_polars.py covers round trips for every shared dtype (plus Categorical, Enum, empty, all-null and chunked), the namespace methods against native Polars at five sizes from 0 to 100,003 rows, the plugin expressions inside lazy plans, the join against pl.DataFrame.join, zero-copy assertions on buffer addresses, and 50M-row timings as assertions with bounds generous enough not to flake. The plugin tests skip when the Rust library has not been built, so a checkout without a Rust toolchain still runs green -- build it before you trust a green run, or 28 of the 90 are skips (62 passed, 28 skipped).

The tier-2 block at the end of the file is the hostile-input pass: the scalar operand against the tier-1 bridge, nulls against native Polars, a sliced Series and a two-chunk one, an empty frame, an all-set validity bitmap with no nulls, the twelve dtypes the numeric expressions refuse (each one a Polars error carrying ArrowMetal's wording, never a pyo3 panic), and the expression inside group_by().agg() and inside collect(engine="streaming").

Generated by python/tests/engine_capabilities.py from runs of the engine; test_polars_engine.py fails when this file differs from a fresh run.

• Polars 1.44.1, IR (14, 7)
• ArrowMetal 0.4.0, commit af9072a7f7
• Engine: MetalEngine(shapes="all", min_rows=0), every node it can translate (the default MetalEngine() takes a subset: Polars, "Which translatable subtrees it runs: the defaults")
• Input: a 64-row frame with the column x under test, k (Int32) and v (Int64); null patterns: none, some (every third row null), all

Cells: Metal -- the whole plan ran on Metal and matched Polars; Polars [n] -- the plan stayed with Polars for reason n below; partly [n] -- part of the plan ran on Metal, the rest stayed with Polars for reason n; - -- Polars' optimised plan had nothing to run; n/a -- Polars itself rejects the plan for this dtype.

Summary

filter_other_column

filter(k > 1): a Filter that carries x

filter_is_not_null

filter(x.is_not_null())

filter_eq_literal

filter(x == <a value of x>)

filter_gt_literal

filter(x > <a value of x>)

select_add

select(x + x, v)

select_is_null

select(x.is_null(), v)

with_columns

with_columns(k * 2): an HStack that carries x

slice

filter(k > 1).slice(2, 10): a Slice that carries x

sort

sort(x) (nulls first)

sort_desc_nulls_last

sort(x, descending=True, nulls_last=True)

top_k

sort(x).head(5)

group_by_key

group_by(x).agg(v.sum()): x as the key

group_by_value_sum

group_by(k).agg(x.sum()): x as the value

group_by_value_min

group_by(k).agg(x.min()): x as the value

group_by_value_max

group_by(k).agg(x.max()): x as the value

group_by_value_mean

group_by(k).agg(x.mean()): x as the value

group_by_value_count

group_by(k).agg(x.count()): x as the value

aggregate_sum

select(x.sum()): a whole-frame aggregate

aggregate_min

select(x.min()): a whole-frame aggregate

aggregate_max

select(x.max()): a whole-frame aggregate

aggregate_mean

select(x.mean()): a whole-frame aggregate

aggregate_count

select(x.count()): a whole-frame aggregate

join_inner

join(right, on=x, how="inner"): x as the key

join_left

join(right, on=x, how="left"): x as the key

join_semi

join(right, on=x, how="semi"): x as the key

join_anti

join(right, on=x, how="anti"): x as the key

unique

unique(subset=[x])

parquet_filter

scan_parquet(f).filter(k > 1): a Parquet Scan that reads x

parquet_sort

scan_parquet(f).sort(x)

Reasons

As the engine's report gives them (Kind: reason, for the first node that stayed with Polars).

1. DataFrameScan: Column 'x' has dtype Decimal(precision=10, scale=2), which the Metal plan does not carry.
2. DataFrameScan: Column 'x' has dtype Categorical, which the Metal plan does not carry.
3. DataFrameScan: Column 'x' has dtype Enum(categories=['apple', 'pear', 'fig', 'kiwi', 'plum']), which the Metal plan does not carry.
4. DataFrameScan: Column 'x' has dtype Binary, which the Metal plan does not carry.
5. DataFrameScan: Column 'x' has dtype List(Int64), which the Metal plan does not carry.
6. DataFrameScan: Column 'x' has dtype Array(Int64, shape=(2,)), which the Metal plan does not carry.
7. DataFrameScan: Column 'x' has dtype Struct({'a': Int64}), which the Metal plan does not carry.
8. DataFrameScan: Column 'x' has dtype Null, which the Metal plan does not carry.
9. Filter: Column 'x' of dtype Date cannot be read by a fused expression.
10. Filter: Column 'x' of dtype Datetime(time_unit='ms', time_zone=None) cannot be read by a fused expression.
11. Filter: Column 'x' of dtype Datetime(time_unit='us', time_zone=None) cannot be read by a fused expression.
12. Filter: Column 'x' of dtype Datetime(time_unit='ns', time_zone=None) cannot be read by a fused expression.
13. Filter: Column 'x' of dtype Datetime(time_unit='us', time_zone='UTC') cannot be read by a fused expression.
14. Filter: Column 'x' of dtype Duration(time_unit='ms') cannot be read by a fused expression.
15. Filter: Column 'x' of dtype Duration(time_unit='us') cannot be read by a fused expression.
16. Filter: Column 'x' of dtype Duration(time_unit='ns') cannot be read by a fused expression.
17. Filter: Column 'x' of dtype Time cannot be read by a fused expression.
18. Filter: The string comparison Gt is not translated.
19. Select: Function StringFunction.ConcatHorizontal has no ArrowMetal translation.
20. Select: Column 'x' of dtype Duration(time_unit='ms') cannot be read by a fused expression.
21. Select: Column 'x' of dtype Duration(time_unit='us') cannot be read by a fused expression.
22. Select: Column 'x' of dtype Duration(time_unit='ns') cannot be read by a fused expression.
23. Select: Column 'x' of dtype Date cannot be read by a fused expression.
24. Select: Column 'x' of dtype Datetime(time_unit='ms', time_zone=None) cannot be read by a fused expression.
25. Select: Column 'x' of dtype Datetime(time_unit='us', time_zone=None) cannot be read by a fused expression.
26. Select: Column 'x' of dtype Datetime(time_unit='ns', time_zone=None) cannot be read by a fused expression.
27. Select: Column 'x' of dtype Datetime(time_unit='us', time_zone='UTC') cannot be read by a fused expression.
28. Select: Column 'x' of dtype Time cannot be read by a fused expression.
29. GroupBy: A group_by on a float key is not translated (NaN and -0.0 grouping).
30. GroupBy: The aggregate 'x' has dtype Date, which the Metal plan does not compute.
31. GroupBy: The aggregate 'x' has dtype Datetime(time_unit='ms', time_zone=None), which the Metal plan does not compute.
32. GroupBy: The aggregate 'x' has dtype Datetime(time_unit='us', time_zone=None), which the Metal plan does not compute.
33. GroupBy: The aggregate 'x' has dtype Datetime(time_unit='ns', time_zone=None), which the Metal plan does not compute.
34. GroupBy: The aggregate 'x' has dtype Datetime(time_unit='us', time_zone='UTC'), which the Metal plan does not compute.
35. GroupBy: The aggregate 'x' has dtype Duration(time_unit='ms'), which the Metal plan does not compute.
36. GroupBy: The aggregate 'x' has dtype Duration(time_unit='us'), which the Metal plan does not compute.
37. GroupBy: The aggregate 'x' has dtype Duration(time_unit='ns'), which the Metal plan does not compute.
38. GroupBy: The aggregate 'x' has dtype Time, which the Metal plan does not compute.
39. GroupBy: A group_by min over Float64 is not translated (ArrowMetal's GroupBy min/max kernels take up to 32-bit floats).
40. GroupBy: The min of a boolean is not translated.
41. GroupBy: The aggregate 'x' has dtype String, which the Metal plan does not compute.
42. GroupBy: A group_by max over Float64 is not translated (ArrowMetal's GroupBy min/max kernels take up to 32-bit floats).
43. GroupBy: The max of a boolean is not translated.
44. GroupBy: The mean of a boolean is not translated.
45. GroupBy: The aggregate count over dtype String is not translated.
46. GroupBy: Column 'x' of dtype Date cannot be read by a fused expression.
47. GroupBy: Column 'x' of dtype Datetime(time_unit='ms', time_zone=None) cannot be read by a fused expression.
48. GroupBy: Column 'x' of dtype Datetime(time_unit='us', time_zone=None) cannot be read by a fused expression.
49. GroupBy: Column 'x' of dtype Datetime(time_unit='ns', time_zone=None) cannot be read by a fused expression.
50. GroupBy: Column 'x' of dtype Datetime(time_unit='us', time_zone='UTC') cannot be read by a fused expression.
51. GroupBy: Column 'x' of dtype Duration(time_unit='ms') cannot be read by a fused expression.
52. GroupBy: Column 'x' of dtype Duration(time_unit='us') cannot be read by a fused expression.
53. GroupBy: Column 'x' of dtype Duration(time_unit='ns') cannot be read by a fused expression.
54. GroupBy: Column 'x' of dtype Time cannot be read by a fused expression.
55. Select: The aggregate 'x' has dtype Duration(time_unit='ms'), which the Metal plan does not compute.
56. Select: The aggregate 'x' has dtype Duration(time_unit='us'), which the Metal plan does not compute.
57. Select: The aggregate 'x' has dtype Duration(time_unit='ns'), which the Metal plan does not compute.
58. Select: The min of a boolean is not translated.
59. Select: The aggregate 'x' has dtype String, which the Metal plan does not compute.
60. Select: The aggregate 'x' has dtype Date, which the Metal plan does not compute.
61. Select: The aggregate 'x' has dtype Datetime(time_unit='ms', time_zone=None), which the Metal plan does not compute.
62. Select: The aggregate 'x' has dtype Datetime(time_unit='us', time_zone=None), which the Metal plan does not compute.
63. Select: The aggregate 'x' has dtype Datetime(time_unit='ns', time_zone=None), which the Metal plan does not compute.
64. Select: The aggregate 'x' has dtype Datetime(time_unit='us', time_zone='UTC'), which the Metal plan does not compute.
65. Select: The aggregate 'x' has dtype Time, which the Metal plan does not compute.
66. Select: The max of a boolean is not translated.
67. Select: The mean of a boolean is not translated.
68. Select: The aggregate count over dtype String is not translated.
69. Join: A join on a float key is not translated (NaN and -0.0 equality).
70. Distinct: A unique over a float column is not translated (NaN and -0.0 equality).
71. Scan: Column 'x' has dtype Decimal(precision=10, scale=2), which the Metal plan does not carry.
72. Scan: Column 'x' has dtype Categorical, which the Metal plan does not carry.
73. Scan: Column 'x' has dtype Enum(categories=['apple', 'pear', 'fig', 'kiwi', 'plum']), which the Metal plan does not carry.
74. Scan: Column 'x' has dtype Binary, which the Metal plan does not carry.
75. Scan: Column 'x' has dtype List(Int64), which the Metal plan does not carry.
76. Scan: Column 'x' has dtype Array(Int64, shape=(2,)), which the Metal plan does not carry.
77. Scan: Column 'x' has dtype Struct({'a': Int64}), which the Metal plan does not carry.
78. Scan: Column 'x' has dtype Null, which the Metal plan does not carry.

DuckDB is an excellent query engine. It reads Parquet, plans joins, pushes down predicates, keeps statistics, spills to disk, and saturates every core you have. ArrowMetal is not a query engine at all: it is a pile of Metal kernels that run one columnar operation very fast on the GPU that is already sitting in your Mac.

Those are complementary, and they meet for free. DuckDB hands results out over the Arrow C Data interface; ArrowMetal takes Arrow buffers in. On Apple silicon the CPU and GPU share one physical memory pool, so am_import copies nothing - it wraps DuckDB's own pages in a Metal buffer. The address on the far side is the same address. python/tests/test_duckdb.py asserts it. Getting to one contiguous chunk does copy when DuckDB returns many; see §5.

The summary. The Python bridge, at 50M rows: a 100,000-key group-by is 16x faster than DuckDB's once the column and its group ids are already on the GPU (1.1x for a single cold query), and a string LIKE scan is 9.3x resident (0.6x cold). DuckDB is ahead on a plain sum over a column, which is memory-bound and which DuckDB does while it scans. The loadable extension works and matches DuckDB's answers exactly; it is behind DuckDB's own SQL as measured, and §4 says why.

A third piece needs no change to the SQL at all. The rewrite extension is a DuckDB optimizer extension: loaded into a connection, it moves eligible aggregates of ordinary queries onto the GPU, with DuckDB's exact answers, and in its default mode only for the shape classes whose every benchmarked query was ahead of DuckDB's own operators, from the size where that held. §4b has what it rewrites, the feasibility finding behind it, and where it is ahead and where it is not.

• 1. Which tier you want
• 2. Install
• 3. Tier 1: the Python bridge
• 4. Tier 2: the loadable extension
• 4b. Tier 3: ordinary SQL, rewritten
• 5. Numbers
• 6. When this is worth it, and when it is not
• 7. Limits

---

1. Which tier you want

If you are reading this to make something faster: use tier 1, or tier 3 when the SQL must stay as it is and its aggregates fall in §4b's table. Tier 2 exists because "call it from SQL" is a real requirement for some people.

---

2. Install

Both tiers need the library built:

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift build -c release --product ArrowMetalC

Tier 1

pip install duckdb
PYTHONPATH=python python -c "import duckdb, arrowmetal as am; print(am.device_name())"

The bridge is imported lazily, so import arrowmetal still works with no duckdb installed; the am.from_duckdb attribute is what pulls it in.

Tier 2

./duckdb-extension/build.sh

That compiles one translation unit, links libArrowMetalC.dylib, and appends the 512-byte metadata footer DuckDB checks before it will dlopen anything. On the first run it downloads duckdb.h and duckdb_extension.h into duckdb-extension/third_party/; afterwards it needs no network. duckdb-extension/CMakeLists.txt drives the same compile if you would rather use cmake:

cmake -S duckdb-extension -B duckdb-extension/build -DCMAKE_BUILD_TYPE=Release
cmake --build duckdb-extension/build

Then load it. Unsigned extensions have to be allowed explicitly, and that is a per-connection config setting, not a SET you can issue afterwards:

import duckdb
con = duckdb.connect(config={"allow_unsigned_extensions": "true"})
con.execute("LOAD 'duckdb-extension/build/arrowmetal.duckdb_extension'")
con.sql("SELECT arrowmetal_version(), arrowmetal_device()").show()

From the CLI: duckdb -unsigned, then LOAD '.../arrowmetal.duckdb_extension';.

Versions. Built and tested against DuckDB 1.5.6 (source_id 069cc9f9b5), platform osx_arm64, pyarrow 25.0.1, on an Apple M4 Max. The extension declares the C extension API version v1.2.0, not the DuckDB version - a C_STRUCT extension is compatible with any DuckDB whose C API is at least what it declares, so it is not pinned to 1.5.6. The platform string is checked strictly: an osx_arm64 build will not load into an osx_amd64 DuckDB. Rebuild with DUCKDB_PLATFORM=... ./duckdb-extension/build.sh to target another one.

Tier 3

PYTHON=/path/to/the/python/with/duckdb ./duckdb-extension/build_rewrite.sh

It reads the DuckDB release and source_id from that Python's duckdb module, shallow-clones the matching DuckDB tag into duckdb-extension/build/duckdb-<version>/ on first run (only the headers are used; nothing of DuckDB is compiled), checks the clone's commit against the source_id, compiles src/arrowmetal_rewrite.cpp, checks that the module exports every DuckDB symbol the extension needs, and stamps the footer with the CPP ABI and that exact release. The result is duckdb-extension/build/arrowmetal_rewrite.duckdb_extension. Built and tested against DuckDB 1.5.6 (source_id 069cc9f9b5) in the duckdb 1.5.6 Python module.

import arrowmetal as am
con = am.duckdb_connect()     # duckdb.connect + allow_unsigned_extensions + LOAD + SET arrowmetal_rewrite = 'auto'

or by hand: duckdb.connect(config={"allow_unsigned_extensions": "true"}), then LOAD '.../arrowmetal_rewrite.duckdb_extension'.

---

3. Tier 1: the Python bridge

python/arrowmetal/duckdb_bridge.py. Nine entry points, all reachable straight off am; six of them are below, and the other three are duckdb_reader, duckdb_table and duckdb_is_zero_copy.

Pull a result onto the GPU

import duckdb, arrowmetal as am
con = duckdb.connect()

cols = am.from_duckdb(con.sql("select region, amount from sales where placed_at >= date '2024-01-01'"))
# -> {"region": MetalArray, "amount": MetalArray}

from_duckdb takes a DuckDBPyRelation, a SQL string with con=, a connection holding a pending result, or any pyarrow table or stream. Columns whose type ArrowMetal cannot lift are left as pyarrow arrays in the same dict; pass on_unsupported="raise" to be told instead.

test_every_duckdb_type_round_trips checks 25 of DuckDB's types: every integer width and signedness, both floats, BOOLEAN, VARCHAR, BLOB, DATE, TIME, TIMESTAMP, TIMESTAMPTZ, INTERVAL, DECIMAL(10,2), DECIMAL(38,2), HUGEINT, LIST, STRUCT and MAP. UUID is checked as its VARCHAR rendering; the native type is not exercised.

Push results back

gb = am.group_by([cols["region"]])
rel = am.to_duckdb(con, "gpu_totals", {"region": gb.keys()[0], "total": gb.sum(cols["amount"])})

con.sql("select * from gpu_totals order by total desc limit 10").show()
con.sql("select r.name, g.total from gpu_totals g join regions r on r.id = g.region").show()

to_duckdb registers through con.register, which takes the Arrow table over the C Data interface - DuckDB reads the GPU-produced buffers in place. The view behaves like any other relation: join it, filter it, insert from it. It lives until con.unregister(name).

Both halves at once

rel = am.duckdb_gpu_query(
    con,
    """select o.customer, o.amount
       from orders o join customers c on c.id = o.customer
       where o.placed_at >= date '2024-01-01'""",
    then=lambda cols: {
        "customer": am.group_by([cols["customer"]]).keys()[0],
        "total":    am.group_by([cols["customer"]]).sum(cols["amount"]),
    },
    name="per_customer")

rel.order("total desc").limit(10).show()

DuckDB does the scan, the predicate and the join. ArrowMetal does the aggregate. That is the whole idea, and it is the arrangement that pays: the join is a planning problem DuckDB is good at, and the group-by over the join's output is a throughput problem the GPU is good at.

A fused query over a DuckDB result

am.query compiles a whole filter-and-aggregate expression into one Metal kernel, so the column is read once no matter how many operators the expression has (see Fused expressions):

q = am.filter((am.col("amount") > 100) & am.col("region").is_in([2, 7])).aggregate([
        ("sum", "total", am.col("amount")),
        ("count", "n", None)])
am.query(am.from_duckdb(con.sql("select region, amount from sales")), q)
# {"total": ..., "n": ...}

Tables larger than memory

fetch_record_batch gives an Arrow C Stream that produces batches lazily. am.duckdb_batches iterates it as {name: MetalArray} with only one batch resident at a time:

total = 0
for batch in am.duckdb_batches(con.sql("select amount from huge_parquet"), rows_per_batch=1 << 22):
    total += batch["amount"].sum()

Two wrappers do the merging for you:

am.duckdb_aggregate(con.sql("select amount, region from huge"),
                    {"total": ("sum", "amount"), "n": ("count", "amount"),
                     "regions": ("count_distinct", "region")},
                    rows_per_batch=1 << 22)

am.duckdb_group_by(con.sql("select region, amount from huge"), "region",
                   {"total": ("sum", "amount"), "avg": ("mean", "amount")})

What is exact, and what is refused. This is the part that usually goes wrong quietly elsewhere, so the bridge is explicit about it:

test_streaming_aggregate_is_batch_size_independent runs the same sum at 3,000 / 30,000 / 1,000,000 rows per batch and requires all three answers to be identical, and identical to DuckDB's.

---

4. Tier 2: the loadable extension

duckdb-extension/. Built against DuckDB's C extension API, which means it needs no DuckDB source tree, no C++ ABI match with the host engine, and no rebuild for each DuckDB patch release. The loader hands it a struct of function pointers and everything after that is plain C.

The functions

SELECT arrowmetal_version();                          -- '0.4.0'
SELECT arrowmetal_device();                           -- 'Apple M4 Max'

SELECT * FROM arrowmetal_agg('sales', 'amount');      -- sum, count, min, max, mean
SELECT * FROM arrowmetal_group_by('sales', 'region', 'amount');  -- key, count, sum, min, max
SELECT * FROM arrowmetal_top_k('sales', 'amount', 100);          -- the 100 largest, descending
SELECT * FROM arrowmetal_sort('sales', 'amount');                -- every value, ascending
SELECT * FROM arrowmetal_query('sales',
    '(query (filter (gt (col "amount") (int 100)))
            (aggregate (sum "total" (col "amount")) (count "n")))');

The results are ordinary relations, so they join:

SELECT r.name, g.sum
FROM arrowmetal_group_by('sales', 'region', 'amount') g
JOIN regions r ON r.id = g.key
ORDER BY g.sum DESC;

arrowmetal_query takes the s-expression from Fused expressions and returns one row per aggregate: name, value (a DOUBLE) and exact (a BIGINT, non-null when the aggregate is integral). Both columns exist because a sum past 2^53 is exact in exact and rounded in value, and pretending otherwise would silently corrupt large integer totals.

Aggregate functions - SELECT am_sum(x) FROM t - are deliberately not registered even though duckdb_create_aggregate_function exists. DuckDB would call them once per 2048-row DataChunk, and a 2048-row GPU dispatch is all latency and no work; the CPU would be ahead and it would look like ArrowMetal's fault. Table functions get the whole column at once, which is the only shape where a GPU dispatch amortises.

It is correct, and DuckDB's own SQL is ahead of it

Every function's answer matches DuckDB's own SQL exactly, nulls included - python/tests/test_duckdb.py checks each one against the equivalent query on the same connection.

The header compiles: am_plan_source is the typedef and am_plan_source_create the function. test_the_public_c_header_compiles compiles the public header as C on every run; the @extension tests skip only when the extension has not been built. Tier 1 does not go through the header and is unaffected either way.

And plain SQL is ahead of it. The extension's timings are not recorded: Benchmarks/duckdb_bench.py measures the Python bridge only, and python/tests/test_duckdb.py checks the extension's answers without timing them.

The same GPU group-by from the Python bridge runs in 4.5 ms at 50M rows (§5), so the time is in the extension's path. Three things account for it:

1. The DataChunk assembly is single-threaded. DuckDB emits ~2048-row chunks and one ArrowMetal array is one contiguous buffer, so the extension memcpys chunk after chunk on one thread while DuckDB's own aggregate is running on all of them. This is the dominant cost.
2. duckdb_query materialises the whole result first, so the rows are copied once inside DuckDB before the extension has touched them.
3. The extension re-reads the table on every call, because a table function has nowhere to cache. The bridge pays the crossing once and then runs twenty operations on resident data.

---

4b. Tier 3: ordinary SQL, rewritten

duckdb-extension/src/arrowmetal_rewrite.cpp is a DuckDB optimizer extension. The query does not change. After DuckDB's own optimizers have run, the extension looks at every aggregate in the plan, and where the shape is one it answers exactly (and, in the default mode, the size is one where the GPU was measured ahead) it puts ARROWMETAL_AGGREGATE where DuckDB's HASH_GROUP_BY or UNGROUPED_AGGREGATE would have been. The scan, the pushed-down filters and the projections below it are still DuckDB's, planned and run exactly as before; so is everything above it.

LOAD 'duckdb-extension/build/arrowmetal_rewrite.duckdb_extension';

SELECT region, sum(amount), count(*), max(amount) FROM sales GROUP BY region;   -- as written

EXPLAIN SELECT region, sum(amount) FROM sales GROUP BY region;   -- ARROWMETAL_AGGREGATE when rewritten
SELECT * FROM arrowmetal_rewrites();    -- every decision, with the reason, the threshold and the GPU path

SET arrowmetal_rewrite = 'auto';        -- the default: supported shapes, at the sizes measured faster
SET arrowmetal_rewrite = 'off';         -- plain DuckDB
SET arrowmetal_rewrite = 'force';       -- every supported shape, whatever its size

The three values are read in any letter case, and 'off' is also spelled 'false' or '0'. Any other value ('on', 'true', ' force' with a space, NULL) is an error from the SET itself, arrowmetal_rewrite: unrecognised value 'on'; expected 'auto', 'off' (also 'false' or '0') or 'force', and the mode stays what it was. RESET arrowmetal_rewrite returns to 'auto'.

From Python, am.duckdb_connect() opens a connection with the extension loaded, am.duckdb_is_rewritten(con, sql) says whether the plan of sql has an ARROWMETAL_AGGREGATE, and am.duckdb_rewrites(con) returns the decision log as a pyarrow.Table.

Why a C++ extension: the feasibility finding

• DuckDB's C extension API has no optimizer hook. Tier 2 is built on it. Its header (duckdb_extension.h, the v1.2.0 C API that DuckDB 1.5.6 ships) has one planner-adjacent entry point, duckdb_add_replacement_scan, which swaps a table function in for a table name before binding; nothing in it sees or changes a logical plan.
• The C++ API has one. OptimizerExtension (duckdb/optimizer/optimizer_extension.hpp) takes an optimize_function that DuckDB calls with the whole logical plan after its own optimizers, registered with OptimizerExtension::Register(DBConfig &, ...). LogicalExtensionOperator and PhysicalOperator let the extension put its own operator into the plan.
• A C++ extension loads into the Python module. It leaves DuckDB's symbols unresolved and takes them from the process that loads it, and the duckdb 1.5.6 Python module exports DuckDB's C++ symbols, OptimizerExtension::Register among them. Compiled against the v1.5.6 headers (the module's source_id, 069cc9f9b5) with -undefined dynamic_lookup and stamped CPP / v1.5.6, the extension loads into that module with allow_unsigned_extensions. No DuckDB build is needed, only its headers. build_rewrite.sh checks the symbols on every build, because a missing one would abort the process at its first call instead of failing the LOAD.
• So the Python-level alternative is not needed. Intercepting queries in a wrapper and routing them by their EXPLAIN would work only from Python and only for queries sent through the wrapper; the optimizer route works for any client of the loaded connection. am.duckdb_connect only loads it.

The price of the C++ route is the pin: DuckDB loads a CPP extension only into the exact release in its footer, where the C-API extension of tier 2 loads into any DuckDB with C API 1.2 or later. A new DuckDB release means rebuilding with build_rewrite.sh, which fetches that release's headers, and possibly changing the source, since DuckDB's C++ API is not stable across releases: against the 1.4.5 headers it does not compile (OptimizerExtension::Register and PhysicalOperator::GetDataInternal are not there yet).

What is rewritten

What DuckDB has already done to the plan stays done: the filters it pushed into the scan, its compressed materialization of the group key, its rewrite of sum(x + 1) into sum(x) plus a count, and its switch from sum to sum_no_overflow where its statistics prove the total fits in 64 bits. test_unsupported_shapes_are_left_alone covers the shapes left to DuckDB and the reason the log gives for each; test_a_parquet_scan and test_a_registered_arrow_table_is_left_to_duckdb the two sources.

Exactly DuckDB's answers

python/tests/test_duckdb_rewrite.py runs every query twice on the same connection, with arrowmetal_rewrite off and forced, and requires the same column types and the same values bit for bit, avg included; only the order of an unordered GROUP BY may differ, since SQL does not define it, and with ORDER BY the order is compared too. Every test that takes a connection runs twice, once where its table fits one block and once with 2,048-row blocks, so the streamed path is covered by the same queries. The tables come from hash() of the row number and reach both ends of every integer type.

• Integer sums are exact to HUGEINT. DuckDB's sum over an integer column returns HUGEINT. A 64-bit column is summed as its high and low 32-bit halves, each of which fits in 64 bits over fewer than 2^31 rows, and the halves are recombined in 128 bits; where DuckDB's statistics already proved the total fits in 64 bits (sum_no_overflow), it is summed directly, and narrower columns cannot leave 64 bits. test_sums_past_int64_are_exact_hugeints sums values at both ends of BIGINT.
• avg uses DuckDB's own arithmetic: double(sum) / double(count) over SMALLINT, and Hugeint::Cast<long double>(sum) / count over INTEGER and BIGINT, the two finalizers in DuckDB's avg.cpp, so the doubles are the same doubles.
• NULLs: count(x) skips them; sum, avg, min and max of a group with no value are NULL; an empty input gives one row of zero counts and NULLs without GROUP BY and no rows with it; the NULL key is one group; a table whose keys are all NULL is one group. Each has a test.
• Floating-point sum and avg are not rewritten: DuckDB's own float sums depend on how its threads split the input, so there is no single answer to match. min and max over floats are not rewritten either: DuckDB orders NaN above every number, where ArrowMetal's min/max skip NaN.

The conformance grid. python/tests/engine_duckdb_grid.py generates the same comparison over a grid instead of chosen queries: sum, avg, min, max, count(v), count(*), all of them in one query, and a GROUP BY with no aggregate, over the ten value types the extension rewrites and six it leaves to DuckDB (DOUBLE, DECIMAL(18,3), HUGEINT, BOOLEAN, TIMESTAMP_NS, TIMESTAMPTZ), with no key or one key of each kind it takes (INTEGER with NULLs, a wide BIGINT, a negative SMALLINT, UTINYINT, DATE, TIMESTAMP, VARCHAR with NULLs), no filter or one of two, value columns with no, 5%, 70% or all NULLs, and tables of 0, 1, 7, 1,000 and 100,000 rows (the last also in 2,048-row blocks), plus integer extremes. Each query runs with the rewrite 'off' and 'force' on one connection. In the run recorded in Benchmarks/results/engine_conformance_2026-10-04.csv, on DuckDB 1.5.6, 33,376 queries: 21,844 rewritten and identical to DuckDB's answer, 11,532 left to DuckDB by the extension (and identical, as they must be), and none different. The per-shape counts are in Benchmarks/results/engine_conformance_2026-10-04_shapes.csv; Coverage summary has the summary.

How it runs

• The sink. ARROWMETAL_AGGREGATE is a parallel sink. Each DuckDB thread reserves row positions for its chunk with one atomic add and copies the aggregate's input columns into page-aligned buffers without a lock; a NULL clears its bit in an Arrow validity bitmap. A string key reserves its rows and bytes together under a lock, so the offsets stay in row order.
• Slabs. A fixed-width column's buffer comes from a pool kept across queries. Each slab is imported into ArrowMetal once, over its whole capacity; a query writes its rows into the same memory and hands ArrowMetal a zero-copy slice of the first rows, so the GPU wrapping is made once per slab rather than once per query. The pool keeps at most 4 GB of mappings, least recently used out first.
• Three GPU paths. Without GROUP BY, one fused query computes every aggregate in one pass (am_query, Fused expressions). With an integer key whose values span at most 2^20, the fused dense group-by keeps one slot per key value, split into several passes when the table would not fit in threadgroup memory. Anything else - a VARCHAR key, a wider key range, a 64-bit min/max under a GROUP BY - takes the hash group-by (am_group_by_keys and am_group_agg_ex). A short VARCHAR key that DuckDB's compressed materialization has already turned into an integer arrives as that integer and takes the integer paths.
• Streamed plans. An ungrouped aggregate, and a group-by whose key DuckDB's statistics put within 65,536 values (with min/max over at most 32-bit columns), is processed in blocks of arrowmetal_rewrite_block_rows rows (default 16,777,216). Each block goes to a GPU worker thread the moment its last row lands, while DuckDB is still scanning; Finalize runs the last, partly filled one and merges the per-block results on the host, exactly (sums in 128 bits). The other plans take up to 2^31 - 1 rows, the GPU kernels' 32-bit row index; a streamed plan is not held to that, since each block is smaller and the partial sums are added in 128 bits.
• Rows past the reservation. The buffers are sized from the source's row count at planning time. Rows beyond that - a prepared statement run after the table grew - are kept aside and appended in Finalize (test_a_prepared_statement_after_the_table_grew, test_a_streamed_plan_past_its_block_directory).
• One GPU. Rewritten queries on different connections take turns on the GPU, under one process-wide lock (test_concurrent_queries_on_two_connections).

When auto rewrites

Two conditions, both recorded for each decision in arrowmetal_rewrites():

1. The router's crossover. DuckDB's estimate of the rows reaching the aggregate is at or above the largest crossover among the query's aggregates in Benchmarks/results/router_2026-09-24.json: the reductions rows without GROUP BY, the group-by rows with it, in the 1,000-group or 100,000-group class by DuckDB's estimate of the group count (split at 10,000, the geometric middle), and the utf8 rows for a VARCHAR key. test_crossovers_match_the_router_sweep holds the constants to the JSON. The constants are read from Benchmarks/results/router_2026-09-24.json, which supersedes the 2026-09-17 sweep; of the rows used here only min(int64, 10% nulls) moved between the two, 50,000,000 then and 10,000,000 now. min without GROUP BY is gated by the ungrouped class's floor of 50,000,000 rows below (or not rewritten, with one or two aggregates), so no auto decision changes; at 10,000,000 rows the reason for such a query now reads "below the measured floor" where the results file, measured before the change, says "below the crossover".
2. A measured floor. The router's crossovers compare kernels on data already on the GPU. Here every row is first copied out of DuckDB's scan, and DuckDB's own aggregate runs while it scans, so the query's shape class must also have been measured faster than DuckDB's operators, from the size in this table, by Benchmarks/duckdb_rewrite_bench.py:

The benchmark's queries in each class are the rows of the results file whose shape_class column names it; a query the file does not list is in auto because of its class, not because it was timed. test_measured_floors_are_in_the_benchmark_results requires every row of a rewritten class to be ahead at the class's floor, and no row of any other class to be rewritten.

The measurements are in Benchmarks/results/duckdb_rewrite_2026-10-04-ac.csv, a quiet run on DuckDB 1.5.6 on AC power (its conditions in Benchmarks/results/duckdb_rewrite_2026-10-04-ac_conditions.txt): for each query and size (1M, 10M, 50M rows), DuckDB's time and the rewrite's in three alternating rounds, each mode's untimed warm-up of at least 100 ms followed by five timed runs: the best wall time with its CPU time, the median, the best first run after 500 ms of idle, the GPU path taken, and what auto decided. Its header records the load average at the start and the end of the run. test_measured_floors_are_in_the_benchmark_results requires every class in the table to be faster in that file at its floor. The floors were first fitted to a run taken while other work shared the machine, kept as history in Benchmarks/results/duckdb_rewrite_2026-09-23_provisional.csv, and checked by a second run on 2026-09-24 (Benchmarks/results/duckdb_rewrite_2026-09-24.csv, one warm-up and the best of five, its conditions in Benchmarks/results/bench_conditions_2026-09-24.txt); the run of 2026-10-02 on DuckDB 1.5.5 (Benchmarks/results/duckdb_rewrite_2026-10-02.csv) and the two runs of 2026-10-04 on DuckDB 1.5.6, on battery power (Benchmarks/results/duckdb_rewrite_2026-10-04.csv) and on AC power, give the same floors; DuckDB's own best times in the AC run are 0.96 to 1.12 times those of the 1.5.5 run, query for query. Reproduce with:

./duckdb-extension/build_rewrite.sh
PYTHONPATH=python python Benchmarks/duckdb_rewrite_bench.py        # 1M, 10M and 50M rows

Each floor is the smallest of the three sizes at which every query of the class is ahead of DuckDB, at that size and above, on the best and on the median run. Every query auto rewrote is ahead of DuckDB, by 1.08x (sum, max, avg (BIGINT) at 50M rows; 1.04x on the median) to 5.32x (100k INTEGER keys alone (no aggregates) at 50M rows; 3.76x on the median). One query of each rewritten class keeps the class out of auto at the size below its floor: sum, max, avg (BIGINT) at 10M rows (0.87x) for the ungrouped class, 100k INTEGER keys: sum, min, max, avg at 1M (0.71x) for the fused group-by with many groups, 1k INTEGER keys: sum, count, min, max, avg at 10M (0.55x) for the fused group-by with fewer groups, and ~1M wide BIGINT keys: sum, count at 10M (0.94x) for the hash group-by with many groups.

The classes left to DuckDB are the ones where its own operators came out ahead in that file, or not ahead consistently: a single sum (DuckDB aggregates while it scans, and the rewrite has to copy the column out first), few groups with at most two aggregates, and VARCHAR keys that DuckDB keeps as strings. In each of them at least one query is behind DuckDB at every size, 50M rows included: sum(INTEGER) at 0.32x, 1k INTEGER keys: sum at 0.59x, ~3k wide BIGINT keys: sum at 0.92x and 1k long VARCHAR keys: sum, count at 0.22x. Some of their queries were ahead: sum and sum, avg over full-range BIGINT values (1.92x and 3.76x at 50M rows), 10k INTEGER keys: sum, count (1.48x at 50M, 0.85x at 10M) and 1k short VARCHAR keys: sum, count (1.12x at 50M). In the ungrouped case DuckDB's time depends on the values, which the plan does not show: avg and sum, avg over non-negative BIGINT values are to improve (0.64x and 0.83x at 50M rows), while the same aggregates over full-range BIGINT values are ahead. Below a rewritten class's floor, or below the router's crossover, some queries were ahead as well, for example sum, count, min, max (BIGINT, 10% NULL) at 10M rows (1.44x) and 100k INTEGER keys alone (no aggregates) at 1M (1.52x); the floor holds each class to its slowest query.

To improve: on the first run after 500 ms of idle, four of the thirteen rewritten query-size pairs are behind DuckDB's first run after the same idle, all at 50M rows: sum, max, avg (BIGINT) 22.38 ms against 8.55 ms (0.38x), sum, count, min, max (BIGINT, 10% NULL) 29.39 against 23.72 ms (0.81x), sum, min, max, avg (INTEGER) 12.54 against 10.34 ms (0.82x) and 1k INTEGER keys: sum, count, min, max, avg 36.68 against 32.61 ms (0.89x). The other nine are 1.09x to 2.26x ahead on that run.

Limits

• Built for one DuckDB release and one platform string at a time; unsigned.
• The decision is made when the plan is: a prepared statement keeps the decision made at PREPARE, and EXPLAIN records a decision of its own.
• A plan keeps the key statistics it was made with. Where DuckDB would have run the group-by as its PERFECT_HASH_GROUP_BY (an integer key whose statistics span at most 2^perfect_ht_threshold - 2 values, 12 bits by default), a key that lands past the table those statistics sized raises DuckDB's own error, Perfect hash aggregate: aggregate group N exceeded total groups M. This likely means that the statistics in your data source are corrupt., as DuckDB does with the rewrite off. That happens when a prepared statement runs after keys outside the planned range were inserted: the plan's compressed materialization narrowed the key to key - min in a smaller unsigned type, in which such keys wrap. Where DuckDB would have used its hash group-by, which does not check, the rewrite does not either, and both answer over the same narrowed keys (test_a_prepared_statement_whose_key_outgrew_its_statistics, test_where_duckdb_does_not_check_the_statistics_neither_does_the_rewrite).
• One known difference in that error: for an 8-bit key (TINYINT, UTINYINT, which the plan does not narrow) below the planned minimum, the group number. Such a key's slot is negative, key - min + 1; DuckDB prints it as 2^128 plus the slot and the rewrite as 2^64 plus the slot, so statistics of [10, 14] and a key of 3 read group 340282366920938463463374607431768211450 from DuckDB and group 18446744073709551610 from the rewrite. The difference is deterministic; the exception class (InvalidInputException) and the rest of the text are the same, and keys of 16 bits or more, or above the maximum, give the same number. A key exactly one below the minimum has slot 0, which is the slot DuckDB keeps for the NULL key: DuckDB returns that key's rows as the NULL group, with no error, and the rewrite returns them under the key itself (test_an_8_bit_key_below_the_planned_minimum).
• The log keeps the last 1,024 decisions of the process and is shared by every connection; a rewritten plan's path, rows_seen, groups and gpu_ms describe its latest run.
• The aggregate's input is held in memory: the buffers are reserved for the source's row count and filled as rows arrive (a streamed plan holds only its blocks in flight). Freed slabs stay mapped in the pool, up to 4 GB.
• The shapes in "What is rewritten" only: one group key, no floating-point sum/avg, no DECIMAL, HUGEINT or BOOLEAN values, no min/max over strings.

---

5. Numbers

Apple M4 Max, DuckDB 1.5.5, pyarrow 25.0.1, ArrowMetal 0.1.0. Best of 5, wall ms, with the process's CPU time beside it. Every figure in this section is from Benchmarks/results/duckdb_bench_50000000_2026-09-07.txt. Reproduce with:

PYTHONPATH=python python Benchmarks/duckdb_bench.py 50000000 5

Both sides compute the same answer from the same in-memory DuckDB table, and the benchmark checks that they agree before it reports a time.

50M rows

Look at the CPU-ms columns as well as the wall-ms. DuckDB's 100k-key group-by at 50M rows costs 1,056 CPU-ms to produce 71.4 ms of wall time - it is using about fifteen cores. The GPU's costs 0.6 CPU-ms. If anything else on the machine wants those cores, that difference is the real one.

What the crossing costs

At 50M rows, one BIGINT column (406 MB):

to_arrow_table       19.7 ms   (407 chunks - DuckDB handing over pointers, not bytes)
combine_chunks       26.6 ms   (a real copy: 407 chunks into one buffer, 15.3 GB/s)
am_import             5.7 ms   (buffer shared, not copied - the pointer is unchanged)
                    -------
total                52.1 ms

The Metal import copies nothing; combine_chunks is the largest single item. It is the price of ArrowMetal wanting one contiguous array per column and DuckDB producing many. am.duckdb_batches avoids it entirely by keeping the chunks separate. Streaming a sum through the GPU one record batch at a time, never materialising the column:

All three are exact. On this run the whole-table crossing (52.1 ms) is ahead of the streaming path (67.5 ms at its best batch size), so streaming is a memory-footprint argument rather than a speed one.

This is also why the "one-shot" column above is mostly below 1.0x: for a single operation the crossing dominates, and DuckDB is ahead. The GPU pays off when you do several things to one dataset, or one expensive thing.

50M rows out of Parquet

con.execute("copy (select i::BIGINT amount, (hash(i) % 1000)::INTEGER region "
            "from range(50000000) r(i)) to 'big.parquet' (format parquet)")

con.sql("select sum(amount) from read_parquet('big.parquet')")
am.from_duckdb(con.sql("select amount from read_parquet('big.parquet')"))["amount"].sum()
am.duckdb_aggregate(con.sql("select amount from read_parquet('big.parquet')"),
                    {"total": ("sum", "amount")}, rows_per_batch=1 << 22)

All three give 1249999975000000. DuckDB is ahead here: a single sum over a Parquet file is exactly the case where DuckDB aggregates during the scan and never materialises anything. Reach for the GPU when the aggregate is the expensive part, not the scan.

The matrix rows, DuckDB alongside

The full matrix has no DuckDB column, because DuckDB is a query engine rather than an array library: the comparison needs a table, a query, and a decision about where the result lands. Benchmarks/duckdb_matrix.py makes those decisions explicit and measures DuckDB on the matrix's sort, group-by, filter and sum rows, with the matrix's own generators, sizes and best-of-five protocol, in four idioms: the same SQL over a registered Arrow table in one chunk and in sixteen batches, and over a native DuckDB table with the result exported to Arrow or kept in a DuckDB temp table. Every result asserts its row count. Reproduce with:

PYTHONPATH=python python Benchmarks/duckdb_matrix.py 10000000 50000000
python Benchmarks/duckdb_matrix.py --report Benchmarks/results/duckdb_matrix_2026-09-12.csv

DuckDB rows measured 2026-09-12: duckdb 1.5.5, threads=16, preserve_insertion_order=True, pyarrow 25.0.1, python 3.13.9, Apple M4 Max, macOS 26.6.2, loadavg at start 2.5, 2026-09-12T23:34:30. ArrowMetal, Polars and pyarrow are the published 2026-09-07 matrix rows (Benchmarks/results/full_matrix_2026-09-07-parallel.csv), not re-measured. Wall ms, best of up to 5 after one warm-up, process CPU ms in brackets. "Best" is the faster of a library's eager and parallel idioms. The last column is the fastest CPU number on the row, any library or idiom, over ArrowMetal.

SQL per operation, over the native table; the Arrow-scan idioms run the same text over the registered Arrow table, and the stays-in-DuckDB idiom wraps it in CREATE OR REPLACE TEMP TABLE r AS:

• sum(int64, 10% nulls): SELECT sum(x) FROM tn (DuckDB sum(BIGINT) returns HUGEINT)
• filter int64 (30% kept): SELECT x FROM tn WHERE m
• argsort int64: SELECT i FROM tn ORDER BY x (DuckDB has no argsort)
• argsort float64: SELECT i FROM tn ORDER BY x (DuckDB has no argsort)
• sort float64: SELECT x FROM tn ORDER BY x
• lexsort (2 int32 keys): SELECT i FROM tn ORDER BY a, b (DuckDB has no argsort)
• sum by int32 key (1000 groups): SELECT k, sum(x) FROM tn GROUP BY k
• mean by int32 key (1000 groups): SELECT k, avg(x) FROM tn GROUP BY k
• count by int32 key (1000 groups): SELECT k, count(x) FROM tn GROUP BY k
• sum by two int32 keys (~1024 groups): SELECT a, b, sum(x) FROM tn GROUP BY a, b
• sum by int32 key (100000 groups): SELECT k, sum(x) FROM tn GROUP BY k
• mean by int32 key (100000 groups): SELECT k, avg(x) FROM tn GROUP BY k
• count by int32 key (100000 groups): SELECT k, count(x) FROM tn GROUP BY k
• sum by int32 key (10000000 groups): SELECT k, sum(x) FROM tn GROUP BY k
• sort utf8: SELECT s FROM tn ORDER BY s

What the table says:

• DuckDB is the fastest CPU engine on low-cardinality group-by and on multi-key sort, and it is a long way ahead of Polars on both: 9.0 ms against Polars' 81.9 for sum by int32 key at 1,000 groups and 50M rows, 265 ms against 903 for lexsort. Against DuckDB those two ArrowMetal rows are 1.8x and 7.0x, where the matrix shows 3.8x (against Acero; 16.8x against Polars) and 24.0x. At 10M rows and 1,000 groups the margin is 1.2x. This is the honest group-by story: the GPU is ahead by 3x or more from 100,000 groups upward and roughly even with DuckDB at a thousand groups.
• DuckDB does not win every CPU row. At 100,000 groups Acero is faster than DuckDB (46.4 ms against 75.9 at 50M rows), on sort float64 Polars is (133 against 294), and on the filter and the plain sum Polars lazy is the fastest CPU idiom by a wide margin. The fastest-CPU column picks the winner per row.
• Exporting the result to Arrow is a large part of DuckDB's wall time on wide results. Sorting 50M float64 takes 294 ms inside DuckDB and 1,308 ms delivered as an Arrow table; the argsorts add about 1.1 to 1.2 s the same way. A Python caller pays that; ArrowMetal's result is already an Arrow buffer. For a group-by result of a thousand rows the two idioms are the same.
• The CPU-time gap does not narrow. DuckDB's 1,000-group sum at 50M rows costs 135 CPU-ms across about fifteen cores, its sorts 3,100 to 4,350; the ArrowMetal rows cost 1.2 to 2.4 CPU-ms.
• A registered Arrow table in one chunk scans on two cores. That idiom is 1.2x slower than the native table on the filter and 29x slower on the 1,000-group sum. Register a chunked table, or create a DuckDB table, before benchmarking DuckDB over Arrow data.

Two things to carry with the numbers. The ArrowMetal, Polars and pyarrow rows were measured on 2026-09-07 and the DuckDB rows on 2026-09-12, on the same machine with the same protocol, so the day differs. And the sort row in the 50M table above (39.2 ms) sorts v = i, a column that is already in order; on uniform random float64 of the same size DuckDB's sort takes 294 ms.

---

6. When this is worth it, and when it is not

Worth it

• High-cardinality group-by over a resident column. 100,000 keys at 50M rows: 16x, at 0.6 CPU-ms against DuckDB's 1,056; a single cold query, crossing included, is 1.1x.
• String matching over a column you are keeping resident. LIKE '%...%' at 50M rows: 9.3x resident; a single cold query is 0.6x.
• Sorting a resident column. 2.2x at 50M rows (17.6 ms against DuckDB's 39.2), and 1.4 CPU-ms against 570.
• Several operations over one dataset. The crossing is paid once; every kernel after that is free of it. This is the single biggest lever - from_duckdb once, then loop.
• Anything where you want the cores back. The GPU group-by uses 0.6 CPU-ms where DuckDB uses 1,056.
• Fused filter-and-aggregate expressions, which read the column once no matter how long they are.

Not worth it

• A single sum/avg over one column, whether from memory or Parquet. Memory-bound, and DuckDB does it while scanning.
• Sorting from cold. The GPU radix sort is ahead of DuckDB's parallel sort on a resident column (2.2x) but behind once the crossing is included (0.7x one-shot).
• Small data. Below a few million rows the dispatch latency is most of the time — about 60-70 µs measured (The persistent-kernel result), 110-160 µs per call for sum and filter at 1,000 rows in the matrix's latency family; use am.batch() to amortise it, or do not bother.
• Anything the extension can do, if speed is the reason. See §4.

The rule of thumb: the GPU is ahead when the compute is large relative to the bytes. Group-by with many keys and string matching are compute-heavy per byte; sum is not.

---

7. Limits

Tier 1

• One MetalArray is one contiguous array, so a multi-chunk DuckDB result is concatenated on the way in. At 50M rows that is 26.6 ms (§5). am.duckdb_batches skips it.
• to_duckdb registers a view on one connection; other connections to the same database will not see it. Use CREATE TABLE ... AS SELECT * FROM the_view to make it durable.
• Nested columns (list, struct, map) import and round-trip, but only the structural kernels operate on them - you cannot group by a struct.
• Streaming refuses median, quantile, stddev, mode and tdigest by design (§3).

Tier 2

• Behind DuckDB's own SQL as measured (§4).
• The eight integer widths, FLOAT, DOUBLE, DATE and TIMESTAMP only -- exactly what arrow_format_for in duckdb-extension/src/arrowmetal_extension.cpp lists. BOOLEAN is not among them; nor are VARCHAR, DECIMAL, TIME, TIMESTAMPTZ, HUGEINT or the nested types. All of them are refused by name at bind time with a message pointing here, and the bridge handles every one.
• The table functions take a table or view name, not a subquery - DuckDB's C table-function API has no way to accept a relation. A Python-registered relation (con.register(...)) is connection-local and the extension's own connection cannot see it; CREATE VIEW first.
• arrowmetal_group_by returns key as BIGINT or DOUBLE, so a key column wider than that (there is none among the supported types) would not fit.
• arrowmetal_query accepts only an (aggregate ...) terminal. (project ...) and (group_by ...) produce columns, which this function's fixed output schema cannot carry; use arrowmetal_group_by or the bridge.
• The extension opens one DuckDB connection when it loads and keeps it for the life of the process, serialised by a mutex. It cannot connect lazily instead: the duckdb_database handle the loader passes does not outlive the load call. That connection is never closed, so the database it belongs to is not freed until the process exits.
• Unsigned. allow_unsigned_extensions has to be set when the connection is created.
• Built for one platform string at a time; osx_arm64 unless you set DUCKDB_PLATFORM.

Tier 3

• §4b, "Limits".

Files

Apache DataFusion is a query engine in Rust: SQL and DataFrames over Arrow, with its own planner, optimizer and multi-threaded operators. datafusion-arrowmetal (in datafusion/) is a physical optimizer rule for DataFusion 55.1. Registered on a SessionContext, it replaces DataFusion's full sort (an ORDER BY without LIMIT) with MetalExec, which runs the sort on the Apple GPU through ArrowMetal's plan runner and hands DataFusion the RecordBatches it expects. It also replaces the hash aggregates (GROUP BY, DISTINCT) of the shapes a measured table takes: count(*) or DISTINCT over two int32 keys, or MIN/MAX of an integer column over one int64 key or two integer keys, of a MemTable of at least 50,000,000 rows. Such an aggregate estimates its number of groups from a sample of its keys when it runs, runs on the GPU at the numbers of groups the table takes at that row count, and otherwise hands the node back to DataFusion's own operators. Hash joins (inner, left, right) are translated too, and a measured join table decides them; it takes no measured join. The SQL does not change, and every other node of the plan stays DataFusion's.

The summary. On an Apple M4 Max with DataFusion's default of one partition per core, a full sort of 250,000 to 50,000,000 rows is 6.9x to 28.8x faster with the rule than DataFusion alone, and at 50M rows it uses 96 to 126 CPU-ms where DataFusion uses 6,104 to 8,311. Over DataFusion's own Parquet reader, a Float64 sort is 10.8x to 16.5x faster at 10M and 50M rows, and a sort over a Parquet file with a string column 3.6x to 5.5x at 1M to 50M rows. The aggregates the default runs on the GPU, count(*) and DISTINCT over two int32 keys and MIN/MAX of an integer column, all at 50M rows, were 2.31x to 4.27x faster than DataFusion alone warm; on the first run after 500 ms of idle they were 1.24x to 2.17x faster than DataFusion alone's first run after the same idle, and after 5 s of idle 1.11x to 1.68x; against DataFusion alone's warm time, that first run was 0.89x to 1.78x after 500 ms and 0.69x to 1.35x after 5 s (Aggregates). The answers are DataFusion's: a differential grid of 13,632 query pairs runs every query with and without the rule and finds 0 mismatches. Top-k (ORDER BY … LIMIT), filters, joins and every other aggregate shape are left to DataFusion by default; the measured numbers for them are under To improve and Joins, rule on and off.

• Install
• Use
• What the default takes
• What it leaves, and why
• DataFusion's semantics
• The differential grid
• Numbers
• To improve
• Limits
• Tests and files

---

Install

The crate needs an Apple silicon Mac and the ArrowMetal dylib:

# 1. The GPU library, from the repository root.
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
    swift build -c release --product ArrowMetalC

# 2. The example, against that build.
cd datafusion
ARROWMETAL_LIB=$PWD/../.build/release/libArrowMetalC.dylib cargo run --example quickstart

The crate is on crates.io:

# Cargo.toml
[dependencies]
datafusion = { version = "=55.1.0", default-features = false, features = ["sql"] }
datafusion-arrowmetal = "0.4.1"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

From a checkout, depend on it by path instead: datafusion-arrowmetal = { path = "../ArrowMetal/datafusion" }.

It depends on the Rust crate, which finds libArrowMetalC.dylib through ARROWMETAL_LIB and the other locations listed in RUST.md, "Finding the dylib". A binary that uses the rule needs the dylib's directory as an rpath entry, set the way that section shows (a build.rs or RUSTFLAGS); the crate's own build.rs does it for its tests and examples.

Versions. DataFusion is pinned to =55.1.0, the release the grid and the benchmark ran on (arrow-rs 59.3.0, the same arrow-rs major as the arrowmetal crate, so batches pass between the two with no conversion). DataFusion 55.1.0 declares rust-version 1.94; the crate was built with rustc 1.95.0.

---

Use

use std::sync::Arc;

use datafusion::arrow::array::{ArrayRef, Int64Array, RecordBatch};
use datafusion::datasource::MemTable;
use datafusion::error::Result;
use datafusion::physical_plan::{collect, displayable};
use datafusion::prelude::SessionConfig;
use datafusion_arrowmetal::{session_context, ArrowMetalConfig, ArrowMetalRule};

#[tokio::main]
async fn main() -> Result<()> {
    let rule = ArrowMetalRule::new(ArrowMetalConfig::default()); // full sorts from 250,000 rows
    let ctx = session_context(SessionConfig::new(), rule.clone()); // DataFusion's defaults + the rule

    let v: Int64Array = (0..1_000_000i64).map(|i| (i * 7919) % 1_000_003).collect();
    let batch = RecordBatch::try_from_iter([("v", Arc::new(v) as ArrayRef)])?;
    ctx.register_table("t", Arc::new(MemTable::try_new(batch.schema(), vec![vec![batch]])?))?;

    // EXPLAIN shows MetalExec in the physical plan.
    ctx.sql("EXPLAIN SELECT v FROM t ORDER BY v DESC").await?.show().await?;

    // Or plan once, print the plan and the rule's report, and run that plan.
    rule.clear_report();
    let plan = ctx.sql("SELECT v FROM t ORDER BY v DESC").await?.create_physical_plan().await?;
    println!("{}", displayable(plan.as_ref()).indent(false));
    println!("{}", rule.report()); // TAKEN / LEFT, with the reason
    let batches = collect(plan, ctx.task_ctx()).await?;
    println!("{} rows", batches.iter().map(|b| b.num_rows()).sum::<usize>());
    Ok(())
}

The EXPLAIN it prints:

+---------------+-----------------------------------------------------+
| plan_type     | plan                                                |
+---------------+-----------------------------------------------------+
| logical_plan  | Sort: t.v DESC NULLS FIRST                          |
|               |   TableScan: t projection=[v]                       |
| physical_plan | MetalExec: sort=[v DESC NULLS FIRST]                |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |                                                     |
+---------------+-----------------------------------------------------+

Three ways to register it, all in src/lib.rs:

The rule goes in before DataFusion's last two physical rules, the post-optimization FilterPushdown and SanityCheckPlan, so the rewritten plan is still checked for distribution and ordering. After a run, MetalExec's metrics (ExecutionPlan::metrics) hold input_time, import_time, kernel_time, export_time and input_batches, and for an aggregate probe_time (the group-count estimate), groups_estimate and handed_back; the benchmark reads its time split from them.

examples/quickstart.rs registers a 1,000,000-row MemTable (an int64, a nullable Float64 and a string column) with the default configuration and 4 partitions, and runs three queries. Its output, from cargo run --example quickstart on an M4 Max:

== SELECT name, region, amount FROM sales ORDER BY amount DESC NULLS LAST, name

ProjectionExec: expr=[name@2 as name, region@0 as region, amount@1 as amount]
  MetalExec: sort=[amount DESC NULLS LAST, name ASC NULLS LAST]
    DataSourceExec: partitions=1, partition_sizes=[1]

TAKEN    SortExec: expr=[amount@1 DESC NULLS LAST, name@2 ASC NULLS LAST], preserve_partitioning=[false] -- input rows 1000000 (exact) vs min_rows 250000

1000000 rows; the first 3:
+---------+--------+--------------------+
| name    | region | amount             |
+---------+--------+--------------------+
| n001395 | 0      | 1429.4285714285713 |
| n005102 | 5      | 1429.4285714285713 |
| n015952 | 4      | 1429.4285714285713 |
+---------+--------+--------------------+

== SELECT name, amount FROM sales ORDER BY amount DESC NULLS LAST LIMIT 3

ProjectionExec: expr=[name@1 as name, amount@0 as amount]
  SortExec: TopK(fetch=3), expr=[amount@0 DESC NULLS LAST], preserve_partitioning=[false]
    DataSourceExec: partitions=1, partition_sizes=[1]

LEFT     SortExec: TopK(fetch=3), expr=[amount@0 DESC NULLS LAST], preserve_partitioning=[false] -- top-k (sort with fetch 3) disabled in config

3 rows; the first 3:
+---------+--------------------+
| name    | amount             |
+---------+--------------------+
| n300762 | 1429.4285714285713 |
| n421196 | 1429.4285714285713 |
| n110979 | 1429.4285714285713 |
+---------+--------------------+

== SELECT region, count(*) AS n, avg(amount) AS mean FROM sales GROUP BY region ORDER BY region

SortPreservingMergeExec: [region@0 ASC NULLS LAST]
  ProjectionExec: expr=[region@0 as region, count(Int64(1))@1 as n, avg(sales.amount)@2 as mean]
    SortExec: expr=[region@0 ASC NULLS LAST], preserve_partitioning=[true]
      AggregateExec: mode=FinalPartitioned, gby=[region@0 as region], aggr=[count(Int64(1)), avg(sales.amount)]
        RepartitionExec: partitioning=Hash([region@0], 4), input_partitions=4
          AggregateExec: mode=Partial, gby=[region@0 as region], aggr=[count(Int64(1)), avg(sales.amount)]
            RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
              DataSourceExec: partitions=1, partition_sizes=[1]

LEFT     SortPreservingMergeExec: [region@0 ASC NULLS LAST] -- input rows ~1000000 are an estimate (accept_inexact is off)
LEFT     SortExec: expr=[region@0 ASC NULLS LAST], preserve_partitioning=[true] -- per-partition sort (preserve_partitioning) with no replaced merge above it
LEFT     AggregateExec: mode=FinalPartitioned, gby=[region@0 as region], aggr=[count(Int64(1)), avg(sales.amount)] -- input rows 1000000 (exact) vs min_rows 250000; count + sum_avg_f64 over 1 i64 key (memory input): the measured table takes it at no group count and size (results/datafusion_groupby_sweep_2026-10-01.csv)
LEFT     AggregateExec: mode=Partial, gby=[region@0 as region], aggr=[count(Int64(1)), avg(sales.amount)] -- Partial aggregate whose Final was not replaced

13 rows; the first 3:
+--------+-------+-------------------+
| region | n     | mean              |
+--------+-------+-------------------+
| 0      | 76924 | 714.7095888611585 |
| 1      | 76923 | 714.7337580454488 |
| 2      | 76923 | 714.739149199677  |
+--------+-------+-------------------+

The report (rule.report(), a Report of Decisions) has one line per node the rule looked at: TAKEN or LEFT with the reason; for a replaced aggregate, one line per run, GPU or HANDBACK, with the estimated number of groups, its range, the sample it came from and the table row that decided (Decision::groups holds the estimate); and FALLBACK for a MetalExec that handed its node back to DataFusion on an ArrowMetal error, on data only DataFusion answers exactly, or when the memory pool refused its reservation. Clones of a rule share one report. It keeps the decisions of the last report_plans plans the rule optimized (default 64; an EXPLAIN and each executed plan count one each), until clear_report() empties it.

---

What the default takes

ArrowMetalConfig::default():

ArrowMetalConfig::all() switches topk and filter on as well. The fields are public to read; a config is built from default() or all() with the with_* setters, e.g. ArrowMetalConfig::all().with_min_rows(0), which the grid and the benchmark use.

A full sort is taken when:

• every sort key is a column (not an expression) of type int8 to int64, uint8 to uint64, Float32, Float64, Utf8, LargeUtf8 or Utf8View;
• every column the sort carries is one of those types or Boolean;
• DataFusion's statistics give the sort's input an exact row count of at least min_rows (MemTables and DataFusion's Parquet scan gave exact counts in the tests and the benchmark);
• the plan shape is SortExec (one partition, or preserve_partitioning = false), SortPreservingMergeExec over SortExec(preserve_partitioning) with the same ordering, or SortPreservingMergeExec → ProjectionExec → SortExec, which DataFusion plans when the SELECT list reorders or computes columns; the projection stays above MetalExec.

MetalExec emits one partition. When the replaced node had more and something above it requires a distribution, the rule adds a RepartitionExec with the replaced node's partitioning; with only projections above it, the output stays at one partition.

Why 250,000 rows. In datafusion_rule_2026-09-29.csv every key type is ahead of DataFusion at 100,000 rows (1.35x to 2.63x, both layouts), and 250,000 rows is the first measured size at which every key type is at 2.64x or more. The file measured below with a warm-up (datafusion_sort_warm_2026-09-29.csv) puts every key type at 6.9x or more at 250,000 rows.

Aggregates. The rule sees an aggregate's row count but not its number of groups, and the same aggregate is ahead of DataFusion at some group counts and behind at others. So an aggregate is taken in two steps:

• At plan time the rule describes it by its shape: the aggregate families (count, sum/avg over an integer column, sum/avg over Float64, min/max over an integer column, min/max over a float column (Float64 or Float32), and DISTINCT for a GROUP BY with no aggregate function), one key or several, the key type class (up to 32-bit integers, 64-bit integers, floats, strings) and whether it reads a MemTable scan directly. It replaces the node with a MetalExec only when the measured table (src/agg_table.rs) takes that shape at some number of groups at the input's exact row count.
• When it runs, MetalExec reads the first batches of each input partition (262,144 rows in all), estimates the number of groups from a stratified sample of their keys (the bias-corrected Chao1 estimator over 512 to 8,192 sampled rows), and looks the shape up at that number of groups. If the table takes it, the rest of the input is collected, a 2,048-row sample of the whole input must agree with the estimate, and the aggregate runs on the GPU. Otherwise the batches read so far and the rest of each partition stream into DataFusion's own partial and final aggregates, over the same partitions as without the rule.

The table is generated by scripts/groupby_table.py from the measurements it names:

1. The sweep (datafusion_groupby_sweep_2026-10-01.csv: every family over one and two int32 and int64 keys, at 200 to about rows / 2 groups, 1M to 50M rows, both table layouts, the rule forced onto the GPU against DataFusion alone; min/max over Float64 and over Float32, one family as the rule describes them, measured on 2026-10-01 after float MIN/MAX moved to the plan runner's own grouped min/max; sum/avg over Float64 and min/max over int64 on 2026-09-30, the other families on 2026-09-29). A series (shape and number of groups) qualifies from the smallest measured row count at which its worst case (over the layouts and the family's queries) was at least 1.65x faster than DataFusion alone, at that size and every larger one, at two sizes or more. 38 of the 120 measured series qualify, from 2M to 10M rows.
2. Idle against idle, after 500 ms and after 5 s: a series is taken at a size only if, at that size and every larger one, the default's first run after 500 ms of idle and its first run after 5 s of idle (the Metal pipelines already compiled) were each at least as fast as DataFusion alone's first run after the same idle, for the same query, size and layout, in every case (the sweep's queries and SELECT DISTINCT region, sub) and layout (the median of three runs of each, contexts and gaps alternating; datafusion_groupby_idle_gaps_2026-09-29.csv, datafusion_groupby_resweep_check_2026-10-01.csv, datafusion_groupby_default_import_2026-10-01.csv (its rows of the current crate), datafusion_groupby_float_minmax_check_2026-10-01.csv and datafusion_groupby_refit_check_2026-10-02.csv, which measure both gaps in one session, and the earlier files). A case with no measurement at one of the gaps counts as below.
3. The hand-back: a shape is replaced at a size only if, at that size and every larger one, every case the default handed back at run time was at 0.97x of DataFusion alone or better on the best run and on the median run (datafusion_groupby_float_minmax_check_2026-10-01.csv, datafusion_groupby_refit_check_2026-10-02.csv, datafusion_groupby_resweep_check_2026-10-01.csv, datafusion_groupby_idle_gaps_recheck_2026-09-29.csv, datafusion_groupby_idle_vs_idle_2026-09-29.csv and the files before them).

Ten series are taken, all over integer keys of a MemTable scan of at least 50,000,000 rows (an int32 class includes narrower keys):

The last column is the series' worst ratio at 10M and at 50M rows (DataFusion alone ÷ ArrowMetal, best run, every case of the series in both layouts, two significant digits) in datafusion_groupby_sweep_2026-10-01.csv, as src/agg_table.rs records it.

Each series' row in src/agg_table.rs ends with the reason for its threshold. count(*) over two int32 keys is taken from 50M rows: at 10M its first run after 5 s of idle was below DataFusion alone's first run after the same idle (0.81x at 100,000 groups with 8,192-row batches, 0.88x at 1,000,000 groups with one batch per partition; datafusion_groupby_default_import_2026-10-01.csv). Of the other 28 series that qualify in the sweep, 23 are left by their first run after idle at 50M rows (7 slower than DataFusion alone's first run after the same idle, 16 with no measurement after 5 s of idle there), and 5 because a hand-back of the same shape at 50M rows cost more than 3% (To improve). The float MIN/MAX series are among them: four qualify warm (one and two keys, 100,000 groups, from 5M or 10M rows), and none holds at 50M rows on both other rules.

Joins

The rule translates a HashJoinExec when:

• its join type is inner, left or right (DataFusion plans the SQL probe LEFT JOIN build of a larger table as a right join with the smaller table on the build side), with equal keys only: no join filter, SQL's = (NullEqualsNothing), no fetch;
• every key is a column, of the same type on both sides, int32, int64 or Utf8 (one key or several);
• every column it outputs is one of the types a sort carries;
• both inputs have an exact row count, and the larger is at least min_rows;
• an output ordering, if the join has one, is the probe side's in one partition.

MetalExec reads below the exchanges under the join (its RepartitionExecs and CoalescePartitionsExecs), collects both inputs, runs the join on the GPU, and deals the result out to as many output partitions as the join had; a hash-partitioned join keeps a RepartitionExec by the same hash above it where the parent requires that distribution.

Which joins the default replaces is a measured table, src/join_table.rs, generated by scripts/join_table.py from datafusion_join_2026-10-01.csv: per join type, key type and build-side size (10,000, 1,000,000 and 10,000,000 rows, each standing for half a decade either side), a cell is taken from the smallest measured probe-side size (10M, 50M rows) from which every case, in both layouts, is at least 1.5x faster than DataFusion alone warm and at least 1.0x on the first run after 500 ms and after 5 s of idle against DataFusion alone's first run after the same idle, at that size and every larger one. The cases: SELECT count(*), sum(b.w) FROM probe p [LEFT] JOIN build b ON p.k = b.k and SELECT p.k, count(*), sum(p.v) … GROUP BY p.k over the join, half the probe rows matching. The table takes none of its 18 cells, so the default replaces no join; every row of the table ends with the reason (Joins, rule on and off).

---

What it leaves, and why

Each of these is a LEFT line in the report with its reason; the rule does not report the nodes it has no operator for (a nested-loop or sort-merge join, a window, a union).

---

DataFusion's semantics

Each ORDER BY key becomes one key of ArrowMetal's sort, with two options that give DataFusion's order in the sort itself, with no extra key or pass:

• Nulls first or last, as the query says, in either direction (NULLS FIRST, NULLS LAST, or DataFusion's default for the direction).
• Floats in IEEE 754 totalOrder, the order arrow-rs and DataFusion use: -NaN < -inf < … < -0.0 < +0.0 < … < +inf < +NaN, NaNs ordered by payload, and descending the exact mirror. So a NaN comes first in a descending sort and -0.0 sorts before +0.0. tests/arrowmetal_repros.rs (sort_float_order_total_is_arrow_rs_order) checks this bit for bit against arrow::compute::sort_to_indices, for Float64 and Float32, with -NaN, NaN payloads, ±0.0, ±inf, subnormals and nulls, in both directions with both null placements.
• Strings (Utf8) in byte order, as arrow-rs compares them; the grid's string keys include "", "B" against "b", and "ü".
• Ties: SQL leaves the order of rows with equal keys open. The grid checks the key sequence exactly and the rows within each run of equal keys as a multiset.

Handed back to DataFusion at run time. MetalExec keeps the subtree it replaced. If ArrowMetal returns an error or the GPU path panics, or the session's memory pool refuses the reservation for the collected input or the result, MetalExec runs that subtree over the batches it collected and the rest of its input, returns DataFusion's answer, and adds a FALLBACK line to the report. No query in the grid or in the benchmarks took that path; tests/rule.rs runs a sort and a group-by in a 3 MiB memory pool, which hand back, and a join in a 6 MiB pool. With aggregates taken, two checks on the data also hand the node back, because on such data only DataFusion can give DataFusion's answer:

• a GROUP BY over a float key that holds NaNs of more than one bit pattern (DataFusion keeps each pattern as its own group; ArrowMetal puts every NaN in one group);
• a grouped float MIN or MAX over a group that holds a NaN, or both -0.0 and +0.0 with a zero result (DataFusion's answer there depends on the order the rows arrive in).

Float comparisons in a WHERE. DataFusion 55.1 compares floats with arrow-rs's IEEE 754 totalOrder kernels after rewriting -0.0 to +0.0 on both sides: a NaN with the sign bit set is below every value, one without it above every value, and -0.0 equals +0.0. A column compared with a literal that is not a NaN (finite, zero or infinite, Float64 or Float32, on either side) becomes one fused expression, IEEE comparisons plus the expression grammar's signbit and is_nan (Fused expressions): x > c is (or (gt x c) (and (is_nan x) (not (signbit x)))), x < c is (or (lt x c) (and (is_nan x) (signbit x))), x = c and x <> c are the IEEE eq and its not. A comparison with a NaN literal is left (totalOrder tells NaN payloads apart). Nothing about the data hands such a filter back.

MIN and MAX over Float64 and Float32 run as the plan runner's own grouped min and max. DataFusion's grouped float MIN/MAX starts each group at the type's largest finite value (MIN) or lowest finite value (MAX) and replaces the running value whenever the new one compares lower (higher) or does not compare at all. So a group whose values are all -inf has a MAX of the lowest finite value (f64::MIN, f32::MIN) and one whose values are all +inf a MIN of the largest; -0.0 and +0.0 compare equal, so the first zero to arrive is kept; and a NaN makes the answer depend on the order the rows arrive in. Before the GPU call, MetalExec reads the value column once on the CPU (up to 8 threads, above 2,000,000 rows) for NaN, -0.0 and +0.0. A column with no NaN and at most one zero sign runs as the plan runner's min/max, with the infinities and the sign of a zero result set as DataFusion gives them. A column with a NaN or both zero signs adds four per-group counts (NaN, value, zero and negative-zero counts); a group with a NaN, or both zero signs and a zero result, hands the node back.

Joins. A replaced join gives DataFusion's answer for an inner, left or right HashJoinExec with equal keys only: a null in a key matches nothing on either side (SQL's =, DataFusion's NullEqualsNothing); a left join keeps every row of the build (left) side and a right join every row of the probe (right) side, with nulls for the other side's columns; the output columns are DataFusion's (the left input's, then the right's, through the join's projection). The plan runner's join keeps the order of its left input, and MetalExec makes the probe side that input for an inner and a right join, so a join whose output carries the probe side's ordering (DataFusion sorts the probe side below an inner join when the probe side is one partition) keeps it.

---

The differential grid

tests/grid.rs runs every query in a plain SessionContext and in one with the rule at ArrowMetalConfig::all().with_min_rows(0), and compares the answers. Every GROUP BY runs three times with the rule: forced onto the GPU (AggregateChoice::ArrowMetal), forced back to DataFusion (AggregateChoice::DataFusion), and with the measured choice looked up at 50M rows (table_rows).

• Tables:
• t, 13 columns: Int32 and Int64 keys; Float64 keys with ±0.0, ±inf and NaN; Float64 keys with -NaN and NaN payloads of both signs; Float32 keys with NaN, ±0.0 and ±inf; Utf8 keys and the same keys as Utf8View and as LargeUtf8; Int32 and Int64 values; Float64 values with NaN, NaN payloads and both zeros; Float64 values whose sums round; Float32 values with -NaN, a NaN payload and -inf.
• u, the other side of the joins (a fiftieth of t's rows, at least 10): Int32, Int64 and Utf8 keys that partly overlap t's and repeat on both sides, Float64 values with NaN and -0.0, and a Boolean column; e, the same columns with no rows.
• ti, float MIN/MAX data: groups whose values are only -inf, only +inf, both infinities, -inf and finite values, only -0.0 or only +0.0, in Float64 and Float32.
• 24 table configurations: 0, 1, 1,000 and 20,000 rows × null fraction 0, 0.1 and 1.0 on every column × two layouts (one partition with target_partitions 1; three partitions with target_partitions 4, which plans per-partition sorts under a merge and hash-partitioned joins).
• 460 queries:
• 154 ORDER BY (12 columns × ASC/DESC × NULLS FIRST/NULLS LAST/default × with and without LIMIT 7, plus 10 multi-column orderings);
• 52 GROUP BY (every aggregate over each key and value column, count over string columns, and MIN/MAX over ti);
• 228 WHERE: 18 mixed predicates, and over four float columns and vf every operator (=, <>, <, <=, >, >=) against +0.0, -0.0, +inf, -inf, 1.5 and -2.5, four with the literal on the left, and two against a NaN literal;
• 26 joins: inner, left and right joins on int32, int64, Utf8 and two-column keys, with an empty side, with an ORDER BY and a GROUP BY above, and the shapes the rule leaves (full, semi, anti, a condition besides the keys, IS NOT DISTINCT FROM, a Float64 key).
• Comparison: floats by bit pattern (so -0.0 ≠ +0.0, and NaN sign and payload must match), except float sum and every avg, compared within 1e-9 relative.
• Joins are forced onto the GPU in every variant (JoinChoice::ArrowMetal).

Run on 2026-10-01 (cargo test, debug build):

The pairs of the forced variant with nothing taken are a CAST in a predicate, a comparison with a NaN literal, the joins the rule leaves, and aggregates over an estimated row count (above a join), each left by the rule with its reason. tests/rule.rs checks both branches of the measured choice with the table looked up at 50M rows: count(*) over two int32 keys drawn from 1,000,000 values runs on the GPU, over 150 values it is handed back, on one batch per partition and on 8,192-row batches. A unit test in src/exec.rs checks that a prefix repeating a few thousand keys, followed by keys seen once, is not confirmed by the 2,048-row sample of the whole input.

---

Numbers

Machine and method. Apple M4 Max (16 CPU cores: 12 performance, 4 efficiency), 64 GB, macOS 27.0; rustc 1.95.0, DataFusion 55.1.0, arrow-rs 59.3.0; the benchmark is examples/bench.rs, built --release with thin LTO.

• SessionConfig::new(): 16 partitions, 8,192-row batches.
• Each query runs in a context without the rule (off) and in one with ArrowMetalConfig::all().with_min_rows(0).with_accept_inexact(true) (on); both answers are compared before anything is timed. A third context with ArrowMetalConfig::default() is only planned, and its decisions are recorded in the CSV's rule_default column. The aggregate files below time the default configuration itself (def).
• Timed: SQL to logical plan, physical planning (the rule runs there) and collect. Best of 5 wall-clock runs. CPU-ms is the process CPU time (user + system, all threads) of the best run; GPU time is not in it.
• datafusion_sort_warm_2026-09-29.csv: before the timed runs of each context, untimed runs repeat until they add up to 100 ms (the GPU runs small plans about three times slower after it idles and reaches its fast state after about 20-25 ms of back-to-back work). The file also records on_first_ms, the first run with the rule, taken with the GPU idle before it. Three passes at 250,000 to 2M rows, of which the tables show the median, each column on its own; one pass at 10M and 50M.
• Tables: MemTables with 16 partitions, of 8,192-row batches dealt round-robin (b8192, the layout DataFusion's scans produce) or one batch per partition (part1). Queries: SELECT q, k1, x FROM extra ORDER BY q (int64 key), … ORDER BY x (Float64), SELECT q, k1, y … ORDER BY y (Float32), SELECT name, q, k1 FROM extra ORDER BY name (String, 1,000 values); keys drawn uniformly (int64 in [0, 10^9), floats in [0, 1)).
• Before each block of queries, the harness waited for a 1-minute load average below 3.5 and no compiler or test process; the load at block start and end (1.49 to 3.39 in this file) is in every row.

Every timed row gave the same answer with the rule as without it, with 0 hand-backs.

Full sorts, rule off and on

8,192-row batches, from datafusion/results/datafusion_sort_warm_2026-09-29.csv (rows crate = now):

* The int64 sort in this layout was the first query each benchmark process ran, so its first run also compiled the GPU pipelines. The same query in the one-batch layout, run next in the same process, took 3.36, 3.44, 4.29, 7.27, 22.25 and 116.46 ms on its first run at the six sizes.

One batch per partition, off ÷ on, same file:

The first run from an idle GPU. A single ORDER BY after the GPU has idled takes longer than the warm figure: 1.95 against 1.11 ms for the Float64 sort at 250,000 rows, 127.94 against 87.90 ms at 50M. Every first-run figure in the 8,192-row table except the starred ones, and every one in the one-batch layout, is below DataFusion's warm time for the same query.

Over DataFusion's Parquet reader

SELECT id, price FROM f ORDER BY price (a Float64 key), with DataFusion's own Parquet scan feeding both runs, from datafusion/results/datafusion_rule_2026-09-29.csv (rows crate = now, case p_sort). The files come from Benchmarks/parquet_bench.py build.

The default configuration takes this sort: the scan reports an exact row count (the rule_default column reads TAKEN SortPreservingMergeExec (input rows 50000000 (exact) vs min_rows 250000)). This file was measured without the warm-up; in it, the in-memory sorts at 10M and 50M rows are within 13% of the warm-up file's figures, rule on. With the rule, input_time includes the Parquet decode.

A Parquet file with a string column

DataFusion 55.1 reads Parquet strings as Utf8View, which the rule carries and sorts by. A file the benchmark writes (the columns above, 1,048,576-row row groups), off and on, two rounds of best of 5 (best of 2 above 2 s), 8,192-row batches, from datafusion/results/datafusion_parquet_string_sort_2026-09-29.csv:

The default configuration takes both (exact row counts from the file's footer).

Aggregates with the default

Every GROUP BY and DISTINCT case of the sweep (with one and two int32 and int64 keys, 200 to about rows / 2 groups) and SELECT DISTINCT region, sub FROM fact (two int32 keys, 10,000 groups), with ArrowMetalConfig::default() timed against DataFusion alone, both layouts, 1M to 50M rows. A case whose default ran on the GPU: three rounds of best of 5 with the two contexts in rotated order, then three runs of each after 500 ms of sleep and three after 5 s, the contexts and the gaps alternating (the median of the three is the first-run time). Any other case (both contexts on the CPU): 30 runs of each, alternating run by run, after a 100 ms warm-up. The files: datafusion_groupby_default_2026-09-29.csv (every case, an earlier table) and the re-measurements after it, whose rows replace its rows for the same case, size and layout (…_taken_…, …_cap_…, …_idle_…, …_handback_…, …_recheck_gpu_…, …_recheck_…, …_recheck2_…, …_idle_vs_idle_…, …_idle_vs_idle_recheck_…, …_idle_gap_…, …_idle_gaps_…, …_idle_gaps_recheck_…, …_resweep_check_2026-10-01…, …_resweep_recheck_2026-10-01…, …_default_import_2026-10-01… and …_default_import_retime_2026-10-01… (their rows of the current crate), …_float_minmax_check_2026-10-01…, …_refit_check_2026-10-02…). …_idle_vs_idle_… holds every case whose decision an earlier version of this table changed and a random fifth of the rest; …_idle_gaps_… every case that version ran on the GPU, at both gaps (22 rows, 19:03 to 19:17 EDT, load 0.94 to 3.14 at the start of each block); …_idle_gaps_recheck_… every case whose decision the final table changed (18 rows, 19:51 to 21:42 EDT, load 1.50 to 3.44 at the start of each block). After the Float64 sum/avg and the min/max families were swept again (2026-09-30), …_resweep_check_… holds every case of those families that a table from that sweep alone replaced (256 rows, 03:34 to 08:34 EDT on 2026-10-01), and …_resweep_recheck_… every case whose decision the final table changed (the 30 min/max cases at 50M rows) and a seeded fifth of the rest (303 rows), 09:09 to 12:55 EDT, load 1.18 to 3.50 at the start of each block; in it the idle runs were taken only for the cases the default ran on the GPU from its 65th row on. After float MIN/MAX moved to the plan runner's own min/max and its two families were swept again (2026-10-01), …_float_minmax_check_… holds every Float64 and Float32 min/max case at 5M, 10M and 50M rows under a table from that sweep alone (240 rows, 23:35 on 2026-10-01 to 00:42 EDT, load 1.69 to 3.49 at the start of each block), and …_refit_check_… every case whose decision the final table changed (count(*) over two int32 keys at 10M rows) and a seeded fifth of the rest (356 rows, 00:48 to 04:04 EDT on 2026-10-02, load 2.95 to 3.50 at the start of each block). Every answer was equal to DataFusion's; no block of the last two files spans a sleep.

The cases are those of datafusion_groupby_sweep_2026-10-01.csv (360 per size, 288 at 2M rows) and SELECT DISTINCT region, sub in both layouts, each on the path the table gives it. In …_refit_check_… every case took the path this table gives it.

On the GPU (the latest measurement of each case: datafusion_groupby_default_import_2026-10-01.csv, its rows of the current crate, and datafusion_groupby_refit_check_2026-10-02.csv). Warm: best run, DataFusion alone ÷ the default. Idle vs idle: DataFusion alone's first run after the idle ÷ the default's first run after the same idle. GPU idle vs DataFusion warm: DataFusion alone warm ÷ the default's first run after idle, the comparison of a single query on an idle GPU with DataFusion's time when it has just run the same query. CPU-ms: process CPU time of the best warm run and of the median run after 500 ms of idle.

Over the 20 rows above (the latest measurement of each case: datafusion_groupby_default_import_2026-10-01.csv, its rows of the current crate, and datafusion_groupby_refit_check_2026-10-02.csv combined), at 50M rows the default was 2.31x to 4.27x faster warm (2.32x lowest on the median of the three rounds), 1.24x to 2.17x idle against idle after 500 ms and 1.11x to 1.68x after 5 s, and on its first run after idle 0.89x to 1.78x (500 ms) and 0.69x to 1.35x (5 s) of DataFusion's warm time. DataFusion alone's first run took 1.11x to 1.78x its warm time after 500 ms and 1.11x to 1.74x after 5 s; the default's took 1.92x to 3.82x and 2.58x to 4.06x. The min/max rows: 2.77x to 4.04x warm, 1.28x to 2.17x idle against idle after 500 ms and 1.11x to 1.68x after 5 s, and 0.98x to 1.78x (500 ms) and 0.90x to 1.35x (5 s) of DataFusion's warm time.

The idle ratios move between measurements of the same case. For the 22 cases in both …_idle_vs_idle_… and …_idle_gaps_…, the later 500 ms idle-against-idle ratio is 0.65x to 1.80x of the earlier one (median 1.00x); for the 6 cases measured after 5 s in both …_idle_gap_… and …_idle_gaps_…, 0.74x to 1.09x (median 1.02x).

count(*) over two int32 keys at 10M rows ran on the GPU under the earlier version of this table (100,000 and 1,000,000 groups). In …_default_import_… its first run after 5 s of idle was below DataFusion alone's first run after the same idle on two of its four rows with the current crate (0.81x at 100,000 groups with 8,192-row batches, 0.88x at 1,000,000 groups with one batch per partition) and on three with the crate before (0.84x to 0.94x), so the table takes it from 50M rows. At 10M rows the default now leaves all ten of its cases to DataFusion at plan time (…_refit_check_…): 0.928x to 1.065x of DataFusion alone on the best run and 0.992x to 1.008x on the median, where the earlier table ran the 100,000 and 1,000,000-group cases 2.58x to 2.88x faster warm (…_default_import_…, its rows of the current crate).

Handed back at run time: the other group counts of the taken shapes at 50M rows, both layouts (32 cases, the latest measurement of each: …_resweep_recheck_2026-10-01…, …_idle_vs_idle_…, …_idle_gaps_recheck_…, …_recheck_…, …_default_import_…, …_default_import_retime_… and …_refit_check_…): 0.980x to 1.030x of DataFusion alone on the best run and 0.971x to 1.081x on the median run; the group-count estimate took 0.018 to 0.242 ms.

Left at plan time: DataFusion's own plan in both contexts, with the rule's walk over the plan in one. The walk added 0.009 to 0.029 ms to planning (median, examples/plancost.rs, datafusion_plancost_2026-09-29.csv: 0.139 against 0.148 ms and 0.120 against 0.131 ms at 1M rows, 0.269 against 0.298 ms and 0.272 against 0.282 ms at 10M; plan and run 0.831 against 0.846 ms, 3.526 against 3.532 ms). The 350 left cases in …_refit_check_… were at 0.928x to 1.077x of DataFusion alone on the best run and 0.927x to 1.065x on the median run, none below 0.97x on both. The 344 left cases in …_idle_vs_idle_…, …_idle_vs_idle_recheck_… and …_idle_gaps_recheck_… were at 0.894x to 1.165x of DataFusion alone on the best run and 0.943x to 1.133x on the median run, none below 0.97x on both. The 298 left cases in …_resweep_recheck_… were at 0.901x to 1.143x on the best run and 0.927x to 1.053x on the median, one below 0.97x on both (avg(q) over one int64 key, 10,000 groups, 50M rows, 8,192-row batches: 21.27 against 22.54 ms); timed again alone, 200 runs of each alternating (datafusion_groupby_resweep_left_retime_2026-10-01.csv), it is at 1.000x best and 0.983x median.

Pipeline compilation (examples/coldstart.rs, datafusion_coldstart_2026-09-29.csv, 10M rows, three fresh processes per query): the first GPU query of a process compiles its Metal pipelines, 42 to 66 ms more than the same query's next run after the same 500 ms of idle (median per query: count(*) 52.8 ms, sum 54.1, min/max 49.1, DISTINCT 53.9, a full sort 49.7).

Where the time goes: a 50M-row sort

MetalExec's metrics for the best rule-on run, 8,192-row batches, from datafusion_sort_warm_2026-09-29.csv. Input is the wait for and collection of the input stream, import the copy of the collected batches into Metal buffers, sort the plan run on the GPU, export the hand-back of the result to arrow-rs; rest is the wall time outside those four (planning, the projection above, slicing the output into batches).

The sort itself is 67% to 75% of the wall time. The import writes each column's 6,104 chunks straight into that column's Metal buffers (Array::from_arrow_chunks), with no concat_batches copy before it.

The columns of one node imported at the same time

From 1,000,000 input rows MetalExec imports its columns on threads of their own, one per column, at the same time (each column's import keeps its own measured copy-thread count). Against the crate before the change, alternating block by block in one session on the same core library (datafusion_sort_import_2026-10-01.csv, the sort family with the rule forced on, warm, three rounds of best of 5), before ÷ now:

At 50M rows with 8,192-row batches the int64 sort's import is 9.85 → 8.07 ms (68.79 → 67.31 ms in all), the Float64 sort's 10.26 → 8.02 ms, the string sort's 10.10 → 7.83 ms. At 10M rows the process CPU time of a full sort is 18.4 to 27.1 CPU-ms, against 15.3 to 20.6 before (int64, 8,192-row batches: 18.4 → 27.1).

The aggregates the default runs on the GPU, timed the same way with ArrowMetalConfig::default() (datafusion_groupby_default_import_2026-10-01.csv, the default's GPU cases at 10M and 50M rows, both layouts, 24 rows): before ÷ now 0.942x to 1.139x on the best run and 0.976x to 1.320x on the median of the rounds, process CPU now ÷ before 0.85x to 1.41x. The first run after idle against DataFusion alone's first run after the same idle: now 1.24x to 2.19x after 500 ms and 0.81x to 1.68x after 5 s; before, in the same session, 0.89x to 2.15x and 0.84x to 1.48x (count(*) over two int32 keys at 10M rows was below 1.0x after 5 s in both crates: now 0.81x and 0.88x on two of its four rows, before 0.84x to 0.94x on three). The cases the default leaves to DataFusion at those sizes (20 rows) were at 0.936x to 1.154x best and 0.928x to 1.177x median of the crate before; the two below 0.95x on both, timed again alone (datafusion_groupby_default_import_retime_2026-10-01.csv, four blocks of each crate alternating, 104 runs of each context per block alternating run by run), are at 0.987x best and 0.988x median (min/max over int64, two int32 keys, 100,000 groups, 10M rows, 8,192-row batches: 18.88 → 19.13 ms) and 1.000x and 1.000x (SELECT DISTINCT region, sub, 50M rows, one batch per partition: 30.08 → 30.09 ms); both run DataFusion's plan in both crates.

Float MIN and MAX

SELECT k, min(x), max(x) FROM grid GROUP BY k (and over two keys k1, k2) with every replaced aggregate forced onto the GPU (AggregateChoice::ArrowMetal), x Float64 in [0, 1), against the crate before the change (which ran each extreme with four per-group helper counts), alternating block by block in one session (datafusion_minmax_2026-10-01.csv, warm, three rounds of best of 5, 8,192-row batches):

Over both layouts: now 1.95x to 3.53x faster than before on the best run and 1.90x to 3.54x on the median of the rounds, and 0.70x to 3.26x of DataFusion alone (behind it at about rows / 2 groups at 50M rows and over one key at 10M, and at 200 groups over one key at 10M rows). Over a Float32 column, which the crate before left to DataFusion, 0.44x to 2.49x of DataFusion alone.

Swept again for the measured table (datafusion_groupby_sweep_2026-10-01.csv: Float64 and Float32 value columns, one and two int32 and int64 keys, 200 groups to rows / 2, 1M to 50M rows, both layouts, rule forced on, 0 answers differ), DataFusion alone ÷ with the rule is 0.33x to 3.41x over Float64 and 0.29x to 2.63x over Float32. The 100,000-group series are at least 1.65x faster in their worst case from 5M rows (one int32 key, one int64 key, two int64 keys) and from 10M rows (two int32 keys); no other float series is at two sizes. With the default timed against DataFusion alone over those cases (datafusion_groupby_float_minmax_check_2026-10-01.csv), the GPU cases were 1.74x to 3.10x faster warm, and each of the four series is left by one of the other two rules at 50M rows:

So the measured table takes no float MIN/MAX shape.

Joins, rule on and off

DataFusion alone against the default configuration with every translatable join replaced (JoinChoice::ArrowMetal; an aggregate above the join stays DataFusion's, as under the default), datafusion_join_2026-10-01.csv: probe tables of 10M and 50M rows, build tables of 10,000, 1,000,000 and 10,000,000 rows holding every other key of the probe's key domain (half the probe rows match), keys as int64, int32 and Utf8 (decimal text of the same values), both layouts, warm (three rounds of best of 5), then three runs of each context after 500 ms and three after 5 s of idle (the median), the contexts and the gaps alternating. Warm is DataFusion alone ÷ with the rule, over the two layouts, for the count-and-sum query and for the GROUP BY query; idle vs idle is DataFusion alone's first run after the idle ÷ the rule's first run after the same idle, over the two queries and layouts. Every answer was equal to DataFusion's.

On integer keys the joins with a count and a sum above them are 1.16x to 2.29x faster warm; the GROUP BY of the probe key above the join, which DataFusion runs over the join's partitions (for a hash-partitioned join, after the same hash re-partitioning), is 1.03x to 1.64x. In every integer cell the rule's first run after idle is below DataFusion alone's first run after the same idle for some case at some probe size and gap (lowest 0.47x). Utf8 keys go through ArrowMetal's key densification before the hash join and are 0.22x to 1.60x warm.

A matmul chain in SQL (attention over coordinate-format matrices)

The query is single-head attention written as one SQL statement, from the attention test of the ddx project (crates/ddx-datafusion/tests/attention.rs, Apache-2.0), copied verbatim with its table names and its value function. Every matrix is a table of rows (i BIGINT, j BIGINT, val DOUBLE): the input x[t, d], the weights wq, wk, wv[d, e] and the target tgt[t, e]. Each matrix product is a SUM(a.val * b.val) over a JOIN … GROUP BY: the projections q, k, v (x joined with a weight matrix on d), the scores s (q joined with k on e), and the output o (the softmax weights joined with v); a max, a normalizing sum and a squared-error loss complete it. DataFusion inlines each common table expression at every read, so one plan holds 18 hash joins and 18 aggregates (53 nodes the rule looks at, counting each partial aggregate). The example is examples/attention.rs.

The joins produce t·d·e rows for each projection and t·t·e rows for s and for o before the grouped sums, counted with SQL for every shape (datafusion/results/attention_2026-10-03.csv, rows rows_*):

Three configurations, timed in the same block per shape in alternating order: alone, SessionContext::new(); default, session_context(SessionConfig::new(), rule) with ArrowMetalConfig::default(); forced, ArrowMetalConfig::all().with_min_rows(0) with AggregateChoice::ArrowMetal and JoinChoice::ArrowMetal. MemTables of 8,192-row batches over 16 partitions. Each configuration's first run gives the loss and the rule's report; then five rounds, in each of which every configuration, in an order rotated by one per round, runs untimed for at least 100 ms and then once timed; then three runs of each after 500 ms of idle. Timed: SQL to logical plan, physical planning and collect. CPU-ms is the process CPU time (user + system, all threads). Each shape's block started at a one-minute load of 2.97 to 3.48. From datafusion/results/attention_2026-10-03.csv:

From 1024, 512, 64 on, every configuration's later rounds are slower than its first (the CSV's wall_runs_ms rows, in round order); at 4096, 1024, 64 the first round's runs are 18,049.96 ms (alone), 18,536.87 ms (default) and 24,805.81 ms (forced), and the later rounds' 19,599.71 to 30,177.41 ms.

The loss. Every run of every configuration gives DataFusion alone's loss to within 1.464e-14 relative (the largest difference, over every run of every shape and configuration, against DataFusion alone's first run of the shape); DataFusion alone's own runs differ from each other by up to that same 1.464e-14. The report records no run-time choice and no runtime fallback under the default or the forced configuration.

What the rule did. Under both configurations, the 14 grouped sums of a product (SUM(x.val * w.val), SUM(q.val * k.val), SUM(a.val * v.val)) are left, since the argument of the sum is an expression, not a column, and their 14 partial aggregates with them; so are the loss's whole-table sum (no GROUP BY keys) and its partial, and the nine joins and three aggregates whose input row counts DataFusion gives as estimates (above a join or an aggregate), with the partial aggregates of two of those three. The remaining nine joins are the x ⋈ weight joins, whose inputs have exact row counts.

• 256, 128, 32: the default leaves those nine joins too, at 32,768 rows for the larger input against min_rows 250,000 (0 taken, 53 left); forced takes them (9 taken, 44 left).
• 512, 256, 64: the default leaves the nine joins at 131,072 rows against min_rows 250,000 (0 / 53); forced takes them (9 / 44).
• 1024, 512, 64: the nine joins pass min_rows (524,288 rows), and the default leaves them since no build-side bucket of the measured join table holds 32,768 rows (0 / 53); forced takes them (9 / 44).
• 2048, 512, 64: as at 1024 with 1,048,576 and 32,768 rows (0 / 53); forced takes the nine joins (9 / 44).
• 4096, 1024, 64: as at 1024 with 4,194,304 and 65,536 rows (0 / 53); forced takes the nine joins, each producing 268,435,456 rows (9 / 44).

With estimated row counts accepted. The forced configuration with with_accept_inexact(true) (the benchmark's on), against DataFusion alone in the same blocks, the same method, from datafusion/results/attention_inexact_2026-10-03.csv: it takes 21 nodes (all 18 joins and three aggregates: the maximum of each row of s, twice, and the normalizing sum, which ran on the GPU) and leaves 30 (the 14 grouped sums of a product with their partial aggregates, and the whole-table sum with its partial).

At 1024, 512, 64 the accepted configuration's first timed round is 801.28 ms and the next four 1,255.76 to 1,391.90 ms, with CPU-ms between 9,659.2 and 10,454.2 in all five. Its losses are within 5.297e-15 relative of DataFusion alone's. At 2048, 512, 64 the accepted configuration was not timed: the process ended without output after its first run (6,217.90 ms), and one process running that shape's first run alone and then accepted has a peak memory footprint of 55,534,428,720 bytes on this 64 GB machine. 4096, 1024, 64 was not run accepted.

---

To improve

The measured shapes where the rule is behind DataFusion alone. The default leaves top-k and filters to DataFusion, and the aggregate shapes below.

Top-k

ORDER BY … LIMIT 100, 8,192-row batches, rule off / rule on in ms (off ÷ on), the median of three rounds from datafusion_sort_warm_2026-09-29.csv (one round at 10M and 50M rows):

One batch per partition: 0.14x to 0.38x. DataFusion's TopK operator keeps the best 100 rows of each partition while its input streams; MetalExec first collects and imports every row. At 50M rows the Float64 top-k's input_time alone is 7.72 ms and its import 9.83 ms, against DataFusion's 6.21 ms for the whole query.

Aggregates

With every replaced aggregate forced onto the GPU (AggregateChoice::ArrowMetal), DataFusion alone ÷ with the rule, the range over one and two keys, int32 and int64 keys, both layouts and the family's queries, from datafusion_groupby_sweep_2026-10-01.csv (the float min/max rows measured on 2026-10-01, the Float64 sum/avg and int64 min/max rows on 2026-09-30). Of these cells the default takes, at 50M rows, the two-int32-key cases of count(*) at 100,000 and 1,000,000 groups and of DISTINCT at 200, 100,000 and 1,000,000 groups, and of min/max over int64 at 100,000 and 1,000,000 groups over two int32 keys and over two int64 keys, and at 1,000,000 groups over one int64 key.

At 1M and 2M rows the 1,000,000-group data holds at least rows / 4 groups and is in the rows / 2 row. The rule describes min/max over Float64 and over Float32 as one family, so a float series' worst case is the lower of the two.

The first run after the GPU idles. 38 series are at least 1.65x faster warm at two sizes or more; the default takes 10. The first run after 500 ms of idle (pipelines compiled) of every case the default has run on the GPU (including SELECT DISTINCT region, sub at 50M rows), over the latest measurement of each case and layout (datafusion_groupby_default_idle_2026-09-29.csv, datafusion_groupby_recheck_gpu_2026-09-29.csv, datafusion_groupby_idle_vs_idle_2026-09-29.csv, datafusion_groupby_idle_gap_2026-09-29.csv, datafusion_groupby_idle_gaps_2026-09-29.csv, datafusion_groupby_resweep_check_2026-10-01.csv, datafusion_groupby_resweep_recheck_2026-10-01.csv, datafusion_groupby_default_import_2026-10-01.csv, datafusion_groupby_float_minmax_check_2026-10-01.csv, datafusion_groupby_refit_check_2026-10-02.csv; the first two with DataFusion's idle run always before the default's, the others alternating):

The 32 cases measured in both …_idle_… (DataFusion's idle run first) and …_idle_vs_idle_… (alternating) differ between the two: the later idle-against-idle ratio is 0.69x to 1.51x of the earlier one (median 0.93x).

Series left at 50M rows by the first run after 5 s of idle, lowest DataFusion idle ÷ default idle over the cases and layouts: count(*) over two int32 keys, 10,000 groups 0.94 (datafusion_groupby_idle_gaps_2026-09-29.csv), and DISTINCT over two int32 keys, 10,000 groups 0.84 (SELECT DISTINCT region, sub, one batch per partition, the same file); min/max over int64 with one int32 key, 100,000 groups 0.84, and with one int64 key, 100,000 groups 0.80, and sum/avg over Float64 with two int64 keys, 100,000 groups 0.91 (datafusion_groupby_resweep_check_2026-10-01.csv); float min/max with one int32 key, 100,000 groups 0.98, and with two int64 keys, 100,000 groups 0.96 (Float32 both, datafusion_groupby_float_minmax_check_2026-10-01.csv). The 16 other series that qualify warm and are left at 50M have no first run after 5 s measured there (count(*) and DISTINCT over one key and over two int64 keys, sum/avg over int64 over two keys). DISTINCT over two int32 keys at 200 groups is left at 10M rows by its first run after 5 s: 0.53x and 0.50x; count(*) over two int32 keys at 100,000 and 1,000,000 groups is left at 10M rows by its first run after 5 s: 0.81x and 0.88x (datafusion_groupby_default_import_2026-10-01.csv).

The hand-back. Handing an aggregate back at run time (reading the first batches, estimating the groups, then running DataFusion's plan) cost, against DataFusion alone, measured run by run (datafusion_groupby_default_handback_2026-09-29.csv, the shapes the earlier table replaced), on the best run / the median run: at 2M rows a median of 0.969x / 0.967x, lowest 0.934x and 0.921x (the latter count(*) over one int64 key, 864,909 groups: 5.30 → 5.61 ms best); at 5M a median of 0.984x / 0.984x, lowest 0.936x and 0.914x; at 10M 0.990x / 0.991x, lowest 0.884x and 0.968x; at 50M 0.993x / 0.995x, lowest 0.944x and 0.924x. Series left at 50M because a hand-back of their shape there is below 0.97x (best / median, datafusion_groupby_resweep_check_2026-10-01.csv): min/max over int64 with one int32 key, 1,000,000 groups (minmax_int_1i32_10k, 0.994x / 0.951x); sum/avg over Float64 with two int32 keys, 100,000 groups (sum_f64_2i32_200, 0.989x / 0.925x) and with two int64 keys, 1,000,000 groups (sum_f64_2i64_200, 0.986x / 0.942x); and (datafusion_groupby_float_minmax_check_2026-10-01.csv) float min/max with one int64 key, 100,000 groups (minmax_f32_1i64_R2, 0.974x / 0.952x) and with two int32 keys, 100,000 groups (minmax_f64_2i32_200, 0.984x / 0.951x). In that file the float shapes' hand-backs at 5M rows were 0.930x to 1.007x best and 0.940x to 1.014x median (23 of 48 below 0.97x on one of the two), at 10M 0.948x to 1.042x and 0.954x to 1.051x (12 of 64), at 50M 0.951x to 1.093x and 0.951x to 1.074x (9 of 64). The 16 series above with no first run after 5 s at 50M have hand-backs of their shape there below 0.97x as well, by shape, with the shape's lowest hand-back at 50M (best / median; the latest measurement of each case in the check files): count(*) over one int32 key, 100,000 and 1,000,000 groups (0.961x / 1.003x); over one int64 key, 100,000 and 1,000,000 groups (0.948x / 0.943x); over two int64 keys, 100,000 and 1,000,000 groups (0.972x / 0.958x); DISTINCT over one int32 key, 1,000,000 groups (0.968x / 0.996x); over one int64 key, 100,000 and 1,000,000 groups (0.901x / 1.039x); over two int64 keys, 100,000 and 1,000,000 groups (0.953x / 1.002x); sum/avg over int64 with two int32 keys, 100,000 and 1,000,000 groups (0.990x / 0.924x); with two int64 keys, 100,000 and 1,000,000 groups (0.944x / 0.992x).

Filters

With ArrowMetalConfig::all(), from datafusion_rule_2026-09-29.csv:

DataFusion's FilterExec streams batch by batch; MetalExec collects the whole input first.

Float comparisons against a literal, now one fused expression each (see DataFusion's semantics), with the rule switched on for every node it translates, from datafusion_filter_2026-10-01.csv (warm: three rounds of best of 5; after idle: the median of three runs after 500 ms and three after 5 s, the contexts and the gaps alternating; f_gt: SELECT region, amount FROM fact WHERE amount > 1000.0, f_lt0: … WHERE amount < 0.0, f_ne0: SELECT region, qty FROM fact WHERE amount <> 0.0, f_sum_ge0: SELECT sum(qty), count(*) FROM fact WHERE amount >= 0.0):

Every case is behind DataFusion alone warm (0.31x to 0.79x), so filter stays off by default.

A matmul chain in SQL with the joins taken

The attention query of A matmul chain in SQL, DataFusion alone ÷ the rule, from attention_2026-10-03.csv and attention_inexact_2026-10-03.csv:

• Forced (the nine x ⋈ weight joins taken), 4096, 1024, 64: 18,049.96 / 24,805.81 ms best (0.73x) and 22,959.19 / 29,472.75 ms median (0.78x); each of the nine joins produces 268,435,456 rows. At 2048, 512, 64: 4,086.14 / 4,326.88 ms best (0.94x), 4,621.95 / 4,619.84 ms median.
• With estimated row counts accepted (all 18 joins and three aggregates taken), 1024, 512, 64: 1,092.99 / 1,299.61 ms median (0.84x), while the best run is 1,077.51 / 801.28 ms. At 2048, 512, 64 one process running the shape's first run alone and then accepted has a peak memory footprint of 55,534,428,720 bytes, and the shape was not timed.

The first query of a process

The first GPU query in a process also compiles the Metal pipelines. In the table above, the int64 sort's first run at 250,000 to 1,000,000 rows (31.67 to 34.32 ms, starred) is slower than DataFusion's warm time for the same query (8.08 to 31.75 ms).

---

Limits

• It collects its input. MetalExec reads every input partition to the end before it sorts, joins or runs an aggregate on the GPU (an aggregate it hands back reads only the first batches of each partition before DataFusion takes over); a join reads both of its inputs to the end. It has no spill path of its own. The collected batches and the result are counted in DataFusion's memory pool (a MemoryConsumer named MetalExec, at the bytes each batch's slices reference); when the pool refuses, MetalExec hands the node back to DataFusion, whose operators can spill. The Metal buffers themselves (the imported columns) are not counted in the pool.
• The GPU call is not cancelled. It runs inside one tokio::task::spawn_blocking call; when the query is dropped, it runs to its end and its result is discarded.
• One GPU job at a time per process. A process-wide lock serialises every MetalExec, so two queries that both reach the GPU run their GPU parts one after the other.
• One blocking thread per node. All GPU work of one MetalExec runs inside one tokio::task::spawn_blocking call, off DataFusion's async worker threads. From 1,000,000 input rows its columns are imported on threads of their own, one per column, at the same time; each imported column is then used only by that blocking thread.
• Exact row counts. By default a node is taken only when DataFusion's statistics give its input an exact row count (a join: both inputs). A sort above a filter, a join or an aggregate has an estimate and is left unless accept_inexact is set; an unknown count is left unless take_when_unknown is set.
• Aggregates. The measured table takes three shapes: count over two int32 (or narrower) keys of a MemTable scan with at least 10,000,000 rows, and DISTINCT over the same keys and MIN/MAX of an integer column over one int64 key or two integer keys with at least 50,000,000 rows. Every other aggregate is left, including every aggregate over a Parquet scan, a filter or a join, and every one with a float or string key.
• Joins. The measured join table takes no join; JoinChoice::ArrowMetal replaces every translatable one. A join reads both inputs to the end before it runs, where DataFusion's hash join streams its probe side.
• Pipeline compilation. The first GPU query of a process compiles its Metal pipelines: 42 to 66 ms at 10M rows (datafusion_coldstart_2026-09-29.csv).
• macOS on Apple silicon only. build.rs stops the build for any other target with a message naming the target; arrowmetal-sys refuses non-macOS targets as well.
• DataFusion 55.1.0 only. The dependency is pinned with =; DataFusion's physical-plan API changes between releases.

---

Tests and files

cd datafusion
ARROWMETAL_LIB=/path/to/libArrowMetalC.dylib cargo test                       # 42 tests + 1 doc-test
ARROWMETAL_LIB=/path/to/libArrowMetalC.dylib cargo test --test rule -- --ignored   # default config at 2M rows

On 2026-10-02: 42 passed, 2 ignored, doc-test passed.

The first two CSVs also hold rows with crate = before: an earlier state of this crate, measured in the same sessions. The tables on this page use the rows with crate = now.

pandas on the Apple silicon GPU

ArrowMetal gives pandas two ways onto the GPU, and you pick how much of your code changes.

Both live in the same package and can be used together.

# macOS arm64. pandas is not a hard dependency.
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
    swift build -c release --product ArrowMetalC
export PYTHONPATH=python

import arrowmetal never imports pandas: the bridge is loaded the first time you touch am.from_pandas, am.to_pandas, am.pandas_bridge or am.pandas_accel.

---

Tier 1: the .am accessor

import pandas as pd, arrowmetal as am
import arrowmetal.pandas_bridge      # registers .am on Series and DataFrame

df = pd.read_parquet("trades.parquet", dtype_backend="pyarrow")   # Arrow-backed: zero-copy

df["price"].am.sum()                    # scalar
df["price"].am.nlargest(100)            # Series, index taken along
df["symbol"].am.contains("BRK")         # boolean Series
df.am.groupby("symbol").sum("size")     # DataFrame indexed by the key
df.am.sort_values(["symbol", "price"])
df.am.merge(reference, on="symbol")
df.am.query(am.filter(am.col("price") > 100).sum(am.col("size")))   # one fused kernel

Series accessor

DataFrame accessor

---

Tier 2: zero-code-change accel mode

import arrowmetal.pandas_accel
arrowmetal.pandas_accel.install()        # or set ARROWMETAL_PANDAS_ACCEL=1

# ... your existing pandas script, unchanged ...

print(arrowmetal.pandas_accel.stats())

or without touching the script at all:

python -m arrowmetal.pandas_accel my_script.py
ARROWMETAL_PANDAS_ACCEL=1 python my_script.py

A call is routed to the GPU only when all four hold:

1. dtype — the column is numeric, boolean or utf8, in any of the three pandas storage flavours (numpy-backed, pd.ArrowDtype / int64[pyarrow], or the masked nullable dtypes Int64, Float64, boolean, string), including pandas 3's default str dtype;
2. size — the frame is at least threshold rows, 2,000,000 by default (install(threshold=...), set_threshold(...), or ARROWMETAL_PANDAS_ACCEL_THRESHOLD). Below that the launch latency costs more than the kernel saves. The measured row counts behind that judgement, per operation, are in The crossover: where the GPU path overtakes the fastest CPU library, and where it overtakes the engine's own CPU loop. This threshold is the pandas caller's veto on top of the engine (it counts the one-time page mapping of a pandas-owned column, which a column already resident in Metal does not pay); a call that passes it still goes through the engine's CPU/GPU router (Design), which gives byte-identical output on either path;
3. arguments — the call uses arguments the kernels implement exactly (the "not routed" column below);
4. the operation is worth it — see the next section. Five reductions, abs and the scalar comparisons are intercepted but deliberately left to pandas.

Everything else runs in pandas, unchanged. Any Exception inside the GPU path is caught, recorded in stats(), and the original pandas method is run instead, so a GPU failure does not surface as a program failure. The one exception is repeated install()/uninstall() cycling — see Limits.

Why some operations are intercepted but not routed

Handing a pandas column to Metal is zero-copy — the GPU reads the very bytes pandas holds, and am.zero_copy_report proves it by address — when the buffer is page aligned, which it is for every allocation at these sizes; a small unaligned buffer costs one copy. But it is not free: the pages have to be mapped into the device's address space once. On an M4 Max, for a 50M-row (400 MB) int64[pyarrow] column:

The kernel is 4.6x faster than pandas. The map is not, and one sum cannot amortise it. So the default routing table leaves alone every operation that reads each byte once and writes at most one byte back — sum, min, max, mean, count, abs, and the six scalar comparisons — and routes the ones that do enough per byte to pay for the map many times over: sorts, hash group-bys, merges, string scans, round, isin, nunique, value_counts, boolean-mask selection.

That table is data, not a hard-coded rule:

accel.NEVER_BY_DEFAULT     # ('sum', 'min', 'max', 'mean', 'count', 'abs', 'eq', 'ne', ...)
accel.ROW_FACTOR           # op -> rows needed as a multiple of the threshold, or None for "never"
accel.route_all()          # route everything anyway
accel.ROW_FACTOR["sum"] = 1

The .am accessor has no such table: it always dispatches to the GPU — bar the per-row host fallback the Unicode case transforms take on non-Latin rows — and s.am.to_metal() gives you the mapped column so that a chain of operations pays the map once. Nothing about correctness changes either way — only where the work runs.

What is intercepted

Anything not in that table — apply, rolling, pivot_table, resample, describe, arithmetic between columns, and every dtype outside the list (categorical, datetime, timedelta, object, interval, period) — is untouched pandas.

Control

from arrowmetal import pandas_accel as accel

accel.install(threshold=5_000_000)
accel.set_threshold(1_000_000)
with accel.disabled():                  # a block of guaranteed plain pandas
    ...
accel.stats()                           # what ran where
accel.stats().to_frame()                # the same as a DataFrame: gpu / cpu / errors per op
accel.reset_stats()
accel.uninstall()                       # every original method back
accel.REGISTRY                          # the contract, as data
accel.ROW_FACTOR                        # the routing table, as data
accel.route_all()                       # route the operations pandas is otherwise faster at

stats() reports four things per operation: gpu (routed), cpu (fell through, with the reason implied by the table above), errors (the GPU path raised and pandas ran instead — each with the exception text), and intercepted, the total number of calls that reached a patched method at all.

---

Zero-copy: what happens

Series.array._pa_array is a pyarrow.ChunkedArray for any Arrow-backed pandas column. The bridge takes its single chunk and hands those buffers to Metal — the same bytes, no copy. Apple silicon's unified memory means the GPU then reads the pages the CPU wrote, with no transfer either.

am.zero_copy_report(df) does not guess: it compares the buffer address pandas holds with the one handed over.

>>> am.zero_copy_report(pd.Series(pd.array([1, 2, 3], dtype="int64[pyarrow]")))
{'zero_copy': True, 'reason': 'Arrow-backed pandas column, buffers shared',
 'dtype': 'int64[pyarrow]', 'arrow_type': 'int64'}

The "none" rows are copy-free when the buffer is page aligned, which it is for every allocation at these sizes; a small unaligned buffer costs one copy.

Results come back the same way: am.to_pandas(...) and every .am method wrap the Metal buffers in a pd.arrays.ArrowExtensionArray, so the returned Series shares memory with the GPU result. Accel mode adds one cast on top, to give you back the dtype pandas itself would have produced.

Nulls, NaN and the index

• A numpy float column has no null, so NaN becomes Arrow null on the way in. That is exactly what skipna=True — pandas' default — already means, so sum, mean, min and max agree.
• An Arrow-backed float column has both, and the bridge does not touch either: a NaN stays a NaN and a null stays a null.
• Coming back, accel mode restores pandas' own convention: a null becomes NaN for a numpy float result, False for a plain bool result, True for a plain bool result of != (NaN != x is True in numpy), and the dtype's own NA for a nullable dtype. The rule is not hand-written per operation: each wrapper runs the original pandas method over a two-row probe of the same dtype and copies the dtype and NA convention it sees.
• The index is carried along by the same permutation as the values — sort_values, nlargest, nsmallest and boolean-mask selection all return the original index rows, in the new order. ignore_index=True gives a fresh RangeIndex, as in pandas. Group-by results are indexed by the key (or by a MultiIndex for several keys), and as_index=False resets it, as in pandas.
• Group-by follows pandas: null keys are dropped unless dropna=False, groups come back sorted by key unless sort=False (then in first-seen order), and a group whose values are all null sums to 0.
• Inner merge preserves left-row order and gives the result a fresh RangeIndex, as pandas does, with the same _x/_y suffixing for overlapping columns.

The one deliberate difference

sort_values on the GPU is stable. pandas' default is kind="quicksort", which is not, so rows with equal sort keys can come back in a different order than pandas would give them (the same order as kind="stable"). Every other value, dtype and index matches. If that matters, use the accessor explicitly or keep sort_values out of the accelerated set with accel.disabled().

Numbers

Measured with PYTHONPATH=python python Benchmarks/pandas_bench.py <rows> <iters> on an Apple M4 Max, pandas 3.0.5 / pyarrow 25.0.1 / Python 3.13, best of five, on the same in-process frame. The accessor and accel rows include the conversion the dtype forces, the map into Metal, the GPU work, and the trip back into a pandas object — nothing is pre-converted or cached between iterations, so these are the numbers a pandas user gets, not kernel times.

routes to is where accel mode sent the call with the default table; the accessor column is always the GPU, so it also shows what those operations would cost if you routed them.

Arrow-backed frame (pd.ArrowDtype, what read_parquet(dtype_backend="pyarrow") gives you)

numpy-backed frame (plain pd.DataFrame({...}))

Every column has to be converted to Arrow first, and pandas' numpy kernels are much faster than its pyarrow ones, so the speedups are smaller — and round/isin are left to pandas here.

Read it this way: an Arrow-backed frame is where the GPU pays. Group-by, sort, top-k, merge and string scanning are 6x to 26x on a 50M-row frame with no code change at all; the operations pandas already does at memory bandwidth are left alone and cost nothing; and the whole-frame operations (sort_values on a DataFrame, df[mask], merge) carry every column across, which is why they land lower than the single-column ones.

Two caveats visible in the numbers. The GPU CPU-ms column (in the benchmark output, not repeated here) is roughly a third of the wall time on group-by and sort, so the cores are free while the GPU works — a real advantage the wall-clock ratio understates. And a numpy float column costs about 2 ms per million values to convert, because building the validity bitmap from the NaNs is a single-threaded pyarrow pass: that is the whole difference between the accessor's 13 ms for abs on an Arrow-backed 50M-row float column and its 111 ms on the numpy one, and it is why every single-pass float operation stays in pandas there.

Limits

• Only these dtypes. Numeric, boolean and utf8. Categorical, datetime, timedelta, object, interval and period columns are left to pandas: their semantics are richer than the kernels here, and a wrong answer is worse than a slow one. (The bridge converts categoricals and datetimes fine — am.from_pandas handles them; it is the accel layer that will not route them.)
• merge is the validate="m:1" inner join. Duplicate or null keys on the right frame, and any outer/left/right join, fall back. Many-to-many joins are not supported in accel mode.
• str.upper/str.lower are ASCII-guarded in accel mode. The kernels implement Unicode's simple 1:1 mapping over every script, but pandas applies the full mapping (ß → SS), so accel mode checks for pure ASCII on the GPU and falls back otherwise; it never returns a different string than pandas would. The .am accessor runs the simple mapping on anything.
• str.contains with a real regex falls back. A literal pattern runs on the GPU.
• Float reductions add in a different order. A GPU tree reduction is not bit-identical to pandas' pairwise sum; expect agreement to about 1e-12 relative, not to the last bit.
• Below the threshold nothing is accelerated. At a million rows most of these operations are already sub-millisecond in pandas and the GPU launch cannot pay for itself. The default of two million rows is deliberately conservative; lower it if your columns are wide.
• Single-pass operations are not accelerated at all, at any size: sum, min, max, mean, count, abs and the scalar comparisons, plus round and isin on a numpy-backed column. The measurement and the reasoning are above; route_all() overrides it, and the .am accessor never applies it.
• A numpy float column costs about 2 ms per million values to convert, because pyarrow builds the validity bitmap from the NaNs in a single-threaded pass. Arrow-backed columns skip it entirely — which is the single biggest thing you can do to make this faster: read your data with dtype_backend="pyarrow".
• Install once. install() and uninstall() rewrite pandas' method slots. Cycling them a hundred-odd times in one process can leave an already-specialized x[mask] call site bound to whichever __getitem__ CPython first saw — an interpreter-level artifact of repeatedly rewriting a dunder. Use accel.disabled() to turn the layer off for a block instead.
• Threads. The wrappers are re-entrancy-guarded per thread, but the GPU queue underneath is a single device queue; several Python threads calling into it will serialise. disabled(), set_threshold() and route_all() are process-wide, not per thread: while one thread is inside a disabled() block every thread runs in pandas. The answers do not change — only where the work happens, and those calls land in neither stats().gpu nor stats().cpu.

Tests

swift build -c release --product ArrowMetalC
PYTHONPATH=python python -m pytest python/tests/test_pandas.py -q

python/tests/test_pandas.py covers: a GPU round trip for twenty pandas dtypes; zero-copy proved by buffer address for each Arrow-backed flavour and disproved for each copying one; every accessor method against its pandas equivalent; and, for accel mode, forty-plus operations run twice — once in plain pandas, once with the layer installed — over numpy-backed, Arrow-backed and masked frames carrying nulls, NaNs, strings and duplicate keys, with assert_series_equal / assert_frame_equal checking values, dtype and index. Each of those also asserts that the GPU path really was taken, so a test cannot pass by quietly falling back. The rest of the file is the fallback surface: unsupported dtypes, unsupported arguments, regex patterns, non-ASCII case changes, NaN in isin, duplicate and null merge keys, an injected exception inside the GPU path, the row threshold in both directions, uninstall() restoring every original, and the ARROWMETAL_PANDAS_ACCEL=1 and python -m arrowmetal.pandas_accel entry points in real subprocesses.

numpy

How a numpy array reaches the GPU, what crosses without a copy and what does not, what the benchmark matrix measures against numpy, and what is not implemented. Every number here is from Benchmarks/results/full_matrix_2026-09-07.csv or from python/tests/test_numpy.py, which asserts each claim in the first two sections on every run. That CSV is the eager run, and the numpy rows quoted below are numpy's plain eager idiom; the published baseline is the parallel run in Benchmarks/results/full_matrix_2026-09-07-parallel.csv, which measures each CPU library's most parallel idiom too (numpy has none, so its numpy-parallel column is empty), and The full matrix and Where ArrowMetal needs to improve are written against it.

1. The crossing

There is no numpy bridge, because none is needed: pyarrow already speaks numpy, and ArrowMetal speaks Arrow.

import numpy as np, pyarrow as pa, arrowmetal as am

x = np.random.rand(50_000_000)          # float64, the numpy default
col = am.array(pa.array(x))             # no copy: pyarrow adopts the numpy buffer, Metal wraps it
r = col.sqrt()                          # GPU, software IEEE-754 binary64
y = r.to_arrow().to_numpy(zero_copy_only=True)   # no copy out either

Two things make this copy-free, and both are checked:

• pa.array(x) adopts the numpy buffer for integer and float dtypes: the Arrow array's data buffer has the numpy array's address.
• am.array(...) wraps a page-aligned buffer in a Metal buffer in place. macOS hands numpy page-aligned memory for every allocation we measured from 1,000 elements up to 50M; a buffer that is not page aligned (tiny arrays of ten or a hundred elements were not) is copied once on the way in, which costs nothing worth measuring at that size.

test_numpy.py proves the import is in place by writing to the numpy array after the import and reading the new value back through the Metal column, and proves the export is in place by comparing the address of to_numpy(zero_copy_only=True) with the Arrow buffer's.

2. What crosses and what it costs

Results come back the same way: any numeric result with no nulls is a numpy view over the Metal buffer through to_numpy(zero_copy_only=True); a result with nulls needs to_numpy(zero_copy_only=False), which materialises NaNs, or stays as Arrow.

NaN is a value, not a null. pa.array(np.array([1.0, np.nan])) has a null count of zero, and ArrowMetal treats it the way numpy does: sum and mean return NaN. That is numpy's semantics and Arrow's default; a pandas column takes the other path (from_pandas=True), where NaN becomes null and is skipped, and pandas covers it. If you want numpy's NaNs skipped on the GPU, pass from_pandas=True to pa.array and pay one pass to build the validity bitmap.

numpy has no nulls, so the GPU column has no validity bitmap and every kernel takes its no-null fast path.

3. Measured against numpy

The benchmark matrix (Benchmarks/full_matrix.py) compares each operation with pandas, and where pandas has no vectorised idiom of its own it measures numpy directly on the same values, without a validity bitmap because numpy has none. Those rows, at their largest measured size, from the 2026-09-07 run on an Apple M4 Max (numpy 2.5.3, the eager idiom, best of up to five calls after a warm-up; cores = CPU-ms / wall ms):

Two readings. numpy runs every one of these on one core, so this is one CPU core against the GPU, with the crossing free because the memory is shared; a 16-thread CPU library is the other comparison, and it is on the same Compare tab. The transcendentals are the narrow rows: ln at 1.1x and sin at 2.0x are software binary64 on a GPU with no double hardware against a vectorised libm, and they are listed in Where ArrowMetal needs to improve as open work.

4. Limits

• NEP 18 dispatch is not implemented: np.sum(x) on a Metal column runs numpy's kernel, not the GPU's.
• Multi-dimensional arrays are not Arrow arrays; pass one column (x[:, i] is strided, so it copies once; np.ascontiguousarray first if you will reuse it).
• bool costs one pass in each direction because of the bit packing.
• Strings are converted, in and out.
• A numpy array smaller than about a thousand elements is usually not page aligned and is copied on import; at that size the dispatch floor, not the copy, is the cost, and the GPU is not the right place for it anyway (see the latency family in the matrix).

ArrowMetal runs Apache Arrow compute on the Apple silicon GPU: Arrow arrays that live in Metal shared memory and GPU kernels that keep Arrow's semantics, reached from Python through a ctypes wrapper over libArrowMetalC.dylib whose input and output go through the Arrow C Data Interface, so it composes with pyarrow, Polars, pandas and DuckDB.

Apple silicon has one physical memory shared by the CPU and the GPU, so an Arrow buffer placed in a Metal shared buffer is at once a valid CPU Arrow buffer and a valid GPU buffer: a column is used where it already is, with no copy across a bus in either direction. While a kernel runs, the CPU is free for the rest of the application, and every table in the repository shows the CPU time of each call next to its wall time.

One row, the same in-process data in every column:

From Benchmarks/results/full_matrix_2026-09-07-parallel.csv, Apple M4 Max (16 CPU cores, 64 GB), best of up to five calls after one warm-up, release build; the 339-row matrix that row comes from, including the 77 rows where the CPU idiom is ahead, is in docs/BENCHMARKS_MATRIX.md.

pip install arrowmetal          # macOS 14 or later on Apple silicon; pyarrow is the only dependency
python -m arrowmetal.bench      # 30 seconds or less
python -m arrowmetal.bench --parquet data.parquet   # read, sum, filter, group-by on your own file

The second line generates 10,000,000 rows, runs sum, filter, sort and group-by sum through pyarrow (and Polars when it is installed) and through ArrowMetal, and checks every ArrowMetal answer against pyarrow's. It prints one table for your Mac, wall and CPU milliseconds per call, with a block ready to paste into a benchmark result issue or the #benchmarks channel of the Discord linked from the README; nothing is sent. The dataset (int64 values uniform in [-1,000,000, 1,000,000) with 10% nulls, standard-normal float64s, int32 keys uniform over 1,000 values) is drawn with pyarrow.compute from seeded SplitMix64 streams, so the command needs nothing beyond pip install arrowmetal.

python -m arrowmetal.bench --parquet data.parquet runs on your own file instead. It reads the file's integer, floating-point and string columns with pyarrow, Polars and ArrowMetal, then runs sum and filter > median on the largest numeric column and group-by sum keyed on the lowest-cardinality integer or string column, CPU against Metal, with the same protocol and every ArrowMetal answer checked against pyarrow's. The report, the Share it block and the prefilled issue link carry the file's row count, column count, row groups, size and codecs and the timings, never its path, column names or values. A file whose columns would take more than a quarter of physical memory is refused, with the limit printed (the bench holds the columns up to four times at once), and a file with no integer, floating-point or string column gets one line saying so.

import pyarrow as pa, arrowmetal as am
amount = am.MetalArray.from_arrow(pa.array([10.0, 20.0, 30.0, 40.0]))
region = am.MetalArray.from_arrow(pa.array([1, 2, 1, 2], pa.int32()))
print(am.group_by([region]).sum(amount).to_arrow())     # [40, 60]: GPU group-by sum, one row per region
print(amount.filter_where(">", 15).to_arrow())          # [20, 30, 40]: GPU filter, back as a pyarrow array

Repository, documentation and the other language bindings: https://github.com/singhpratech/ArrowMetal. Site: https://arrowmetal.org.

Install

Two ways: a wheel built with scripts/build_wheel.sh and installed with pip install python/dist/arrowmetal-0.4.0-*.whl (macOS 14 or later on Apple silicon; the wheel bundles libArrowMetalC.dylib and the Polars expression plugin libarrowmetal_polars.dylib, and pyarrow is its only dependency), or from this repository.

From source, the development path. Needs the Swift toolchain:

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
  swift build -c release --product ArrowMetalC
# that builds .build/release/libArrowMetalC.dylib
pip install pyarrow
PYTHONPATH=python python -c "import arrowmetal as am; print(am.device_name())"

From a wheel, which carries both dylibs and needs no Swift or Rust toolchain at install time:

# the wheel build frontend, once
pip install build
# swift build, cargo build of the Polars plugin, then the wheel
scripts/build_wheel.sh
# pyarrow comes with it
pip install python/dist/arrowmetal-0.4.0-*.whl
# optional bridges
pip install "$(echo python/dist/arrowmetal-0.4.0-*.whl)[polars,duckdb,pandas]"
python -c "import arrowmetal as am; print(am.device_name())"

The wheel is arrowmetal-0.4.0-py3-none-macosx_14_0_arm64.whl; the build script prints its size. It is macOS arm64 only: it links Metal and holds an arm64 binary. Extras polars, duckdb, pandas and test pull in the libraries the corresponding bridges and the test suite want; none of them are needed to import arrowmetal.

The first thing to run needs nothing beyond the wheel itself (pyarrow comes with it):

import pyarrow as pa, arrowmetal as am

# this small array is copied in; large pyarrow buffers are page aligned
# and are borrowed; out is zero-copy
col = am.array(pa.array([1, None, 3, 40]))
big = col.filter_where(">", 2)                 # GPU
print(big.sum(), big.to_arrow())               # 43  [3, 40]

keys = am.array(pa.array([0, 1, 0, 2], pa.int32()))
print(keys.group_by(3).sum(col).to_arrow())    # [4, null, 40]

With the polars extra (or any installed Polars), a Series crosses through Arrow the same way:

import polars as pl
col = am.array(pl.Series([1, None, 3, 40]).to_arrow())
print(pl.from_arrow(col.filter_where(">", 2).to_arrow()))

Float64 columns (NumPy's and pandas' default) run entirely on the GPU: compare, filter, take, sum and arithmetic use a software IEEE-754 implementation that is bit-exact with the CPU. Use with am.batch(): around a chain of operations to pay one GPU round trip instead of one per call.

See Benchmarks/python_gpu_bench.py for a side-by-side with Polars, pyarrow.compute and pandas on the same data.

Building a wheel

pip install build                    # the wheel build frontend
scripts/build_wheel.sh               # swift build, then the wheel (cargo builds the Polars plugin)
# just the wheel, when .build/release/libArrowMetalC.dylib exists
python/build_wheel.sh
# install the wheel and Polars into a fresh virtualenv and run all four Polars tiers
scripts/check_wheel.sh

python/build_wheel.sh copies .build/release/libArrowMetalC.dylib into python/arrowmetal/_lib/, builds the Polars expression plugin with cargo build --release in polars-plugin/ against that same dylib and copies libarrowmetal_polars.dylib beside it, replaces the plugin's build-machine rpath with @loader_path (so it loads the packaged libArrowMetalC.dylib), re-signs both, runs python -m build --wheel in python/ with the platform tag macosx_14_0_arm64, checks that both dylibs really are inside the archive, and prints the wheel's path and size. scripts/build_wheel.sh is the same thing with swift build -c release --product ArrowMetalC in front. The version comes from one place, __version__ in arrowmetal/__init__.py; pyproject.toml reads it.

ARROWMETAL_DYLIB points the script at a dylib somewhere other than .build/release, ARROWMETAL_POLARS_PLUGIN_DYLIB at an already built plugin (then cargo is not needed), and PLAT_TAG overrides the platform tag.

scripts/check_wheel.sh [wheel] installs the wheel with its polars extra (Polars from PyPI) into a fresh virtualenv in a temporary directory, and from there, with a scrubbed environment (no PYTHONPATH, ARROWMETAL_* or DYLD_* variables, no cargo on PATH), runs a tier-1 group-by, tier-2 sum, filter_sum and device expressions, a tier-3 collect_gpu and a tier-4 MetalEngine collect against Polars' own answers, checks that the process loaded exactly one libArrowMetalC.dylib, the packaged one, and runs python -m arrowmetal.bench (10,000,000 generated rows) and python -m arrowmetal.bench --parquet on a 1,000,000-row file.

Where the dylib is found

arrowmetal looks for libArrowMetalC.dylib in this order:

1. $ARROWMETAL_LIB, the full path to a dylib. It wins over everything else, so one specific build can be pinned; a bundled arrowmetal/_lib/ copy in the same checkout never shadows it. Setting it to a path that does not exist is an error, not a silent fall-back to some other dylib.
2. Bundled in the installed package, arrowmetal/_lib/libArrowMetalC.dylib — what a wheel install has.
3. The development build, .build/release/libArrowMetalC.dylib beside a source checkout (then .build/debug, then /usr/local/lib and /opt/homebrew/lib).

When none of the three exists the import fails with an OSError that names all three and how to satisfy each.

Tests

pip install pytest
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift build -c release --product ArrowMetalC
PYTHONPATH=python python -m pytest python/tests -q

python/tests/test_arrowmetal.py checks import and round trip for int64/float64/float32/bool/string with nulls, reductions, compare/filter/filter_where, take, slice, cast, arithmetic (float64 included), boolean logic, group-by, the string kernels, with am.batch(): against the unbatched results, and Polars interop. Every result is compared to pyarrow.compute or plain Python. Polars tests skip when Polars is absent; the suite needs a real Metal device.

Index arrays — argsort, top_k, partition_nth_indices, lexsort_indices, am.join, rank and the window functions' row_number / rank / dense_rank — are pa.uint32(), row numbers up to 2^32 - 1 (a longer input raises). pyarrow returns uint64 for the same functions, with the same values. take accepts int32, int64 and uint32 arrays, NumPy arrays and lists (python/tests/test_index_type.py).

Not covered by the dense-key MetalArray.group_by(key_count) path: min/max on 64-bit values, mean on Float32, and Float64 values for any aggregate; python/tests pins those as expected errors. am.group_by(keys) runs every aggregate on Float64 values (python/tests/test_arrowmetal.py::test_group_by_arbitrary_keys_matches_pyarrow).

Polars

Four tiers, all in this repository — see docs/POLARS.md:

import polars as pl, arrowmetal as am

# -> dict[str, MetalArray], zero copy
am.from_polars(df)
# tier 1: the GPU around Polars
df.arrowmetal.group_by("k").sum("v")
# tier 2: inside a lazy plan (the wheel carries the plugin)
pl.col("v").arrowmetal.sum()
# tier 3: Polars runs the plan, the GPU finishes it
lf.arrowmetal.collect_gpu(query)

import arrowmetal still does not import Polars: the bridge loads on first use (or at import when Polars is already loaded). Tier 2 is a Rust plugin that a wheel built from this tree carries in arrowmetal/_lib/; from a source checkout it is one extra build, cd polars-plugin && cargo build --release. The plugin is built against the Polars 1.44 plugin ABI. The other tiers are pure Python over the Arrow C Data Interface. PYTHONPATH=python python -m pytest python/tests/test_polars.py -q runs the suite.

Swift

The native API. The Python package, the C ABI and every other binding call these types. Everything on this page is in Getting started, Sources/ArrowMetalExamples/main.swift or the tests under Tests/ArrowMetalTests, and the numbers are the ones those cite.

Install

// Package.swift
dependencies: [.package(url: "https://github.com/singhpratech/ArrowMetal", from: "0.4.0")],
targets: [.target(name: "MyApp", dependencies: [.product(name: "ArrowMetal", package: "ArrowMetal")])]

macOS 14 or later, iOS 17 or later (Package.swift); Swift 6.3.3 is the toolchain the tests run under. Build and test from a checkout with the Xcode toolchain selected:

export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
swift build -c release
swift test -c release        # release is required: a release-only miscompile has bitten this project once

Example

import ArrowMetal

let prices = try MetalArray<Float>([9.5, nil, 12.0, 3.25])       // nullable, lives in Metal shared memory
let mask   = try prices.compare(.gt, 5)                          // GPU, Arrow boolean bitmap out
let picked = try prices.filter(mask)                             // GPU stream compaction
print(try picked.sum(), try picked.max(), picked.nullCount)      // .float(21.5), 12.0, 0

// Multi-column batches behave like an Arrow RecordBatch.
let orders = try MetalRecordBatch(names: ["region", "amount"], columns: [.int32(region), .float32(amount)])
let hits = try orders.filter(try region.compare(.eq, 2).and(try amount.compare(.gt, 100)))
let window = try hits.slice(offset: 32, length: 1000)             // zero-copy view

// Chains share one GPU round trip.
let total = try MetalContext.shared.batch {
    try amount.filter(try region.compare(.eq, 2).and(try amount.compare(.gt, 100))).sum()
}

batchAsync is the same without the wait; return a pending array from the body and use the async accessors (sumAsync and friends) for scalars. Six end-to-end scenarios (analytics on a record batch, feature preparation, Float64 with NaN, C Data Interface interop, sliced windows, one batched command buffer) run with swift run -c release arrowmetal-examples.

What the API is

The copy rule

Copy-free out, always. Copy-free in when the producer's buffers are page aligned; one copy otherwise. importArrowArray reports which it did. Details and the measurements behind them are in Design and numpy.

Tests

1,046 test functions in 87 files under Tests/ArrowMetalTests, run in release; the oracles are plain-Swift CPU references, hand-computed vectors and pyarrow answers pinned as literals (Testing). Tests that need a real GPU skip on virtual Metal devices.

Numbers

The Swift-level baselines (one core, all 16 cores, Accelerate) are in Getting started and Benchmark method; the operation-by-operation matrix is measured from Python over the same kernels (The full matrix).

Limits

• One GPU, one device; no multi-GPU.
• The kernels are generated Metal Shading Language, compiled at first use and cached per process, so the first call of each kernel family pays a compile of a few milliseconds.
• Everything the other pages list under limits applies here first, since this is the layer they wrap.
• Index arrays (argsort, topK, partitionNthIndices, lexsortIndices, hashJoin, the ranks) are MetalArray<UInt32>, row numbers up to 2^32 - 1; a call past that throws. take accepts MetalArray<Int32>, MetalArray<Int64> and MetalArray<UInt32> (Design).

C

One header, one dynamic library, no Swift in sight for the caller. include/arrowmetal.h declares 286 entry points over libArrowMetalC.dylib; every other binding in this repository (Python, Rust, Go, TypeScript, R, the Polars plugin, the DuckDB extension) is built on it, so it is tested by all of their suites as well as by python/tests, which calls it through ctypes, and by a test that compiles the header as C.

Build the library

export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
swift build -c release --product ArrowMetalC        # .build/release/libArrowMetalC.dylib

Conventions

• Handles: am_array* is an opaque, Metal-resident Arrow array; am_release frees it.
• Every function returns 0 on success and non-zero on error; am_last_error() has the message, thread-local, valid until the next call on that thread. Argument guards return 2 without setting a message.
• In and out through the Arrow C Data Interface: am_import moves the caller's ArrowArray (its release is nulled) and borrows the schema; am_export fills an ArrowArray/ArrowSchema pair whose release callback owns the buffers. am_import_device/am_export_device speak the C Device interface with ARROW_DEVICE_METAL; am_stream_* speak the C Stream interface.
• Scalars are passed as a pointer to a value of the array's compute element type: am_compute_format names it (for a dictionary array am_format reports the index type, and the kernels compute on the values).
• Copy-free out, always. Copy-free in when the producer's buffers are page aligned; one copy otherwise.
• Index arrays are uint32 (I): am_argsort*, am_top_k*, am_lexsort*, am_partition_nth*, am_join and the ranks (am_window ops 0-2, am_rank_ex, a plan's row_number / rank / dense_rank). A call whose row numbers would pass 2^32 - 1 returns an error. am_take accepts int32, int64 and uint32 indices. Dictionary codes, index_in, group ids, inverse_permutation and list_parent_indices stay int32; indices_nonzero is uint64 (Design).
• A column held as several arrays (a ChunkedArray, one column of a stream of RecordBatches) imports in one call: am_import_chunks(schema, arrays, n, &out) takes n contiguous ArrowArray structs of the one type schema describes and returns one array of their total length, the same array am_import of their concatenation gives. Each chunk's offset, length, validity bitmap (or its absence) and null count (-1 included) are honoured, and each chunk is copied straight into the final buffers on the CPU cores in parallel, with no concatenated copy in between. On success every chunk is moved; on failure a chunk whose release is still set is still the caller's. It takes the integer, float, float16, boolean, null, decimal, temporal, interval and fixed_size_binary types, utf8 / large_utf8 / binary / large_binary (a total over 2 GB is refused, as for one large_utf8 array) and utf8_view / binary_view; for any other type (dictionary, nested, run-end encoded, extension) it returns 3 and reads and moves nothing, and am_import_chunks_supported(schema) answers the question up front. One chunk is am_import of it.

Example

#include "arrowmetal.h"

struct ArrowSchema schema; struct ArrowArray array;   /* filled by any Arrow producer */
am_array* col = NULL;
if (am_import(&schema, &array, &col) != 0) { fprintf(stderr, "%s\n", am_last_error()); return 1; }

int64_t two = 2; am_array* mask = NULL; am_array* hits = NULL;
am_compare_scalar(col, 4 /* gt */, &two, &mask);
am_filter(col, mask, &hits);

int64_t sum; double f; int kind, is_null;
am_reduce(hits, 0 /* sum */, &sum, &f, &kind, &is_null);

struct ArrowSchema out_schema; struct ArrowArray out_array;
am_export(hits, &out_schema, &out_array);             /* copy-free; the consumer calls out_array.release */
am_release(hits); am_release(mask); am_release(col);

Reductions, element-wise compare and arithmetic, casts, boolean logic, selection, sorts and top-k, group-by with its aggregates, strings, temporal, decimal, nested types, hashing and set lookup, window functions, joins, the fused expression runner (am_query), the JSON plan runner (am_plan_*), Parquet, the streaming executor (am_stream_*), batching (am_batch_*) and the device interfaces. The header's comments are the reference for each; Arrow functions, one by one maps every Arrow function name to its entry point.

Limits

• Thread-local errors mean a green-threaded runtime must read am_last_error on the thread that made the call (the Go binding pins the OS thread around every call for this reason).
• am_import does not report whether it wrapped or copied; a caller infers it from buffer alignment.
• macOS arm64 only: the library links Metal.

arrow-rs is where Rust keeps Arrow-shaped data. This document is how you point that data at the GPU.

Two crates live in rust/:

Both are on crates.io at 0.4.0; a path or git dependency on a checkout works too.

For Apache DataFusion, datafusion/ holds datafusion-arrowmetal, built on the arrowmetal crate: a physical optimizer rule that runs full ORDER BY sorts on the GPU with DataFusion's answers (DataFusion).

Everything below was run in this repository on 2026-09-07 on an Apple M4 Max, macOS 26.6.2, rustc 1.95.0, arrow-rs 59.3.0, ArrowMetal 0.1.0.

---

Install

ArrowMetal is a Metal library. It needs an Apple silicon Mac; there is no other target.

# 1. Build the GPU library. This is the only non-cargo step.
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
    swift build -c release --product ArrowMetalC

That produces .build/release/libArrowMetalC.dylib. Then, in your own crate:

# Cargo.toml
[dependencies]
arrow = "59"
arrowmetal = "0.4.0"          # crates.io; or { path = "../ArrowMetal/rust/arrowmetal" } from a checkout

and build with the dylib's location known:

ARROWMETAL_LIB=/path/to/libArrowMetalC.dylib cargo build --release

The dylib comes from the swift build above, or from the Python wheel: pip install arrowmetal places it at site-packages/arrowmetal/_lib/libArrowMetalC.dylib, and the crates were checked against that copy (a scratch crate depending on arrowmetal = "0.1.0" from crates.io, built with ARROWMETAL_LIB pointing at the wheel's dylib, prints 0.1.0 on Apple M4 Max).

Finding the dylib

arrowmetal-sys/build.rs searches, first hit wins:

1. $ARROWMETAL_LIB — the full path to the dylib. This is the one to set.
2. $ARROWMETAL_LIB_DIR — a directory holding it.
3. <repo>/.build/release, then <repo>/.build/debug — a SwiftPM build in the ArrowMetal checkout.
4. /usr/local/lib, /opt/homebrew/lib.

Anything else is a hard build error naming every path it looked in and the swift build line, rather than a link that succeeds and a dyld failure at run time.

The dylib's install name is @rpath/libArrowMetalC.dylib, so the directory that was found is baked into arrowmetal's own binaries, examples and tests as an LC_RPATH entry. Your binary needs the same entry, and then no DYLD_LIBRARY_PATH at run time: without it the link succeeds and dyld reports no LC_RPATH's found on launch (measured with the published crate). Because Cargo hands a build script's link arguments only to the crate that owns it, arrowmetal-sys republishes that directory through its links = "ArrowMetalC" key (DEP_ARROWMETALC_LIB_DIR) and arrowmetal/build.rs repeats the -rpath for its own binaries, examples and tests. A crate that depends on arrowmetal and builds a binary should do the same:

// your-crate/build.rs
fn main() {
    // DEP_ARROWMETALC_LIB_DIR is visible only to crates that depend on arrowmetal-sys directly,
    // so read the same variable the build searched with.
    println!("cargo:rerun-if-env-changed=ARROWMETAL_LIB");
    if let Ok(lib) = std::env::var("ARROWMETAL_LIB") {
        if let Some(dir) = std::path::Path::new(&lib).parent() {
            println!("cargo:rustc-link-arg=-Wl,-rpath,{}", dir.display());
        }
    }
}

Without a build script, one environment variable at build time does the same:

RUSTFLAGS="-C link-arg=-Wl,-rpath,$(dirname "$ARROWMETAL_LIB")" cargo build --release

Both were run against the published arrowmetal = "0.1.0" and the wheel's dylib; the binary then starts from any directory. If you move the dylib after building, set DYLD_LIBRARY_PATH or rebuild.

---

Example

This is the body of rust/arrowmetal/examples/quickstart.rs, so it is compiled by every cargo build --examples. Run it with cargo run --release --example quickstart; it prints 2 groups: [0, 1] -> [10, 27] and sum of everything: Some(Int64(42)).

use arrow::array::Int64Array;
use arrowmetal::{group_by, Array, CompareOp};

fn main() -> Result<(), arrowmetal::Error> {
    let region = Int64Array::from(vec![0i64, 1, 0, 2, 1]);
    let amount = Int64Array::from(vec![Some(10i64), Some(20), None, Some(5), Some(7)]);

    let region = Array::from_arrow(&region)?;   // to the GPU
    let amount = Array::from_arrow(&amount)?;

    let big = amount.compare_scalar(CompareOp::Gt, 5i64)?;   // boolean mask
    let kept_amount = amount.filter(&big)?;                  // 10, 20, 7
    let kept_region = region.filter(&big)?;

    let gb = group_by(&[&kept_region])?;
    let totals = gb.sum(&kept_amount)?.to_arrow()?;          // back to arrow-rs
    let keys = gb.keys(0)?.to_arrow()?;

    let ints = |a: &dyn arrow::array::Array| -> Vec<i64> {
        a.as_any().downcast_ref::<Int64Array>().unwrap().values().to_vec()
    };
    println!("{} groups: {:?} -> {:?}", gb.group_count(), ints(&keys), ints(&totals));
    println!("sum of everything: {:?}", amount.sum()?);
    Ok(())
}

---

The copy rule

Copy-free out always; copy-free in when the producer's buffers are page aligned, one copy otherwise.

The decision is made per buffer, so a nullable column can have its values wrapped and its validity bitmap copied. That case is common; see the table.

Out (Array::to_arrow) is always copy-free. ArrowMetal's buffers are MTLBuffers in shared memory; it hands their pointers straight to the C Data Interface with a release callback, and arrow-rs reads the GPU's memory in place. Measured over inputs that all carry nulls, so both buffers are exercised: the exported values and validity pointers were page aligned in every case (tests/copy_rule.rs::arrowmetal_exported_buffers_are_page_aligned), and the export call itself took 0.001 ms (median) for a 10M-row column.

In (Array::from_arrow) is copy-free only when the buffer pointer is aligned to a 16 KiB page, because that is what MTLDevice.makeBuffer(bytesNoCopy:) requires; otherwise that buffer is copied once. Whether an arrow-rs array clears the bar is a property of the allocator, not of arrow-rs, so it is measured rather than assumed. 32 allocations at each size, exported through arrow::ffi and read back as the pointers am_import receives:

So in practice:

• Values are wrapped from a few thousand rows up, and copied below that.
• A nullable column of fewer than roughly 130,000 rows normally has its validity bitmap copied, even when its values are wrapped. The bitmap is one bit per row, so the copy is about a kilobyte at 8,192 rows and about 16 KB at the point it stops happening — small, but real.

Both are the system allocator serving a large enough request with mmap and handing back page-aligned memory. The bitmap is eight times smaller than the values, so it crosses that threshold eight times later. None of it is a guarantee — a custom global allocator, a different macOS, or a buffer sliced out of an arena can all change it. Run cargo test --test copy_rule -- --nocapture in rust/ to print the table for your own machine.

Copy-free is not free: at 10M int64 rows the import still cost 1.11 ms, which is Metal mapping 80 MB of pages into the GPU's address space. See the timing below.

Two more claims, inferred rather than tested:

• A slice does not cost a copy either way. An Arrow offset is carried on the handle rather than applied, so array.slice(7, n) should import the same buffers the unsliced array would.
• An array that came out of ArrowMetal and goes back in is never copied in either buffer, whatever the original alignment: ArrowMetal recognises its own exported buffers and shares the MTLBuffer objects directly.

Both follow from reading Sources/ArrowMetal/CInterop.swift plus the alignment measurement above, and both have tests behind their observable halves — a sliced array round-trips exactly (round_trip_of_a_sliced_array, a_slice_survives_the_round_trip), and an ArrowMetal export is always page aligned, which is the precondition for the no-copy path (arrowmetal_exported_buffers_are_page_aligned). But whether a given import copied is not observable through the C ABI: the Swift side computes an ImportResult.zeroCopy flag and am_import discards it. The ABI does not report it, so these two are arguments, not measurements, and are labelled as such.

---

What is covered

Each row is backed by a test in rust/arrowmetal/tests/ against arrow-rs's own kernel on the same data, at lengths 0, 1, 33, 1024, 1025, 100,001 and 1,000,001, with and without nulls.

Element types: Int64 and Float64 are swept against arrow-rs. Boolean arrays are exercised as comparison output and filter masks, and UInt32 as sort indices: argsort, argsort_with, top_k_with and lexsort return an arrow UInt32Array, the type sort_to_indices returns (row numbers up to 2^32 - 1; a longer input is an error), and take accepts Int32, Int64 and UInt32 indices. Every other Arrow type the ABI supports reaches the GPU through the same entry points but has no test in this crate — the Swift and Python suites cover them (Testing).

Errors are Result<T, arrowmetal::Error>, and the message is am_last_error()'s, read on the spot because that slot is thread-local.

Handles (Array, GroupBy, Source, PlanResult) are !Send and !Sync on purpose: the ABI's error slot and its command-buffer batching are both thread-local. Use one set of handles per thread. That is asserted at compile time (assert_not_impl_any!), not just documented.

Scalar operands, and the one type that is refused

The ABI reads sizeof(element type) bytes through a void*, so a scalar of the wrong width would be an out-of-bounds read. Every scalar entry point therefore checks the Rust type against the array's own am_format string first: compare_scalar(Gt, 1i32) on an Int64Array is an Err, not a read off the end.

That check is only as good as am_format, and there is exactly one Arrow type for which am_format does not describe what the kernels compute on. A dictionary-encoded array reports its index type ("i") while every kernel decodes the dictionary and computes on the value type. A 4-byte i32 scalar would be accepted for a Dictionary(Int32, Float64) column whose kernel then reads 8 bytes — unsound from safe Rust, and wrong besides (compare_scalar(Lt, 2i32) returns [false, false, false] where arrow-rs says [true, false, false], and the correct 2.0f64 scalar is rejected).

Array::from_arrow therefore refuses DataType::Dictionary outright, with an error naming the key and value types and telling you to decode first:

let decoded = arrow::compute::cast(&dict, &DataType::Float64)?;
let gpu = arrowmetal::Array::from_arrow(decoded.as_ref())?;   // now type-checks correctly

This is a limitation of the C ABI, not of the Arrow type — the kernels handle dictionaries correctly, and Swift and Python callers use them. It is an ABI defect on ArrowMetal's side: am_format reports the C Data Interface's top-level format, which for a dictionary is its index type. The entry point that reports the type the kernels compute on exists (am_compute_format, include/arrowmetal.h:32); this crate does not use it and refuses dictionaries, which keeps the sentence above true for everything this crate accepts. tests/compute.rs::dictionary_arrays_are_refused_at_import pins the rejection and the decode path.

sort and argsort keep ArrowMetal's default order, which is Arrow C++'s: nulls last in both directions, -0.0 tied with +0.0, and NaN next to the nulls. arrow-rs orders differently — nulls where SortOptions::nulls_first says, floats by total_cmp (-NaN < -inf < … < -0.0 < +0.0 < … < +inf < +NaN) — and the _with forms take that order as an option:

use arrow::compute::SortOptions as ArrowSortOptions;
let opts: arrowmetal::SortOptions = ArrowSortOptions { descending: true, nulls_first: true }.into();
let idx = gpu.argsort_with(opts)?;          // what arrow::compute::sort_to_indices orders
let top = gpu.top_k_with(100, opts)?;       // its first 100 rows, by GPU selection
let multi = arrowmetal::lexsort(&[&a, &b], &[opts, opts])?;

From<arrow::compute::SortOptions> sets float_order: FloatOrder::Total; SortOptions::default() is the plain sort. FloatOrder::NanLargest is the third float order, Polars': every NaN one value above +inf in both directions, -0.0 tied with +0.0. Neither option adds a pass to the GPU sort: the null placement is where the partition that takes the null rows out of the radix sort puts them, and totalOrder is the key map in front of the radix passes. tests/sort_options.rs compares every combination against arrow::compute::sort bit for bit.

One divergence from arrow-rs, found and pinned

min / max on a float column containing NaN. The two libraries use different, each internally consistent, rules — neither is a bug:

arrow-rs orders NaN at the top of a total order. Both min and max say so in the same sentence — "For floating point arrays any NaN values are considered to be greater than any other non-null value" (arrow_arith::aggregate, 59.3.0) — so min returns the smallest non-NaN and max returns NaN. That was reported as apache/arrow-rs#101 and closed as intended behaviour in 2022.

ArrowMetal skips NaN in both, the way it skips a null, and reports "no valid value" when every valid element is NaN. That follows Arrow C++ / pyarrow on the mixed case; on an all-NaN column pyarrow returns NaN where ArrowMetal returns null, which the C header already documents.

tests/compute.rs::nan_handling_diverges_from_arrow_rs_and_is_pinned asserts the ArrowMetal and arrow-rs columns and fails if either moves. The pyarrow column is not asserted from Rust — pyarrow is not a dependency of this crate; it is pinned in the Python suite, by test_nan_is_skipped_by_min_and_max_in_both and test_all_nan_min_max_is_null_in_arrowmetal_and_nan_in_pyarrow in python/tests/test_differential.py. With no NaN in the data the two agree exactly, which is what every other reduction test relies on.

---

What is not wrapped

include/arrowmetal.h has 286 entry points. arrowmetal-sys declares 41 of them — every one called by the safe crate, none declared and unused — and the safe crate covers the list above. Everything below is reachable from Swift, Python and the C ABI, and not from this crate. There is no technical obstacle to any of it; it is unwrapped because it is untested here, and an untested wrapper does not go into 0.1.0.

Wrapping any of them is mechanical: add the extern "C" line to arrowmetal-sys/src/lib.rs, the method to arrowmetal/src/lib.rs, and a test to arrowmetal/tests/compute.rs against arrow-rs (or a hand oracle where arrow-rs has no counterpart). tests/signatures.rs will check the new declaration against the header on the next cargo test.

---

The measured timing

One run, 2026-09-07, Apple M4 Max, macOS 26.6.2, rustc 1.95.0, arrow-rs 59.3.0, ArrowMetal 0.1.0.

Method. One 10,000,000-element Int64Array of pseudo-random values in [-1_000_000, 1_000_000), no nulls, built once and shared by every row. Filter predicate x > 0, which keeps 5,000,125 of the 10,000,000 rows (50.0%). std::time::Instant around the call, wall time, single-threaded, nothing subtracted, std::hint::black_box on every input and result. "Single-threaded" describes the harness: it spawns no threads and pins nothing, and arrow-rs's kernels ran with their default parallelism, which for sum and filter is none — arrow-arith and arrow-select depend on no thread pool and these are single-threaded, autovectorised kernels, so arrow-rs got everything it has here. Three untimed warm-up iterations, then five timed ones; the table is the best of the five (medians were within a few percent, and are in rust/README.md). Both libraries' answers are asserted equal before anything is timed. Outside a batch every ArrowMetal call commits its command buffer and waits, so a "kernel" number is a complete GPU round trip, not an enqueue. Source: rust/arrowmetal/examples/bench.rs; reproduce with cargo run --release --example bench in rust/.

• kernel — the GPU call on an array already imported, mask already on the GPU. This is what each step of a longer chain costs.
• end to end — what one operation on an arrow-rs array costs, import included. What that covers differs by row: the sum row is import + the reduction and has no export, because a reduction returns a scalar through out-parameters rather than an array; the compare + filter row is import + compare + filter + to_arrow. Supporting numbers (best of 5): import 1.11 ms, export 0.000 ms.

Where arrow-rs is ahead. On a single sum over an arrow-rs array, arrow::compute::sum is ahead by 2.3×: 0.91 ms against ArrowMetal's 2.11 ms. The kernel is 3.3× faster; the gap is entirely the 1.11 ms it takes to hand 80 MB (decimal MB; 76 MiB) to Metal, plus about 0.7 ms of handle setup and first-touch. The import is copy-free at this size (the values buffer came back aligned to 4 MiB), so that is page mapping, not a memcpy — and it is still 1.1 ms that one cheap kernel does not earn back.

The break-even is roughly "more than one pass over the data": compare + filter is two passes and ArrowMetal is already 1.5× faster end to end. Import once, chain, export once. A single wrapped reduction does not earn its import back.

---

Limits

• Apple silicon only. arrowmetal-sys/build.rs refuses any non-macOS target; on Intel macOS it fails at the dylib search instead.
• Two element types are tested against arrow-rs: Int64 and Float64, plus Boolean masks and Int32 indices. Anything else works through the same entry points but is untested from Rust.
• One thread per handle set. Handles are !Send and !Sync; am_last_error and batching are thread-local.
• No RecordBatch type of its own. Columns go across one at a time. The plan runner's Source takes a named set of columns, or a slice of arrow-rs RecordBatches (Source::from_batches, each column imported with Array::from_arrow_chunks).
• arrowmetal-sys declares 41 of the ABI's 286 entry points, and every one of them is called by the safe crate (nothing is declared and unused). The table above lists what is missing.
• The docs.rs build of 0.1.0 failed. Under DOCS_RS the -sys build script returns before it emits anything, so the env!("ARROWMETAL_SYS_LIB_DIR") and env!("ARROWMETAL_LINKED_LIB_DIR") lookups in the two crates have nothing to read and rustdoc stops. The documentation for 0.1.0 is this file and the crate-level comment in rust/arrowmetal/src/lib.rs.
• arrow is pinned to major version 59. The C Data Interface structs are ABI-stable, so a different arrow-rs major would very likely work, but it is not tested here.
• A batch defers validation. Inside batch(|| …) an operation that would fail — take with an out-of-range index, say — returns Ok at the call and fails the whole batch at the end, which batch reports as an Err. The value the closure produced is discarded: a readback (a reduction, an export) forces a mid-batch flush, so some kernels in a failed batch have run and some have not, and the ABI does not say where the line fell. Nested batch calls are safe — only the outermost opens and commits one — but that is counted in this crate, not in the ABI: am_batch_end closes whatever is open regardless of nesting.
• No async. Every call is synchronous: outside a batch each one commits its command buffer and waits. The ABI has an async path (Sources/ArrowMetal/Async.swift); this crate does not use it.

Testing

cd rust
ARROWMETAL_LIB=/path/to/libArrowMetalC.dylib cargo test --release

66 tests, plus 5 no_run doc-tests (compiled, not executed); 0 failures on 2026-10-02 (Testing). What each file compares against is in rust/README.md and in Testing.

A Go module that hands an arrow.Array from Apache Arrow Go to the GPU and takes the answer back, over the Arrow C Data Interface. It wraps a deliberately small part of the C ABI: import and export (one array or a column in chunks), the four reductions, compare, filter, take, sort, argsort, top-k and lexsort (with per-key null placement and float order), group-by, and the JSON plan runner. Everything it wraps has a test against Arrow Go's own compute or against a plain Go loop on the same data. Everything it does not wrap is listed under What is not wrapped.

Module path: github.com/singhpratech/ArrowMetal/go/arrowmetal. Source: go/arrowmetal/.

---

Install

# 1. The GPU library. It is a build product of the Swift package; `go get` cannot fetch it.
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
    swift build -c release --product ArrowMetalC

# 2. The module.
go get github.com/singhpratech/ArrowMetal/go/arrowmetal

The module is tagged go/arrowmetal/v0.4.0: the module lives in a subdirectory, so the Go module proxy wants the tag prefixed with the module's path within the repository, not a bare vX.Y.Z. A checkout works too: clone the repository and add

replace github.com/singhpratech/ArrowMetal/go/arrowmetal => /path/to/ArrowMetal/go/arrowmetal

to your own go.mod, or just run inside go/arrowmetal.

Requirements: macOS on Apple silicon, a Metal device, and Go 1.25.0 or newer (the module's go directive, which is what arrow-go/v18 requires) with cgo enabled — CGO_ENABLED=1, the default when a C toolchain is present; Xcode's clang is what this was built and tested with. The module depends on github.com/apache/arrow-go/v18 and nothing else. Everything here was measured on Go 1.27.1.

Finding the dylib

The binding does not link libArrowMetalC.dylib at build time. It dlopens it the first time you use the package, so the location is a run-time decision and a missing library is a clear Go error rather than a dyld message. The search, in order:

1. $ARROWMETAL_LIB — the full path to the dylib, not a directory. If it is set and does not load, that is the error; there is no fallback.
2. .build/release/libArrowMetalC.dylib, and the same at one, two and three levels up, relative to the working directory. (go test runs in the package directory, so ../../.build/release is the repository's own build; a program run from the repository root finds .build/release.)
3. The same four paths relative to the directory holding the running binary.
4. Plain libArrowMetalC.dylib, letting dyld search DYLD_LIBRARY_PATH, /usr/local/lib and the rest.

When none of them loads, the error names ARROWMETAL_LIB, every path it tried and why each failed, and the swift build line that produces the file. arrowmetal.Init() performs the load and returns that error, so a program can fail early with it; every other entry point calls Init first. arrowmetal.LibraryPath() reports where it came from.

---

Example

package main

import (
    "fmt"

    "github.com/apache/arrow-go/v18/arrow/array"
    am "github.com/singhpratech/ArrowMetal/go/arrowmetal"
)

func main() {
    // PageAlignedAllocator is a memory.Allocator whose buffers ArrowMetal can borrow (see Copy rule).
    b := array.NewInt64Builder(am.NewPageAlignedAllocator())
    defer b.Release()
    b.AppendValues([]int64{5, 3, 9, 1}, []bool{true, true, false, true}) // 9 is null

    src := b.NewArray()
    defer src.Release()

    gpu, err := am.Import(src)             // arrow.Array -> GPU
    if err != nil {
        panic(err)
    }
    defer gpu.Release()

    sum, _ := gpu.Sum()                    // nulls skipped, like Arrow
    mask, _ := gpu.CompareScalar(am.Gt, int64(2))
    kept, _ := gpu.Filter(mask)
    out, _ := kept.Export()                // back to arrow-go, no copy
    defer out.Release()

    fmt.Println(sum, out)                  // 9 [5 3]
}

Run it with ARROWMETAL_LIB=/path/to/.build/release/libArrowMetalC.dylib go run ..

---

Sort options

Argsort, Sort, TopK and Lexsort keep the order they always had: nulls last and NaN after +Inf in both directions, -0.0 tied with +0.0. Argsort, TopK, Lexsort and their With forms return uint32 index arrays (row numbers up to 2^32 - 1; a longer input is an error), and Take accepts int32, int64 and uint32 indices. The With forms take a SortOptions per key:

type SortOptions struct {
    Descending bool
    Nulls      NullPlacement // NullsLast (the default) or NullsFirst, in either direction
    FloatOrder FloatOrder    // FloatIEEE (the default), FloatTotal or FloatNanLargest
}

func (a *Array) ArgsortWith(o SortOptions) (*Array, error)
func (a *Array) SortWith(o SortOptions) (*Array, error)
func (a *Array) TopK(k int64, largest bool) (*Array, error)
func (a *Array) TopKWith(k int64, o SortOptions) (*Array, error) // o.Descending: the k largest
func LexsortWith(columns []*Array, keys []SortOptions) (*Array, error)

FloatIEEE is Arrow C++'s order, which arrow-go's compute.SortIndices also uses: -0.0 ties +0.0, every NaN is one value, and the NaN rows sit next to the nulls in both directions. FloatTotal is IEEE 754 totalOrder, the order arrow-rs and Rust's total_cmp use: -NaN < -Inf < … < -0.0 < +0.0 < … < +Inf < +NaN, NaNs by payload, and a descending sort is its exact mirror. FloatNanLargest is Polars' and NumPy's order: -0.0 ties +0.0, and every NaN is one value greater than every number, +Inf included, in both directions (last ascending, first among the values descending); the null placement is independent of it. Integer, string and temporal keys ignore the float order. The zero SortOptions gives what Argsort(false) gives. TopKWith(k, o) answers with the first k indices ArgsortWith(o) gives. Neither option adds a pass to the GPU sort.

// src is [2, null, NaN, -0, 7]
plain, _ := gpu.Argsort(true)       // [4 0 3 2 1]: nulls and NaN stay last
opts := am.SortOptions{Descending: true, Nulls: am.NullsFirst, FloatOrder: am.FloatTotal}
total, _ := gpu.ArgsortWith(opts)   // [1 2 4 0 3]: the null, then +NaN > 7 > 2 > -0
top, _ := gpu.TopKWith(2, opts)     // [1 2]
polars, _ := gpu.ArgsortWith(am.SortOptions{Descending: true, FloatOrder: am.FloatNanLargest})
                                    // [2 4 0 3 1]: NaN > 7 > 2 > -0, the null last

(ExampleArray_ArgsortWith in example_test.go runs this.) A plan's sort key takes the same options as JSON: {"column": "x", "descending": true, "nulls": "first", "float_order": "total"} or ["x", true, {"nulls": "first", "float_order": "nan_largest"}] (Query engine).

Chunked import

A column held as several arrow.Arrays of one type imports as one Array of their total length, without array.Concatenate:

func ImportChunks(chunks []arrow.Array) (*Array, error)
func ImportChunked(c *arrow.Chunked) (*Array, error)
func ImportColumn(batches []arrow.RecordBatch, i int) (*Array, error)
func NewSourceFromBatches(name string, batches []arrow.RecordBatch) (*Source, error)
func ChunksSupported(dt arrow.DataType) (bool, error)

gpu, err := am.ImportChunked(chunked) // chunks [1 2] and [3 null 5]
sum, _ := gpu.Sum()                   // gpu.Len() 5, gpu.NullCount() 1, sum 11

(ExampleImportChunks runs this.) Every chunk crosses the C Data Interface on its own and am_import_chunks copies its buffers straight into the final Metal buffers, on the CPU cores in parallel. Each chunk's offset, length, validity bitmap (or its absence) and null count are honoured. One chunk is Import, with its copy rule. A type the chunked import does not take (dictionary, nested, run-end encoded, extension; ChunksSupported reports it) is concatenated with array.Concatenate and imported. The chunks stay the caller's: ImportChunks hands each one to C exactly once, pins its buffers for the call, and the library releases each export it took; the caller releases the chunks as usual. NewSourceFromBatches registers every column of a run of record batches this way, and the Source keeps those handles until its own Release.

---

Copy rule

The project's rule, unchanged: copy-free out always; copy-free in when the producer's buffers are page aligned, one copy otherwise.

Concretely, on the way in ArrowMetal wraps a buffer with MTLBuffer(bytesNoCopy:) when the buffer's start address is a multiple of the page size — 16384 bytes on Apple silicon, which arrowmetal.PageSize() reports — and copies the bytes into a page-aligned Metal buffer otherwise. On the way out the exported arrow.Array references the same unified memory the GPU wrote, always.

What Arrow Go's default allocator does — measured

memory.DefaultAllocator is memory.NewGoAllocator(), the Go heap, unless the program is built with the mallocator build tag — the same object, not a similar one, which is why only one of them appears below. The Go runtime backs large allocations with whole spans of its own 8 KiB pages, so an Arrow buffer from it is 8 KiB aligned and only incidentally 16 KiB aligned.

TestAllocatorAlignment builds an Int64 array 20 times per size and records the values buffer's offset past a page boundary. On an M4 Max, Go 1.27.1, arrow-go v18.7.0:

The hit rate is not a property, so it is not quoted here. Across fresh runs of the same test the count at 10M int64 came back 5/20, 15/20 and 14/20 — it depends on the state of the heap when the buffer is allocated, and it will differ on your machine and between two runs on mine. What is reproducible, and what the test asserts, is the offset: a Go-heap buffer is either exactly on a page boundary or exactly 8192 bytes past one. That follows from Go's 8 KiB page meeting Apple silicon's 16 KiB one, and it means a copy-free import with the default allocator is a coin flip whose outcome you cannot observe from Go.

arrowmetal.PageAlignedAllocator implements memory.Allocator on top of posix_memalign at page granularity, so the import can always borrow.

Go pointers, cgo, and why pinning is not optional

Alignment and legality are separate questions, and PageAlignedAllocator answers only the first.

cdata.ExportArrowArray writes &buf.Bytes()[0] straight into a malloc'd buffer array (cdata_exports.go:388). With the default allocator that address is in the Go heap, so this is an unpinned Go pointer stored into non-Go memory — which the cgo pointer rules forbid, and which GOEXPERIMENT=cgocheck2 turns into a hard fatal error: unpinned Go pointer stored into non-Go memory. This is apache/arrow-go#70, closed upstream without the default path being made legal. arrow-go's own answer is C memory: memory/mallocator (libc malloc, no build tag needed to use it, NewMallocatorWithAlignment(n) for a chosen power-of-two alignment) and, for programs linked against C++ Arrow, CgoArrowAllocator behind the ccalloc build tag. A mallocator buffer at alignment 16384 is page aligned and outside the Go heap, so it satisfies both the cgo rules and ArrowMetal's copy rule; PageAlignedAllocator is the same idea in forty lines, kept so the binding has no build-tag story to tell.

Import therefore pins every buffer it publishes with runtime.Pinner (Go 1.21+) before handing anything to C, and keeps the Pinner inside the returned Array until Release, unpinning only after am_release — ArrowMetal may have been borrowing those exact bytes. Pin ignores anything outside the Go heap, so a buffer from PageAlignedAllocator costs nothing here and needs nothing.

The consequence for you: any allocator is safe, including the default. The suite is run under GOEXPERIMENT=cgocheck2 as well as normally, and removing the pin makes it fail on the first round trip. Choose PageAlignedAllocator when you want the copy avoided, not to make the code legal.

What the copy costs, measured

Less than you would expect, because at 76 MB the copy is not the dominant cost of crossing the boundary; setting up the Metal buffer is. From the table below: importing 10M int64 rows takes 1.13 ms when the buffer can be borrowed and 1.41 ms when it must be copied — the copy adds about 0.3 ms to a 1.4 ms operation, roughly 20%. Export is 917 ns, three orders of magnitude cheaper, which is what "copy-free out always" buys.

There is no way to ask the library whether a given import copied. The Swift core computes it (ImportResult.zeroCopy) but the C ABI's am_import does not return it, so this binding cannot report it either and the table above measures alignment in Go instead.

---

Timing

10M Int64 rows (76 MB, i.e. MiB — 80 MB decimal), no nulls, one M4 Max, one process, wall clock, best of 5 timed runs after one untimed warm-up. Reproduce with go run ./cmd/amtiming from go/arrowmetal; the source of every row is cmd/amtiming/main.go.

ArrowMetal 0.1.0, Go 1.27.1, arrow-go v18.7.0. The filter predicate is x > 0 and keeps 50.0% of the rows.

Reading this table

arrow-go's compute is single-threaded and ArrowMetal's kernels run on the whole GPU, so the multiples above are not per-core figures.

• Arrow Go is ahead at Sum end to end. 2.22 ms against Arrow Go's 1.16 ms. A single 76 MB sum is a memory-bandwidth problem that the CPU is already good at, and the ~1.2 ms of import overhead is most of the ArrowMetal number. Only when the array is already on the GPU is ArrowMetal ahead on Sum, 296 µs against 1.16 ms. If your program's shape is "load an arrow-go array, sum it once, throw it away", this binding is the wrong tool.
• ArrowMetal is ahead at Filter, in every shape. Even paying import and export on every call, 3.22 ms against 48.93 ms is 15×; resident it is 32×. Filter does more work per byte than Sum and the fixed cost stops dominating.
• The end-to-end rows with a copying import came out faster than the page-aligned ones in this run (1.87 vs 2.22 ms for Sum, 2.98 vs 3.22 ms for Filter), which is the opposite of what the Import rows say. The gap is a few tenths of a millisecond either way, the same size as the copy itself, and it is run-to-run noise from Metal's buffer allocation. The defensible statement is the narrow one: at 76 MB, borrowing saves about 0.3 ms of a 1.4 ms import, and end to end that saving is inside the noise. It is not on its own a reason to change allocators — and it is not a correctness reason either, since Import pins whatever it is given.
• arrow/math.Int64.Sum ignores nulls, and the data here has none. It is the fastest thing arrow-go has for this and it is what the Sum row compares against, because arrow-go's compute package registers no aggregate function at all — no sum, mean or min_max in its registry. TestArrowGoHasNoAggregates records this and will log a nudge if a later release adds them. Reported as apache/arrow-go#1296; the maintainer asked for the kernels, and a design note with a tested prototype of sum, mean, min_max, min, max, count, any and all is on that issue (the research behind it is Round 10 in Findings against pyarrow). Two documentation pull requests from the same work, #1302 on allocator alignment and #1303 on what the compute registry holds, were merged the day they were opened.
• The plain Go filter loop writes into a []int64 rather than building an arrow.Array, so it is doing strictly less work than the other two Filter rows. It is included because it is what a Go programmer writes when they have not reached for Arrow yet.

Chunked import against concatenating first

go run ./cmd/amchunks: a column in chunks, each chunk its own allocation from Arrow Go's default allocator, imported with ImportChunks against array.Concatenate then Import, both end to end with the handle (and the concatenation) released inside the timed call. M4 Max, 2026-09-29, Go 1.27.1, arrow-go v18.7.0; three rounds, each row warmed for 100 ms and then timed 10 times; best of the three rounds, and the median of the per-round medians in parentheses; CPU is process CPU time per call. Source: Benchmarks/results/bindings_chunked_import_2026-09-29.csv.

The chunked import copies each chunk once, straight into the Metal buffers, on the CPU cores in parallel; the concatenation copies every byte once, and the import then borrows or copies the result by the copy rule above. The parallel copy takes 1.8x to 2.8x the CPU time of the concatenation per call (the last column).

The existing calls, before and after

go run ./cmd/ambench times the calls that existed before the sort options and the chunked import, built from the previous commit and from this one against the same libArrowMetalC.dylib, in four alternating rounds (each row warmed for 100 ms, a 500 ms idle and one call timed on its own, then 30 calls); best of the four rounds, the median of the per-round medians in parentheses. The Go code of these calls is unchanged; the shim resolves seven more entry points at load. At 10M rows:

At 1,000 and 1,000,000 rows three rows were slower in both best and median in the four rounds (Argsort(true) at 1,000 rows, Lexsort at 1,000 and 1,000,000 rows); timed again alone, eight alternating rounds of 100 calls, this commit over the previous one is 1.02 / 1.14 (best / median) for Argsort(true) at 1,000 rows (0.39 / 0.50 ms against 0.38 / 0.44 ms) and 0.97 / 1.04 and 1.02 / 1.02 for Lexsort at 1,000 and 1,000,000 rows. Every row, with first-call-after-idle and CPU time: Benchmarks/results/bindings_call_overhead_2026-09-29_summary.csv.

---

What is covered

include/arrowmetal.h and include/arrow_abi.h in the module are copies of the repository's include/ headers; refresh them (cp include/arrowmetal.h go/arrowmetal/include/) whenever the header changes, or python/tests/test_header_copies.py and TestHeadersMatchRepository fail.

Every item below has at least one test in go/arrowmetal; the oracle is named. 66 test functions and three Examples, 69 runnable; 584 cases counting subtests, all green, plain, under -race and under GOEXPERIMENT=cgocheck2.

Sizes are at or below 10M elements throughout.

---

What is not wrapped

The C ABI has 286 entry points; this binding resolves 40 of them (34 it needs, and 6 newer ones it uses when the loaded library has them: an older library still loads, and the calls that need them return an error naming the missing entry point). Not wrapped, and not tested from Go:

• Arithmetic and math: am_arith_scalar, am_arith_array, am_unary, am_binary, am_cumulative, am_window, the checked variants.
• Strings and text: am_str_unary, am_str_match, am_str_transform (all 26 ops), am_str_concat, am_str_dictionary_encode, the regex surface.
• Temporal: am_temporal_extract, am_temporal_cast_unit, am_round_temporal_ex, am_add_interval.
• Types: decimals, dictionaries beyond what import/export carries through, nested types (list, struct, map, union), run-end encoding, and the _ex option variants other than the sorts' (am_rank_ex, am_is_in_ex, …).
• Structural and conditional: am_is_null, am_fill_null, am_drop_null, am_if_else, am_coalesce, am_is_in, am_index_in, the Kleene operators.
• Aggregates beyond the four: am_reduce_ex (product, variance, stddev), and the grouped aggregates past sum/count/mean/min/max — the Agg method takes any op number from the header's table, but only those five are tested from Go.
• Batching: am_batch_begin / am_batch_end. A Go binding for these needs the batch and every call inside it pinned to one OS thread, which the current one-call-at-a-time pinning does not give you across calls.
• Whole subsystems: Parquet (am_parquet_*), streaming (am_stream_*), the device interface (am_import_device / am_export_device), joins (am_join).
• arrow.Table has no entry point of its own; its columns go in as arrow.Chunked (ImportChunked), and a run of arrow.RecordBatches as a Source (NewSourceFromBatches).

Adding any of these is mechanical: a prototype in amshim.h, a pointer and a forwarder in amshim.c, a method in Go, and a test with an oracle.

---

Limits

• macOS on Apple silicon only. The module compiles anywhere cgo does, but the dylib it needs exists only for Apple silicon, so on any other platform Init() returns the loader's error.
• cgo is required. CGO_ENABLED=0 builds will fail to compile the package.
• One goroutine at a time per handle. An Array, GroupBy, Source or PlanResult is not safe for concurrent use. Separate handles on separate goroutines are fine.
• Every call pins its goroutine to an OS thread for the duration. am_last_error() is thread-local, so the failing call and the message read have to happen on the same thread; Go is otherwise free to move a goroutine between calls. The cost is a runtime.LockOSThread pair per call, which is tens of nanoseconds against kernels measured in hundreds of microseconds.
• Release your handles. A finalizer is set as a backstop, but GPU memory should not wait for the garbage collector, and the finalizer runs at an unpredictable time.
• Go heap buffers stay pinned for the life of the handle — and only that long. Importing an array built with Arrow Go's default allocator hands C a pointer into the Go heap that ArrowMetal keeps until the handle is released, so Import pins those buffers and Release unpins them (see Go pointers, cgo, and why pinning is not optional). Pinned objects cannot be moved or freed, so a handle you forget to release holds its source buffers as well as its GPU memory. The converse is the open edge: an arrow.Array you exported from a borrowed Go-heap import still names those bytes after Release has unpinned them — they stay alive, through arrow-go's own reference chain, but they are no longer pinned, so the binding is back to relying on Go's collector not moving heap objects for as long as that exported array lives. Pinning does not close that window. Importing from PageAlignedAllocator does, since C memory is never the collector's to move.
• Scalar comparison covers the primitive types only, and refuses a scalar it cannot represent. CompareScalar accepts a Go value for int8…int64, uint8…uint64, float32, float64 and bool, range-checked against the column's element type. Refused, with an error naming the value and the Arrow type: an integer outside the column's width (int64(1000) against int8 is an error, not a silent −24); a finite float too large for the column (1e300 against float32, which would become +Inf); a non-zero float too small for it (5e-46 against float32, which would become 0 and match every zero in the column); and an integer past the mantissa width, 2^24 for float32 and 2^53 for float64, which would land on a neighbouring value. Ordinary rounding is not refused: 0.1 against a float32 column compares against float32(0.1), as it does in every Arrow implementation. A string, decimal or temporal scalar returns an error naming the Arrow format string rather than guessing.
• The vendored headers can drift. go/arrowmetal/include/arrowmetal.h and arrow_abi.h are copies of the repository's include/, because a Go module can only see files inside its own directory. TestHeadersMatchRepository compares them when it is run inside a checkout and skips otherwise.

---

Testing

See Testing. In short, from go/arrowmetal:

ARROWMETAL_LIB=$PWD/../../.build/release/libArrowMetalC.dylib go test ./...

# and once more with the cgo pointer checker armed, which is where an unpinned
# Go pointer handed to C shows up as a hard failure rather than as luck:
ARROWMETAL_LIB=$PWD/../../.build/release/libArrowMetalC.dylib \
    GOEXPERIMENT=cgocheck2 go test -count=1 ./...

The library path is optional inside the repository — ../../.build/release is on the search list — but it is the reliable way to say which build you mean.