Testing DuckDB 2.0 on our Iceberg datasets — LakehouseBox

We are excited about DuckDB 2.0, scheduled for the second half of October. The 2.0 preview brings major engine changes—most notably asynchronous I/O across the execution pipeline, far more parallelized remote reads, and partition-aware planning for lakehouse formats. That is a big deal for us because DuckDB is how our users' agents query their data: each agent runs it on its own machine against Apache Iceberg tables in our object storage, through lhbox duckdb, and our own services run it next to the data.

Over the last few days we have been preparing our stack for 2.0 and running performance and correctness checks to catch issues early. Because the preview announcement does not include numbers for remote Iceberg reads, we are sharing our measurements here so others testing 2.0 can compare notes.

Update, 22 October: DuckDB 2.0.0 is out, and the spatial fix we describe below shipped with it (duckdb#27249). We re-ran the benchmarks on the released 2.0.0, and the results match the figures below. The Paris rail query now reads 11.5 MiB in 12 GETs, against 9.4 MiB on DuckDB 1.5.6 and 268 MiB on the preview without the fix. LakehouseBox's server-side queries run on 2.0.0 by default since 22 October.

Builds: the figures below were first measured on the preview 2.0.0.dev2610011535 (alpha v2.0.0-alpha43763, Iceberg extension 5b9ff899a1) on 14 October 2026 and confirmed on 2.0.0, compared against DuckDB 1.5.6.

Where we ran the tests

Every test queried the same Apache Iceberg tables stored in Nuremberg, Germany, from two different machines and network paths:

Cold and warm TPC-H reads

In the same data center, every cold query was faster on 2.0, with per-query speedups ranging from 1.05× (q21) to 3.78× (q11)—a 1.93× geometric mean across the 22 queries, and a 1.69× reduction in total suite wall time (105.9 s down to 62.7 s). Cold Iceberg queries spend significant time waiting on sequential round-trips: catalog metadata, manifest list, manifest files, Parquet footers, and column chunks. DuckDB 2.0 reads the same total byte volume (~12.1 GB vs. ~12.3 GB) across 52 % more HTTP GET requests (9,346 vs. 6,126), keeping smaller range requests in flight concurrently via async I/O.

From Madrid, the bottleneck shifts from request latency to link bandwidth. A cold pass transfers ~12 GB, and 1.5.6 already averages 45 MB/s over our residential connection. Selective queries still speed up (q02 is 1.8× faster, q16 is 2.0× faster), large scans break even, and a few queries transfer noticeably more data in 2.0 (q15 pulls 381 MB in 2.0 vs. 132 MB in 1.5.6; q14 pulls 461 MB vs. 292 MB), likely from prefetching ranges the scan ultimately does not need. Over a local data-center link that extra transfer is free; over a home connection it dominates query time.

For warm queries (repeating the query on the same connection), 1.5.6 performs zero network reads across all 22 queries from both locations. The 2.0 preview re-fetches parts of the data on repeat runs (q15 transfers 69 MB on the first repeat and 21 MB on the second). In Nuremberg that extra 30 MB across the suite is invisible; from Madrid it totals 170 MB and makes the warm pass 46 % slower (19.8 s vs. 13.6 s). Separately, the two heaviest joins (q09 and q21) were slower warm on 2.0 even with zero network traffic (q09 went from 3.9 s to 6.3 s on the 2-vCPU server and from 1.3 s to 2.0 s on the M1 laptop).

Iceberg writes: 409 files down to 4

We also timed loading the SF10 dataset into Iceberg with each version (CREATE TABLE … AS SELECT, with lineitem ordered by l_shipdate):

DuckDB 1.5.6 flushes small ~5 MB Parquet files during Iceberg writes, which previously required background compaction (merging lineitem from 409 files down to 17). DuckDB 2.0 writes target-sized ~536 MB files out of the box, finishes the 8-table load 22 % faster, and records total-files-size in the Iceberg snapshot summary.

File count also explains part of the read speedup. When we ran the cold TPC-H suite in Nuremberg against the uncompacted 4-file lineitem layout written by 2.0, DuckDB 1.5.6 sped up by 18 % (105.9 s → 86.4 s) while 2.0 stayed flat (62.7 s → 65.2 s), narrowing 2.0's per-query geomean lead from 1.93× to 1.42×. Much of 2.0's read advantage comes from amortizing the overhead of many small files; writing larger Parquet files helps every reader.

Real queries on public datasets

Interactive sessions and AI agents rarely run 22-query TPC-H sweeps; they typically start a fresh DuckDB process, ATTACH a catalog, and run one selective query. We timed that pattern against our public Iceberg catalog (commons/pubdata), signed in and reading straight from our storage in Nuremberg, running five fresh processes per query and engine. (Readers without an account get these tables through our CDN instead, so their timings will differ.)

For selective queries, 2.0 returns a cold answer 1.4× to 2.9× faster from Madrid, and catalog ATTACH itself drops from 0.28 s to 0.19 s. The Overture pharmacy lookup gains the most (9.39 s → 3.28 s) because it issues 1,146 small HTTP reads that 2.0 overlaps.

The point-in-polygon scan over admin_areas illustrates the tradeoff of splitting large reads over higher latency. The query scans 460 MB across a few large files: 1.5.6 fetches them in 10 HTTP GETs, whereas 2.0 splits the read into 226 smaller GETs. In Nuremberg 2.0 is 1.15× faster (2.26 s → 1.97 s); from Madrid those 226 requests pay the extra round-trip cost and 2.0 is 17 % slower.

One spatial regression

One row in the chart above matters to us more than its size suggests: the Paris rail bounding-box query (geometry && box). Our spatial tables are sorted along a Hilbert curve so that a filter like this skips most files, and on DuckDB 1.5.6 it reads 9.9 MB. The 2.0 preview read the whole 281 MB table. Tracking it down took two steps. The first was in the Iceberg extension: we reported it as duckdb-iceberg#1464 on 30 September and the fix (#1467) was merged the same day. The scan still read the whole table, and the second cause turned out to be in DuckDB core: a cast the optimizer wrongly treated as one that could fail. We filed it as duckdb#27247 on 14 October and proposed a small fix in duckdb#27249 the same afternoon.

Update, 18 October: the DuckDB team merged #27249 into the 2.0 release branch, so the fix is set to ship with 2.0.0. Four days from report to merge is one reason we are comfortable building on DuckDB. We have not re-measured yet: the timings above are from the preview build without the fix, and we will update this row once a build that includes it is available. Until then, filtering on the plain bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax columns prunes on both versions (under 1 MB read).

Update, 20 October: the first preview builds with the fix are out, and we measured the newest, 2.0.0.dev2610192346. The Paris rail query in the snippet below now reads 11.5 MiB in 12 GETs, against 9.4 MiB in 8 on DuckDB 1.5.6 and 268 MiB in 189 on the preview without the fix, with the same answer. Our Overture box questions (places in central Madrid, buildings around Puerta del Sol, roads in central Valencia) read 8.5, 6.1 and 9.3 MB, against 2.5, 6.1 and 9.3 MB on 1.5.6 and 1.1 to 4.0 GB on the earlier preview, again with the same answers. Spatial pruning is back. The chart still shows the earlier preview's timings; the full benchmark re-run waits for 2.0.0. Our opt-in 2.0 engine now runs this build.

INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs; INSTALL spatial; LOAD spatial;
ATTACH 'commons/pubdata' AS pubdata (TYPE iceberg, ENDPOINT 'https://public-catalog.lakehousebox.com',
                                     AUTHORIZATION_TYPE 'none');
EXPLAIN ANALYZE
SELECT class, round(sum(length_m) / 1000, 1) AS km
FROM pubdata.transport.rail_segments
WHERE geometry && ST_MakeEnvelope(2.20, 48.80, 2.45, 48.92)
  AND ST_Intersects(geometry, ST_MakeEnvelope(2.20, 48.80, 2.45, 48.92))
GROUP BY 1 ORDER BY 2 DESC;

What we are doing next

In the same data center, DuckDB 2.0's async I/O cuts cold-query wall time on Iceberg tables by 41 % (1.69× faster in total time, 1.93× per-query geometric mean) and writes 100× fewer Parquet files out of the box. Over a residential internet connection, selective single-table queries also see a clear latency drop, while large scans remain bound by link bandwidth.

DuckDB 2.0.0 passed our checks, spatial pruning included, and is the default engine for server-side queries since 22 October; lhbox query --engine 1.5 still runs 1.5.6 for anyone who needs it while they move.

Explore how it works