GitHub - terrafloww/rasteret: Index-first, fast GeoTIFF access layer for ML and analysis. Parse headers once, cache in Parquet, read pixels 20x faster.

Made to beat cold starts.
Index-first access to cloud-native GeoTIFF collections for ML and analysis.

Every cold start re-parses satellite image metadata over HTTP - per scene, per band. Sentinel-2, Landsat, NAIP, every time. Your colleague did it last Tuesday, CI did it overnight, PyTorch respawns DataLoader workers every epoch. A single project repeats millions of redundant requests before a pixel moves.

Rasteret parses those headers once, caches them in Parquet, and its own reader fetches pixels concurrently with no GDAL in the path. Up to 20x faster on cold starts.

Rasteret separates the workflow into two parts:

• Control plane: Parquet metadata, cached COG headers, and user columns like labels or splits
• Data plane: on-demand byte-range reads from the original GeoTIFF/COG objects

Key Features -

• Easy - three lines from STAC search or Parquet file to a TorchGeo-compatible dataset
• 20x faster, saves cloud LISTs and GETs - Our custom I/O reads tiles fast with zero STAC/header overhead once a Collection is built
• Zero data downloads - work with terabytes of imagery while storing only megabytes of metadata.
• No STAC at training time - query once at setup; zero API calls during training with Collection you can extend.
• Reproducible - same Parquet index = same records = same results
• Native dtypes - integer imagery stays integer; missing/edge coverage is represented via fill values (nodata or 0) instead of NaNs
• Shareable cache - enrich our Collection with your ML splits, patch geometries, custom data points for ML, and share it, don't write folders of image chips!

Rasteret is an opt-in accelerator that integrates with TorchGeo by returning a standard GeoDataset. Your samplers, DataLoader, xarray workflows, and analysis tools stay the same - Rasteret handles the async tile I/O underneath.

Installation

Requires Python 3.12+.

uv pip install "rasteret[xarray]"       # + xarray output
uv pip install "rasteret[torchgeo]"     # + TorchGeo for ML pipelines
uv pip install "rasteret[aws]"          # + requester-pays buckets (Landsat, NAIP)
uv pip install "rasteret[azure]"        # + Planetary Computer signed URLs

Built-in datasets

Rasteret ships with a growing catalog of datasets. Each entry includes license metadata and a commercial_use flag for quick filtering.

Pick an ID, pass it to build() and go:

$ rasteret datasets list
ID                          Name                                       Coverage       License              Auth
aef/v1-annual               AlphaEarth Foundation Embeddings (Annual)  global         CC-BY-4.0            none
earthsearch/sentinel-2-l2a  Sentinel-2 Level-2A                        global         proprietary(free)    none
earthsearch/landsat-c2-l2   Landsat Collection 2 Level-2               global         proprietary(free)    required
earthsearch/naip            NAIP                                       north-america  proprietary(free)    required
earthsearch/cop-dem-glo-30  Copernicus DEM 30m                         global         proprietary(free)    none
earthsearch/cop-dem-glo-90  Copernicus DEM 90m                         global         proprietary(free)    none
pc/sentinel-2-l2a           Sentinel-2 Level-2A (Planetary Computer)   global         proprietary(free)    required
pc/io-lulc-annual-v02       ESRI 10m Land Use/Land Cover               global         CC-BY-4.0            required
pc/alos-dem                 ALOS World 3D 30m DEM                      global         proprietary(free)    required
pc/nasadem                  NASADEM                                    global         proprietary(free)    required
pc/esa-worldcover           ESA WorldCover                             global         CC-BY-4.0            required
pc/usda-cdl                 USDA Cropland Data Layer                   conus          proprietary(free)    required

Use your own datasets

• Use build_from_stac() for any STAC API
• Use build_from_table() for Parquet files that already contain GeoTIFF/COG URLs

You can also build collections using CLI rasteret collections build read more details here

Here's a guide to add a dataset to rasteret's catalog so everyone benefits. The catalog is open to edit by anyone and will be community-driven.

Each new dataset entry is around ~20 lines of Python pointing to a STAC source or a Parquet source. One PR adds a dataset, every rasteret user sees it in rasteret datasets list on the next release of rasteret.

Quick start

Build a Collection

import rasteret

collection = rasteret.build(
    "earthsearch/sentinel-2-l2a",
    name="s2_training",
    bbox=(77.5, 12.9, 77.7, 13.1),
    date_range=("2024-01-01", "2024-06-30"),
)

build() picks the dataset from the catalog, parses COG headers, and caches everything as Parquet. The next run loads in milliseconds.

Inspect and filter

collection        # Collection('s2_training', source='sentinel-2-l2a', bands=13, records=42, crs=32643)
collection.bands  # ['B01', 'B02', ..., 'B12', 'SCL']
len(collection)   # 42


# Filter in memory, no network calls
filtered = collection.subset(cloud_cover_lt=15, date_range=("2024-03-01", "2024-06-01"))

subset() accepts cloud_cover_lt, date_range, bbox, geometries, split, and split_column (when your split field uses a custom name). Use collection.where(expr) when you need an Arrow predicate on custom metadata columns.

ML training (TorchGeo)

from torch.utils.data import DataLoader
from torchgeo.samplers import RandomGeoSampler
from torchgeo.datasets.utils import stack_samples

dataset = collection.to_torchgeo_dataset(
    bands=["B04", "B03", "B02", "B08"],
    chip_size=256,
)

sampler = RandomGeoSampler(dataset, size=256, length=100)
loader = DataLoader(dataset, sampler=sampler, batch_size=4, collate_fn=stack_samples)

Analysis (xarray)

ds = collection.get_xarray(
    geometries=(77.55, 13.01, 77.58, 13.08),  # bbox, Arrow array, Shapely, or WKB
    bands=["B04", "B08"],
)
ndvi = (ds.B08 - ds.B04) / (ds.B08 + ds.B04)

Fast arrays (NumPy)

arr = collection.get_numpy(
    geometries=(77.55, 13.01, 77.58, 13.08),
    bands=["B04", "B08"],
)
# shape: [N, C, H, W] for multi-band, [N, H, W] for single-band

Benchmarks

Cold-start comparison with TorchGeo

Same AOIs, same scenes, same sampler, same DataLoader. Both paths output identical [batch, T, C, H, W] tensors. TorchGeo runs with its recommended GDAL settings for best-case remote COG performance.

The difference comes from how headers are accessed: the rasterio/GDAL path re-parses IFDs over HTTP on each cold start, while Rasteret reads them from a local Parquet cache. See Benchmarks for full methodology.

HuggingFace Major-TOMCore 'images-in-parquet' vs Rasteret

Baseline method: datasets.load_dataset(..., streaming=True, filters=...) with local GeoTIFF decode, compared against Rasteret prebuilt index reads. Reproduce with examples/major_tom_benchmark/03_hf_vs_rasteret_benchmark.py.

For exploration workflows, Major TOM notebooks often use HF streaming generators; Rasteret is optimized for reading the same patches directly from source COGs using an index-first cache.

Notebook: 05_torchgeo_comparison.ipynb

Scope and stability

Rasteret is optimized for remote, tiled GeoTIFFs (COGs). It also works with local tiled GeoTIFFs for indexing, filtering, and sharing collections. Non-tiled TIFFs and non-TIFF formats are best handled by TorchGeo or rasterio.

Documentation

Full docs at terrafloww.github.io/rasteret:

Contributing

The catalog grows with community help:

• Add a dataset: write a ~20 line descriptor in catalog.py, open a PR. See prerequisites and guide
• Improve docs: fix a typo, add an example, clarify a section
• Build something new: ingest drivers, cloud backends, readers. See Architecture

All contributions are welcome. See Contributing for dev setup and we are happy to discuss all aspects of library. Ideas welcome on GitHub Discussions or join our Discord to just chat.

Technical notes

License

Code: Apache-2.0