Show HN: Open-source edge data governance library
csvpath.orgCsvPath Framework is a Python data firewall that protects downstream systems from bad data getting past the ingress point. Data is much easier to fix when you shift quality control left, remove manual data checking, stop accumulating orphaned ingestion code, and decrease the skills consumed by data pipelines.
The Framework:
- registers inbound data into an immutable, versioned staging area, with a cryptographic identity,
- validates and upgrades it using one or more of five validation languages,
- generates ops and lineage metadata for OpenLineage and OpenTelemetry,
- and publishes results data and metadata to a versioned immutable archive with no-code integrations downstream.
CsvPath Framework refers to these bullets as "data preboarding".
Supports: JSON Schema, ODCS, XSD, XPath, and CsvPath Validation Language (CVL) validation. Ingests: JSON, JSONL, XML, CSV, Excel, Pandas DataFrames, live Python objects. Backends: AWS, Azure, GCP, SFTP, local; also FlightPath Server
Website: https://www.csvpath.org
No comments yet.