GitHub - julik/herringbone: A (slightly unhinged) pure-Ruby Parquet file reader/writer

GitHub

5 min read Original article ↗

A pure-Ruby reader and writer for Apache Parquet files.

  • No Thrift gem and no native extensions of its own: the Thrift compact protocol, all encodings and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems). ZSTD and LZO (from Hadoop-era files) are also read in pure Ruby; writing ZSTD needs zstd-ruby
  • Full nesting support (structs, lists, maps, any depth) via Dremel record shredding/assembly
  • Reads files from parquet-mr, Arrow, Spark, Impala, DuckDB, Rust writers etc.
  • Optimized reads with batches and pages
  • Parquet modular encryption (per column and of the footer), interoperable with parquet-mr and Arrow
  • Ruby 3.0+

This README covers the common cases. The MANUAL has everything else: reader and writer options, schemas and column types, bloom filters, Numo arrays, encryption, the redaction rules, combining files, type mapping and the inspector.

Installation

Optionally add snappy (faster), zstd-ruby (writing ZSTD, much faster reading of it), brotli, xxhash (faster bloom filters) or numo-narray-alt (read(as: :numo)). Herringbone uses whichever of them are loaded, see Optional gems.

Exporting from Rails

Stream a relation straight into S3, no temp file needed:

s3 = Aws::S3::TransferManager.new
s3.upload_stream(bucket: "exports", key: "payments.parquet") do |io|
  # Schema will be auto-inferred, find_each will be used automatically
  Herringbone.write(io, Payment.where(status: "settled", created_at: 1.month.ago..))
end

Or to a local file:

File.open("orders.parquet", "wb") do |file|
  Herringbone.write(file, Order.where(created_at: 1.year.ago..), compression: :zstd)
end

Any Enumerable of Hashes, Structs or Arrays works too; the schema is inferred from the first 1000 rows. See ActiveRecord and Writing to S3. For Google Cloud Storage, use gcs_put, see Writing to Google Cloud Storage.

Reading

File.open("data.parquet", "rb") do |file|
  reader = Herringbone::Reader.new(file)
  reader.num_rows

  reader.each_row { |row| p row }        # Hashes with String keys, nested values as Hash/Array
  reader.each_batch(1000) { |rows| ... } # Arrays of up to 1000 rows
  reader.read                            # all rows at once

  reader.read(columns: %w[id name], where: { user_id: 42 })
  reader.read(where: { status: %w[paid shipped], created_at: 1.week.ago.. })
end

Reads stream page by page, and where: skips row groups and pages it can rule out using statistics, page indexes and bloom filters. See Reading.

Writing CSV-style

Name the columns with headers: and append the rows as Arrays, like with the CSV gem. The types are inferred.

File.open("people.parquet", "wb") do |file|
  Herringbone::Writer.open(file, headers: [:id, :name, :age]) do |w|
    w << [123, "John", 12]
    w << { id: 124, name: "Jane" } # Hashes work too
  end
end

Rows you already have in an Array or an Enumerable go in one call: Herringbone.write(file, rows, headers: [:id, :name, :age]).

To control the types, declare a schema and use Herringbone::Writer, see Writing.

Encrypting

Parquet files tend to wander off: to S3 buckets, laptops, other teams. Encrypting them takes one key and one line, no KMS, no Hadoop configuration. Make a key once and keep it with your other secrets:

Herringbone::Key.generate.hex # => "9f86d081884c7d65..." - put it in your credentials or ENV

Then write with it:

File.open("people.parquet", "wb") do |file|
  Herringbone::Writer.open(file, headers: [:id, :name, :email], encryption: ENV["PARQUET_KEY"]) do |w|
    w << [1, "John", "john@example.com"]
  end
end

and read with it:

Herringbone::Reader.new(file, decryption: ENV["PARQUET_KEY"]).read

The schema, the values and the statistics are all encrypted (AES-256-GCM); without the key the file is just noise. encryption: key does the same for Herringbone.write and Herringbone::Writer. Your colleagues can open the file with the same key in pyarrow 25+ (pq.read_table(path, decryption_properties=pyarrow.parquet.encryption.create_decryption_properties(bytes.fromhex(key)))), Arrow, DataFusion, Trino or Spark, and herringbone inspect people.parquet asks for the key. When the time comes to rotate, read with all your keys, decryption: [new_key, old_key], and each file finds its own. See Encryption for per-column keys, plaintext footers and which tools read what.

Redacting

Herringbone.redact rewrites a file with rows removed or values replaced, for GDPR "forget me" requests and pseudonymization. The parts of the file nothing touches are copied byte for byte. It reads from a seekable IO or from a Herringbone::Reader, which brings its own decryption, and writes to any IO with #write.

# Forget me: remove the rows
Herringbone.redact(io, output_io) do |r|
  r.where(user_id: 42).delete
end

# Forget me, but keep the row for accounting: blank the personal columns
Herringbone.redact(io, output_io) do |r|
  r.where(user_id: 42).replace(email: nil, name: nil, address: nil)
end

# Pseudonymize a column across the whole file, mask another, drop a third
Herringbone.redact(io, output_io) do |r|
  r.replace(:email) { |email| OpenSSL::HMAC.hexdigest("SHA256", KEY, email.downcase) }
  r.replace(:phone) { |phone| phone && "***#{phone[-3..]}" }
  r.drop :ssn, :ip_address
end

See Redaction for reusable redactions, reports and recipes.

Combining files

Herringbone.combine concatenates files with the same schema by copying their row groups byte for byte, without decoding anything:

Herringbone.combine([january, february, march], output_io)

Files with different columns combine with schema: :union, which fills the gaps with nulls and widens types (int32 and int64 to int64), or schema: :intersect, which keeps the fields all of them have. See Combining files.

Looking inside a file

bin/herringbone cat FILE [N]                 # rows as JSON lines
bin/herringbone inspect FILE                 # schema, row groups, column chunks
bin/herringbone inspect FILE --format=html   # a byte map of the file, opened in your browser

See Inspecting files and Command line.

Development

See Development and CONTRIBUTING.