Apache Parquet for .NET

Hello! ·

35 min read Original article ↗

ℹ️ Important

This project is retired on Nov 10 2026, which conicides with .NET 11 release date. Documentation below is kept for historical reasons only. If you need the source code, feel free to download it.

Why am I killing it?

  • I personally believe .NET is the wrong ecosystem for handling Apache Parquet files. There are much better alternatives like Apache Spark, DuckDB, Apache Arrow, etc. While this hobby lasted for quite some time, practically in 2026 it makes no sense whatsoever. I’d personally advise to move to other more suitable platforms for data processing rather than “switching” to other .NET libraries, but that’s just me.
  • I personally don’t have time to support this project, nor any desire to transfer ownership, mostly due to the current trend in software development and having zero trust that a potential future owner won’t screw it up. Unfortunately the last year was spent mostly rejecting LLM-generated slop making the precious rare hobby time unpleasant, rather than something I was looking forward to.
  • I haven’t used this project in any of my code for some time now. It was used in the wild extensively in the past in some very critical scenarios, therefore the library is considered more or less stable.
  • I no longer agree with GitHub’s terms & conditions or its roadmap. When I joined, GitHub was awesome, but today I see it as a place where everyone throws their LLM-generated slop, or a platform for training LLMs. Unlike popular opinions, I am not anti-“AI” or any kind of nonsense like that, just to be clear. Therefore it would be morally wrong to continue working on that platform. Changing hosting providers could work if I had the energy to do so.
  • General disappointment in .NET 11 and the current state of .NET probably added quite a bit to it. I can go on about this indefinitely, and it was really hard for me to give up this, in my opinion, great platform. But, in my personal opinion, it never had or will have a future I am looking for.
  • If you were relying on Parquet.Net in your hobby projects, don’t worry: the NuGet package is not going anywhere, so you have plenty of time to migrate to something better. If you were a company relying on this library, that was a huge mistake. My personal GitHub account was sponsored at $10/month minus taxes, and most likely not because of this project, so I know you were not serious.

In general, I really enjoyed working on this project in the past, it taught me a lot about .NET internals, performance optimisations, trade-offs between performance and usability, backward compatibility and on and on and on. I have also created a companion library that reads and writes Delta Lake tables, implementing locking, transactions, schema evolution, partitioning, z-ordering and so on (hence some visible slowdown in contributing to the Parquet.Net in the recent years). Unfortunately, for the similar reasons, the library has never left the private Gitea instance before I could kill it, but was used in production for a couple of years with mostly success.

Icon

Fully managed, safe, extremely fast, open-source .NET library to 📖read and ✍️write Apache Parquet files designed for .NET world (not a wrapper). Targets only modern .NET runtimes such as .NET 10 and .NET 8.

Whether you want to build apps for Linux, MacOS, Windows, iOS, Android, Tizen, Xbox, PS4, Raspberry Pi, Samsung TVs or much more, Parquet.Net has you covered.

Features at a glance

  • 0️⃣ Has zero dependencies - pure library that just works anywhere .NET works i.e. desktops, servers, phones, watches and so on.
  • 🚀Really fast. Faster than Python and Java, and alternative C# implementations out there. It’s even faster than native C++ implementations. Hardware acceleration is added in more and more places.
  • 🏠NET native. Designed to utilise .NET and made for .NET developers, not the other way around.
  • ❤️‍🩹Not a “wrapper” that forces you to fit in. It’s the other way around — forces Parquet to fit into .NET.
  • 🦄Unique Features:
    • The only library that supports dynamic schemas.
    • Supports all parquet types, encodings and compressions.
    • Fully supports C# class serialization, for all simple and complex Parquet types.
    • Provides low-level, high-level, and untyped API.
    • Access to file and column metadata.
    • Fine-tune encodings per column.
    • Integration with DataFrames (Microsoft.Data.Analysis).

Quick start

Parquet is designed to handle complex data in bulk. It’s column-oriented meaning that data is physically stored in columns rather than rows. This is very important for big data systems if you want to process only a subset of columns - reading just the right columns is extremely efficient.

Suppose we have the following data we’d like to save to a parquet file:

Timestamp (DateTime)Event name (string)Meter value (double)
2025-11-18 22:07:00start100
2025-11-18 22:08:00stop500
2025-11-18 22:09:00pause200

High level API

Let’s face it, we are all busy, so high level API that don’t need you to know Parquet internals too much is a very good option. High level APIs tries to mimic class serialization, just like you do when working with JSON documents, with a few caveats.

Sample above can be rewritten by first declaring data type to hold all 3 schema fields:

1class Event {
2    public DateTime Timestamp { get; set; }
3    public string EventName { get; set; }
4    public double MeterValue { get; set; }
5}

Now let’s generate some fake data:

1List<Event> data = Enumerable.Range(0, 1_000_000).Select(i => new Event {
2    Timestamp = DateTime.UtcNow.AddSeconds(i),
3    EventName = i % 2 == 0 ? "on" : "off",
4    MeterValue = i 
5}).ToList();

And write these to a file:

1await ParquetSerializer.SerializeAsync(data, "/mnt/storage/data.parquet");

To read, simply call:

1DeserializationResult<Event> data = await ParquetSerializer.DeserializeAsync<Event>("/mnt/storage/data.parquet");

Class serialization is really fast as it generates compiled expression trees on the fly. That means there is a small delay when serializing the first entity, which in most cases is negligible. Once the class is serialized at least once, further operations become much faster (around ~40x compared to reflection on large amounts of data (~5 million records)).

💡 Tip

Class serialization philosophy is based on the idea that we don’t need to reinvent the wheel when it comes to converting objects to and from JSON. Instead of creating our own custom serializers and deserializers, we can leverage the existing JSON infrastructure that .NET provides. This way, we can save time and effort, and also make our code more consistent and compatible with other .NET applications that use JSON.

Note that classes (or structs) in general purpose programming languages represent rows, but parquet is columnar. Therefore, there are natural limitations to what data structures are supported in parquet serialization:

  • In order for the deserializer to work, classes need to have a parameter less constructor.
  • Both properties and fields are supported, and naturally when serializing those need to be readable, and when deserializing they need to be writeable. This might limit your use cases if you are trying to deserialize into immutable objects, and in this case you should probably keep DTOs specifically designed for parquet format, which is still easier than using low level API.
  • The deserializer does not “overwrite” class members; i.e. if you are deserializing into a list property and the default constructor already initializes the list with some values, the Parquet deserializer will append data to the list instead of overwriting it.
  • While you can serialize struct, deserialization is only supported to class due to internal optimisation requirements.

Low level API

Choose low-level API for extreme performance and full control. Let’s rewrite the high-level sample with low-level API. Because parquet is a columnar format, you’d need to prepare 3 chunks of data, where each chunk is an array of values per column (first chunk is[2025-11-18 22:07:00, 2025-11-18 22:08:00, 2025-11-18 22:09:00] and so on). And because parquet is strongly typed, you also need to prepare a schema definition, declaring what type of data the file contains. Schema in Parquet.Net is represented as an instance of ParquetSchema class, which contains a collection of DataFields (columns):

1var schema = new ParquetSchema(
2    new DataField<DateTime>("Timestamp"),
3    new DataField<string>("EventName"),
4    new DataField<double>("MeterValue"));

The next step is to create ParquetWriter class, which builds parquet file skeleton inside the passed stream, and allows you to create a ParquetRowGroupWriter that can write column data to the file. Row group in parquet is a group of all columns from the schema. A file can have any number of row groups, but there must be at least one present. If file is small enough (less than 64Mb or so) it will usually have a single row group. Row groups allow parquet files to contain massive amounts of data and also enable read parallelism (but not write parallelism), Each row group contains all the columns from the schema, but different “rows” or data. This is how you create a simple file with a single row group and write all 3 columns to it:

 1using Stream fs = System.IO.File.OpenWrite("/mnt/storage/data.parquet"); 
 2await using ParquetWriter writer = await ParquetWriter.CreateAsync(schema, fs);
 3using ParquetRowGroupWriter groupWriter = writer.CreateRowGroup();
 4
 5await groupWriter.WriteAsync<DateTime>(schema.DataFields[0],
 6    new[] {
 7        new DateTime(2025, 11, 18, 22, 07, 00),
 8        new DateTime(2025, 11, 18, 22, 08, 00),
 9        new DateTime(2025, 11, 18, 22, 09, 00)});
10
11await groupWriter.WriteAsync(schema.DataFields[1],
12    new[] { "start", "stop", "pause" });
13
14await groupWriter.WriteAsync<double>(schema.DataFields[2],
15    new[] { 12.34, 56.78, 90.12 });

Notice that ParquetWriter implements IAsyncDisposable, hence await using. Column data is written directly by calling WriteAsync on the row group writer, passing the schema field and the values array. For strings and byte arrays, there are convenient overloads that accept IReadOnlyCollection<string?> and IReadOnlyCollection<byte[]?> respectively.

To read this file back (or just any file created with this or any other parquet software) you can do pretty much the reverse action:

 1using Stream fs = System.IO.File.OpenRead("/mnt/storage/data.parquet");
 2await using(ParquetReader reader = await ParquetReader.CreateAsync(fs));
 3// optionally access schema: reader.Schema
 4for(int i = 0; i < reader.RowGroupCount; i++) { 
 5    using ParquetRowGroupReader rowGroupReader = reader.OpenRowGroupReader(i);
 6    
 7    DataField[] dataFields = reader.Schema.GetDataFields();
 8
 9    // read non-nullable columns by pre-allocating a buffer and calling ReadAsync
10    DateTime[] timestamps = new DateTime[rowGroupReader.RowCount];
11    await rowGroupReader.ReadAsync<DateTime>(dataFields[0], timestamps);
12
13    // strings have a convenience overload that accepts string[]
14    string[] eventNames = new string[rowGroupReader.RowCount];
15    await rowGroupReader.ReadAsync(dataFields[1], eventNames);
16
17    double[] meterValues = new double[rowGroupReader.RowCount];
18    await rowGroupReader.ReadAsync<double>(dataFields[2], meterValues);
19
20    // do something with the data...
21}

Just like with writing, you create ParquetReader on the source stream (also using await using). Upon creating the reader, Parquet.Net reads stream metadata from the end of the file (hence the requirement for the source stream to have random access) and initializes the internal structures without reading any data. You can even access schema immediately by calling to reader.Schema to inspect it. Then enumerate row groups and read columns one by one by calling ReadAsync<T> with a pre-allocated buffer. For nullable columns, you can use ReadAsync<T> with a Memory<T?> buffer instead.

Customising serialization

Serialization tries to fit into the C# ecosystem like a ninja 🥷, including customizations. It supports the following attributes from System.Text.Json.Serialization Namespace:

  • JsonPropertyName - changes mapping of column name to property name. See also ignoring property casing.
  • JsonIgnore - ignores property when reading or writing. Alternatively, there is [ParquetIgnore].
  • JsonPropertyOrder - allows to reorder columns when writing to file (by default they are written in class definition order). Only root properties and struct (classes) properties can be ordered (it won’t make sense to do the others).

Where built-in JSON attributes are not sufficient, extra attributes are added.

  • Enum is supported by treating it as underlying type (by default it’s int). There might be some improvements in future versions, such as mapping them to corresponding string values.
  • string in .NET is a reference type, which means it can be null. However, Parquet specification allows types to be declared required or optional. To fit into .NET ecosystem as closely as possible, this library will serialize .NET strings as optional by default. If you want to change this behaviour, annotate string member with [ParquetRequired] attribute. Parquet.Net will also expect string to be optional when deserializing from a file and you will get an exception until you add [ParquetRequired] attribute to the relevant class property.
  • Structs, maps, and lists also support [ParquetRequired] attribute.
  • Dates (DateTime) are serialized as INT96 numbers, which include nanoseconds in the day. In general, INT96 is obsolete in Parquet; however, older systems such as Impala and Hive are still actively using it to represent dates. Therefore, when this library sees an INT96 type, it will automatically treat it as a date for both serialization and deserialization. If you prefer a non-legacy date type, annotate a property with [ParquetTimestamp], which by default serializes dates with millisecond precision. If you need to increase precision, use the [ParquetTimestamp] attribute with an appropriate precision, like [ParquetTimestamp(ParquetTimestampResolution.Microseconds)]. Storing dates with microsecond precision relies on .NET DateTime, which can only store microseconds starting from .NET 7.
  • TimeSpan is serialized with millisecond precision, but you can increase it by adding the [ParquetMicroSecondsTime] attribute.
  • decimal is serialized with precision (number of digits in a number) of 38 and scale (number of digits to the right of the decimal point in a number) of 18. If you need to use different precision/scale pair, use [ParquetDecimal] attribute, which allows to specify different precision and scale, i.e. [ParquetDecimal(40, 20)].

You can also serialize more complex types supported by the Parquet format, like lists or maps. These are called nested types and they can be useful for organizing your information. However, they also come with a trade-off: they make your code slower and use more CPU resources. That’s why you should only use them when you really need them and not just because they look cool. Simple columns are faster and easier to work with, so stick to them whenever you can.

Structures

ℹ️ Note

Struct in Parquet is not the same as struct in C#. Parquet Struct represents a collection of data columns grouped together. In C# terms, Parquet struct is C# class (or C# struct).

Structures are just class members of a class and are completely transparent. Parquet.Net understands structures of any nesting level without any extra configuration. For instance, AddressBookEntry class may contain a structure called Address:

 1class Address {
 2    public string? Country { get; set; }
 3    public string? City { get; set; }
 4}
 5
 6class AddressBookEntry {
 7    public string? FirstName { get; set; }
 8    public string? LastName { get; set; }   
 9    public Address? Address { get; set; }
10}

Lists

One of the interesting things about lists is that Parquet can handle any kind of data structure in a list. You can have a list of atoms, like 1, 2, 3, or a list of lists, [[1, 2], [3, 4], [5, 6]], or even a list of structures. For instance, a simple MovementHistory class with Id and list of ParentIds is totally natively supported:

1class MovementHistory  {
2    public int? PersonId { get; set; }
3    public List<int>? ParentIds { get; set; }
4}

Or as a more complicate example, here is a list of structures (classes in C#):

 1class Address {
 2    public string? Country { get; set; }
 3    public string? City { get; set; }
 4}
 5
 6class MovementHistory {
 7    public int? PersonId { get; set; }
 8    public string? Comments { get; set; }
 9    public List<Address>? Addresses { get; set; }
10}

By default, Parquet.Net assumes that both lists and list elements are nullable. This is because list is a class in .NET, and classes are nullable. Therefore, coming back to the previous example definition:

1class MovementHistory {
2    public List<Address>? Addresses { get; set; }
3}

will produce the following low-level schema:

1OPTIONAL group Addresses (LIST) {
2  repeated group list {
3    OPTIONAL group element {
4        OPTIONAL binary Country (UTF8);
5        OPTIONAL binary City (UTF8);
6    }
7  }
8}

Usually this is not a problem, however you might encounter a problem when deserializing some files to handcrafted C# classes when nullability of the files and classes do not match the default.

To fix this, you can use [ParquetRequired] attribute on the list property:

1class MovementHistory {
2    [ParquetRequired]
3    public List<Address>? Addresses { get; set; }
4}

which will in turn produce the following low-level schema:

1REQUIRED group Addresses (LIST) {
2  repeated group list {
3    OPTIONAL group element {
4        OPTIONAL binary Country (UTF8);
5        OPTIONAL binary City (UTF8);
6    }
7  }
8}

As you can see, the “Addresses” container is now “required”. If you also need to ensure that each element (“Address” instance) within the list is non-nullable (i.e., required), you can use the [ParquetListElementRequired] attribute on the property. This attribute controls the nullability of the list elements in the Parquet schema, marking them as required.

1class MovementHistory {
2    [ParquetRequired, ParquetListElementRequired]
3    public List<Address>? Addresses { get; set; }
4}

which will produce the following low-level schema:

1REQUIRED group Addresses (LIST) {
2  repeated group list {
3    REQUIRED group element {
4        OPTIONAL binary Country (UTF8);
5        OPTIONAL binary City (UTF8);
6    }
7  }
8}

Maps

Maps are useful constructs to serialize key-value pairs where each row can have a different number of keys and they are not known beforehand. It’s like duck typing in a strongly typed world. A property will be treated as a native Parquet map if its type is the generic IDictionary<TKey, TValue>.

Legacy Repeatable

Parquet files can contain simple repeatable fields (AKA arrays) that store multiple values for a single column, which is not widely supported. To read an array of primitive values, use the ParquetSimpleRepeatable attribute on the property:

1class Primitives {
2    [ParquetSimpleRepeatable]
3    public List<bool>? Booleans { get; set; }
4}

ℹ️ Warning

Deserializing legacy repeatable fields incurs a massive performance penalty if arrays are large. Arrays are immutable in .NET, therefore every column value will re-create a new array of Length + 1, copy all the elements, and add one more.

Supported collection types

Similar to JSON supported collection types, here are collections Parquet.Net currently supports:

TypeSerializationDeserialization
Single-dimensional array (T[])✔️❌
Multi-dimensional arrays❌❌
IList<T>✔️❌
List<T>✔️✔️
IDictionary<TKey, TValue>❌❌
Dictionary<TKey, TValue>✔️✔️

* Technically impossible or very hard to implement. ** Technically possible, but not implemented yet.

Appending to files

ParquetSerializer supports appending data to an existing Parquet file. This can be useful when you have multiple batches of data that need to be written to the same file.

To use this feature, you need to set the Append flag to true in the ParquetOptions object that you pass to the SerializeAsync method. This will tell the library to append the data batch to the end of the file stream instead of overwriting it. For example:

1await ParquetSerializer.SerializeAsync(dataBatch, ms, new ParquetOptions { Append = true });

However, there is one caveat: you should not set the Append flag to true for the first batch of data that you write to a new file. This is because a Parquet file has a header and a footer that contain metadata about the schema and statistics of the data. If you try to append data to an empty file stream, you will get an IOException because there is no header or footer to read from. Therefore, you should always set the Append flag to false for the first batch (or not pass any options, which makes it false by default) and then switch it to true for subsequent batches. For example:

1// First batch
2await ParquetSerializer.SerializeAsync(dataBatch1, ms, new ParquetOptions { Append = false });
3
4// Second batch
5await ParquetSerializer.SerializeAsync(dataBatch2, ms, new ParqueOptions { Append = true });
6
7// Third batch
8await ParquetSerializer.SerializeAsync(dataBatch3, ms, new ParquetOptions { Append = true });

By following this pattern, you can easily append data to a Parquet file using ParquetSerializer.

Specifying row group size

Row groups are a logical division of data in a parquet file. They allow efficient filtering and scanning of data based on predicates. By default, all the class instances are serialized into a single row group, which is absolutely fine. If you need to set a custom row group size, you can specify it in ParquetOptions like so:

1await ParquetSerializer.SerializeAsync(data, stream, new ParquetOptions { RowGroupSize = 10_000_000 });

Note that small row groups make parquet files very inefficient in general, so you should use this parameter only when you are absolutely sure what you are doing. For example, if you have a very large dataset that needs to be split into smaller files for distributed processing, you might want to use a smaller row group size to avoid having too many rows in one file. However, this will also increase the file size and the metadata overhead, so you should balance the trade-offs carefully.

Ignoring property casing

Since v5.0 Parquet.Net supports ignoring property casing when deserializing data. This can be useful when you have a class with properties that have different casing than the column names in the parquet file. For example, if you have a class with properties like FirstName and LastName, but the parquet file has columns named first_name and last_name, you can use the IgnorePropertyCasing option to ignore the casing differences and match the properties with the columns. Here is an example:

 1class BeforeRename {
 2  public string? lowerCase { get; set; }
 3}
 4
 5class AfterRename {
 6  public string? LowerCase { get; set; }
 7}
 8
 9var data = Enumerable.Range(0, 1_000).Select(i => new BeforeRename {
10    lowerCase = i % 2 == 0 ? "on" : "off"
11}).ToList();
12
13// serialize to memory stream
14using var ms = new MemoryStream();
15await ParquetSerializer.SerializeAsync(data, ms);
16ms.Position = 0;
17
18// this will deserialize the data, but `LowerCase` property will be null, because it does not exist in the parquet file.
19DeserializationResult<AfterRename> data2 = await ParquetSerializer.DeserializeAsync<AfterRename>(ms);
20
21// this will successfully deserialize the data, because property names are case insensitive
22DeserializationResult<AfterRename> data3 = await ParquetSerializer.DeserializeAsync<AfterRename>(ms,
23    new ParquetOptions { PropertyNameCaseInsensitive = true });
  • Compression can be selected after constructing ParquetWriter, where compression method CompressionMethod and/or compression level (CompressionLevel) can be set. They default to Snappy, which is very reasonable.
  • Metadata reading and writing is supported on both parquet reader and writer.
  • Statistics can be read on a particular row group at zero cost by calling to GetStatistics(DataField field).
  • You can find other useful options in ParquetOptions class that can always be supplied to reader, writer, and serializer.

Encoding hints

When writing, default encoding may not be always what you want, therefore you can give hints to parquet writer on which encoding to use for which column. Any method that accepts ParquetOptions can have hints set. For instance, to use BYTE_STREAM_SPLIT encoding on id column you can do the following:

1var options = new ParquetOptions();
2options.ColumnEncodingHints["id"] = EncodingHint.ByteSplitStream;
3await using(ParquetWriter parquetWriter = await ParquetWriter.CreateAsync(parquetSchema, stream, options: options)) {
4    using ParquetRowGroupWriter groupWriter = parquetWriter.CreateRowGroup();
5    await groupWriter.WriteAsync<int>(dataField, Enumerable.Range(0, 256).ToArray());
6}

Appending to files

Pseudo appending to files is supported, however it’s worth keeping in mind that row groups are immutable by design, therefore the only way to append is to create a new row group at the end of the file. It’s worth mentioning that small row groups make data compression and reading extremely ineffective, therefore the larger your row group the better.

The following code snippet illustrates this:

 1//write a file with a single row group
 2var id = new DataField<int>("id");
 3var schema = new ParquetSchema(id);
 4var ms = new MemoryStream();
 5
 6await using(ParquetWriter writer = await ParquetWriter.CreateAsync(schema, ms)) {
 7    using(ParquetRowGroupWriter rg = writer.CreateRowGroup()) {
 8        await rg.WriteAsync<int>(id, new int[] { 1, 2 });
 9    }
10}
11
12//append to this file. Note that you cannot append to existing row group, therefore create a new one
13ms.Position = 0;    // this is to rewind our memory stream, no need to do it in real code.
14await using(ParquetWriter writer = await ParquetWriter.CreateAsync(schema, ms, append: true)) {
15    using(ParquetRowGroupWriter rg = writer.CreateRowGroup()) {
16        await rg.WriteAsync<int>(id, new int[] { 3, 4 });
17    }
18}
19
20//check that this file now contains two row groups and all the data is valid
21ms.Position = 0;
22await using(ParquetReader reader = await ParquetReader.CreateAsync(ms)) {
23    Assert.Equal(2, reader.RowGroupCount);
24
25    using(ParquetRowGroupReader rg = reader.OpenRowGroupReader(0)) {
26        Assert.Equal(2, rg.RowCount);
27        int[] values0 = new int[rg.RowCount];
28        await rg.ReadAsync<int>(id, values0);
29        Assert.Equal(new int[] { 1, 2 }, values0);
30    }
31
32    using(ParquetRowGroupReader rg = reader.OpenRowGroupReader(1)) {
33        Assert.Equal(2, rg.RowCount);
34        int[] values1 = new int[rg.RowCount];
35        await rg.ReadAsync<int>(id, values1);
36        Assert.Equal(new int[] { 3, 4 }, values1);
37    }
38}

Note that you have to specify that you are opening ParquetWriter in append mode explicitly - await ParquetWriter.CreateAsync(schema, ms, append: true). Doing so makes parquet.net open the file, find the file footer and delete it, rewinding current stream position to the end of actual data. Then, creating more row groups simply writes data to the file as usual, and DisposeAsync() on ParquetWriter generates a new file footer, writes it to the file and closes down the stream.

Please keep in mind that row groups are designed to hold a large amount of data (50000 rows on average) therefore try to find a large enough batch to append to the file. Do not treat parquet file as a row stream by creating a row group and placing 1-2 rows in it, because this will both increase file size massively and cause a huge performance degradation for a client reading such a file.

Type mapping

Parquet specification defines eight physical types: BOOLEAN, INT32, INT64, INT32, INT64, INT96, FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY.

LogicalPhysicalLogical annotation.NET typeConverted (legacy)
BooleanBOOLEANbool
IntegersINT32 or INT64IntTypebyte/sbyte, short/ushort, int/uint, long/ulongINT_/UINT_
FloatFLOATfloat
DoubleDOUBLEdouble
Big integerINT96BigInteger

working on it…

Parallelism

File streams are generally not compatible with parallel processing. You can, however, open file stream per parallel thread i.e. your Parallel.For should perform file opening operation. Or you can introduce a lock on file read, depends on what works better for you. I might state the obvious here, but asynchronous and parallel are different things.

Here is an example of reading a file in parallel, where a unit of parallelism is a row group:

 1await using var reader = await ParquetReader.CreateAsync(path);
 2var count = reader.RowGroupCount;
 3
 4await Parallel.ForAsync(0, count,
 5    async (i, cancellationToken) => {
 6        // create an instance of a row group reader for each group
 7        await using (var gr = await ParquetReader.CreateAsync(path)) {
 8            using (var rgr = gr.OpenRowGroupReader(i)) {
 9              // process the row group ...
10            }
11        }
12    }
13);

Columns

In the low-level API, columns are read and written directly through ParquetRowGroupWriter and ParquetRowGroupReader. Writing is done by calling WriteAsync<T> on the row group writer, passing the DataField from the schema, the values array, and optionally repetition levels:

1await groupWriter.WriteAsync<int>(field, new int[] { 1, 2, 3 });

For nullable columns, pass a nullable array:

1await groupWriter.WriteAsync<int>(field, new int?[] { 1, null, 3 });

Note that type parameter is always a non-nullable type, but method overloads accept both nullable and non-nullable collections, and this is how we know whether an extra step to extract nulls is required internally. For strings and byte arrays, convenience overloads accept IReadOnlyCollection<string?> and IReadOnlyCollection<byte[]?>, which internally convert to ReadOnlyMemory<char> and ReadOnlyMemory<byte> respectively, so you can use these as type parameters to the parent methods.

Reading is done by pre-allocating a buffer and calling ReadAsync<T> on the row group reader:

1int[] values = new int[rowGroupReader.RowCount];
2await rowGroupReader.ReadAsync<int>(field, values);

For nullable columns:

1int?[] values = new int?[rowGroupReader.RowCount];
2await rowGroupReader.ReadAsync<int>(field, values);

For advanced scenarios where you need direct access to definition levels and repetition levels, use ReadRawAsync<T> which accepts separate Memory<T>, Memory<int>? for definition levels, and Memory<int>? for repetition levels.

There is also read method ReadRawColumnDataAsync<T> that pre-allocates all buffers and returns a RawColumnData<T> container. RawColumnData holds these pre-allocated buffers and returns back into the pool in Dispose, therefore you must make sure you are using this return type in disposable pattern flow.

Schema

Schema is defined by creating an instance of ParquetSchema class and passing a collection of Fields. There are several types of fields in schema, and the most common is DataField, derived from the base abstract Field class (just like all the rest of the field types) and simply means it declares an actual data rather than an abstraction.

You can declare a DataField by specifying a column name, and its type in the constructor, generic form is also supported:

1var field1 = new DataField("id", typeof(int));
2var field2 = new Field<int>("id");

There are specialised versions for DataField allowing to specify more precise metadata about certain parquet data type, for instance DecimalDataField allows to specify precision and scale other than default values.

Non-data field wrap complex structures like list (ListField), map (MapField) and struct (StructField).

Full schema type hierarchy can be expressed as:

classDiagram
    Schema "1" o-- "1..*" Field
    
    Field <|-- ListField
    Field "1" --o "1" ListField: list item

    Field <|-- MapField
    Field "1" --o "1" MapField: key field
    Field "1" --o "1" MapField: value field

    Field <|-- StructField
    Field "1..*" --o "1" StructField: struct members


    class Schema {
        +List~Field~: Fields
    }

    class Field {
        +string Name
        +SchemaType SchemaType
        +FieldPath Path
    }

    class ListField {
        +Field: Item
    }
    
    class MapField {
        +Field: Key
        +Field: Value 
    }
    
    class StructField {
        +List~Field~: Fields
    }

    class DataField {
        +Type ClrType
        +Type ClrNullableIfHasNullsType
        +bool IsNullable
        +bool IsArray
    }
    
    class DateTimeFormat {
        << enumeration >>
        Impala
        DateAndTime
        DateAndTimeMicros
        Date
        Timestamp
    }

    class DateTimeDataField {
        +DateTimeFormat: DateTimeFormat
        +bool: IsAdjustedToUTC
        +DateTimeTimeUnit: Unit
    }

    DateTimeFormat "1" --o "1" DateTimeDataField: format

    class DecimalDataField {
        +int Precision
        +int Scale
        +bool: ForceByteArrayEncoding
    }
    
    class BigDecimalDataField {
        +int Precision
        +int Scale
    }
    
    class TimeUnitPrecision {
        << enumeration >>    
        Millis
        Micros
        Nanos
    }

    class TimeDataField {
        +bool: IsAdjustedToUtc
        +TimeUnitPrecision: Precision
    }

    TimeUnitPrecision "1" --o "1" TimeDataField: precision

    Field <|-- DataField
    DataField <|-- DateTimeDataField
    DataField <|-- TimeDataField
    DataField <|-- DecimalDataField
    DecimalDataField <|-- BigDecimalDataField

Lists

To declare a list, use ListField class and specify:

  1. The name of the list field.
  2. What data it is holding.

ListField’s second parameter is of type Field, meaning you can specify anything as its member - a primitive DataField or anything else.

Lists of primitive types

To declare a list of primitive types, just specify DataField as a second parameter. For instance, to declare a list of integers:

1new ListField("item", new DataField<int>("id"));

ℹ️ Warning

Parquet format allows you to use arrays which can also contain a list of primitive types. Whilst supported, it is strongly discouraged due to poor compatibility with other systems. In fact, it’s impossible to find anything that will write arrays instead of lists, even for primitive types.

To declare an array, or a “repeatable field” in schema you need specify it as IEnumerable<T> where T is one of the types Parquet.Net supports. For example:

1var se = new DataField<IEnumerable<int>>("ids");

You can also specify that a field is repeatable by setting isArray in DataField constructor to true.

When writing to the field you can specify any value which derives from IEnumerable<int>, for instance

1ds.Add(1, new int[] { 1, 2, 3 });

When reading schema back, you can check if it’s repeatable by calling to .IsArray property.

You should always prefer lists to arrays because:

  • Lists are more flexible - arrays can only contain a primitive type, whereas lists can contain anything.
  • Most big data platforms just default to lists.
  • Schema evolution is not possible with arrays.

Lists of structs

Of course a list can contain anything, but just to demonstrate it can, this is how you would declare a list of structs:

classDiagram
    direction LR
    Root "1" o-- "1..*" MyStruct
    
    class Root {
        +TopLevelId: int
        +List~MyStruct~: Structs
    }
    
    class MyStruct {
        +Id: int
        +Name: string
    }
1var idField = new DataField<int>("Id");
2var nameField = new DataField<string>("Name");
3
4var schema = new ParquetSchema(
5    new DataField<int>("TopLevelId"),
6    new ListField("Structs",
7        new StructField("MyStruct",
8            idField,
9            nameField)));

Special cases

  • Null Values by default are not allowed, and need to be declared explicitly by specifying a nullable type:
1new DataField<int?>("id");
2new DataField("id", typeof(int?));
  • Dates are stored by default as int96 number because of backward compatibility issues. To override date format storage you can use DateTimeDataField instead of DataField<DateTime> which allows to specify precision, for example the following example lowers precision to only write date part of the date without time.
1new DateTimeDataField("date_col", DateTimeFormat.Date);
  • Decimal by default uses precision 38 and scale 18, however you can set different precision and schema by using DecimalDataField schema element (see constructor for options).

Complex types

Structs

Structures are the easiest to understand. A structure is simply a container with extra fields i.e. table inside a table cell. From parquet’s point of view, there is no difference between a struct’s column and top-level column, they are absolutely identical.

Structures are mostly used to logically separate entities and simplify naming for a user. To demonstrate, let’s say you have the following very simple class hierarchy:

classDiagram
    direction RL
    Root "1" *-- "1" Address
    class Root {
        +string name
    }
    class Address {
        +string line1
        +string postcode
    }

In tabular form, it can be represented like this

nameaddress.line1address.postcode
(column)(column)(column)

which is also identical to

nameaddress
Columnline1 (column) | postcode (column)

Each table still has 3 physical columns, they are just named differently.

To make schema for this, we’ll use StructField which accepts other fields as children:

1var schema = new ParquetSchema(
2   new DataField<string>("name"),
3   new StructField("address",
4      new DataField<string>("line1"),
5      new DataField<string>("postcode")
6   ));

To write data, we use plain columns:

 1using var ms = new MemoryStream();
 2await using(ParquetWriter writer = await ParquetWriter.CreateAsync(schema, ms)) {
 3    using ParquetRowGroupWriter rgw = writer.CreateRowGroup();
 4
 5    await rgw.WriteAsync(schema.DataFields[0], new[] { "Joe" });
 6
 7    await rgw.WriteAsync(schema.DataFields[1], new[] { "Amazonland" });
 8
 9    await rgw.WriteAsync(schema.DataFields[2], new[] { "AAABBB" });
10}

To read back, again, the data is in plain columns:

 1ms.Position = 0;
 2
 3await using(ParquetReader reader = await ParquetReader.CreateAsync(ms)) {
 4    using ParquetRowGroupReader rg = reader.OpenRowGroupReader(0);
 5
 6    DataField[] dataFields = reader.Schema.GetDataFields();
 7
 8    string[] name = new string[rg.RowCount];
 9    await rg.ReadAsync(dataFields[0], name);
10
11    string[] line1 = new string[rg.RowCount];
12    await rg.ReadAsync(dataFields[1], line1);
13
14    string[] postcode = new string[rg.RowCount];
15    await rg.ReadAsync(dataFields[2], postcode);
16
17    Assert.Equal(new[] { "Joe" }, name);
18    Assert.Equal(new[] { "Amazonland" }, line1);
19    Assert.Equal(new[] { "AAABBB" }, postcode);
20}

Note that the only indication that this is a part of struct is Path property in the read schema containing struct name:

Lists and Arrays

Arrays aka repeatable fields is a basis for understanding how more complex data structures work in Parquet.

A column in Parquet can contain not just a single but multiple values per row. Sometimes they are called repeated fields (because the data type value repeats) or arrays. In order to create a schema for a repeatable field, let’s say of type int you could use one of two forms:

1var field = new DataField<IEnumerable<int>>("items");

To check if the field is repeated you can always test .IsArray Boolean flag.

Parquet columns are flat, so in order to store an array in the array which can only keep simple elements and not other arrays, you would flatten them. For instance to store two elements:

  • [1, 2, 3]
  • [4, 5]

in a flat array, it will look like [1, 2, 3, 4, 5]. And that’s exactly how parquet stores them. Now, the problem starts when you want to read the values back. Is this [1, 2] and [3, 4, 5] or [1] and [2, 3, 4, 5]? There’s no way to know without an extra information. Therefore, parquet also stores that extra information an an extra column per data column, which is called repetition levels. In the previous example, our array of arrays will expand into the following two columns:

#Data ColumnRepetition Levels Column
010
121
231
340
451

In other words - it is the level at which we have to create a new list for the current value. In other words, the repetition level can be seen as a marker of when to start a new list and at which level.

To write this in C# code:

1var field = new DataField<IEnumerable<int>>("items");
2// ...attach field to schema and create writer...
3await groupWriter.WriteAsync<int>(field,
4   new int[] { 1, 2, 3, 4, 5 },
5   new int[] { 0, 1, 1, 0, 1 });

Lists

Although arrays are useful, most of the systems write lists of data using List type. Unlike arrays, which can only contain primitive types, lists can contain anything. The most common use of lists is lists of structs.

Lists of Structs

To demonstrate, I’ll come back to the beginning of this document, and slightly change the relationship. Now our Root class does not just contain Address structure, but a list of address structures:

classDiagram
    direction LR
    Root "1" *-- "*" Address : list of
    class Root {
        +string name
    }
    class Address {
        +string line1
        +string postcode
    }

And we’d like to save the following data:

 1[
 2    {
 3        "name": "Joe",
 4        "addresses": [
 5            {
 6                "line1": "Amazonland",
 7                "postcode": "AAABBB"
 8            },
 9            {
10                "line1": "Disneyland",
11                "postcode": "CCCDDD"
12            }
13        ]
14    },
15    {
16        "name": "Bob",
17        "addresses": [
18            {
19	            "line1": "Cryptoland",
20    	        "postcode": "EEEFFF"
21            }
22        ]
23    }
24]

Knowing how structs and arrays are serialized, we can flatten this hierarchy to the following form so that it can be saved to Parquet:

nameRLaddresses.list.element.line1RLaddresses.list.element.postcodeRL
JoeAmazonland0AAABBB0
BobDisneyland1CCCDDD1
Cryptoland0EEEFFF0

where RL column indicated repetition levels of the column to the left.

name does not have any repetition levels as it’s a normal plain simple column.

line1 is a part of a list and it has a slightly longer name than usual. This is because of parquet naming conventions for lists. List must always annotate a 3-level structure:

  • Outer group, which is a name of your business property (addresses)
    • Middle level, always called list annotates repeatable field.
      • A field always called element that annotates the list contents. When lists contain a primitive type, this field is that type. In our case it’s a structure called element containing two fields - line1 and postcode.

If it feels complicated, it IS! Therefore general recommendation would be to use plain columns whenever possible. Nested types in parquet carry both mental and performance overhead 🤯

Moving on, let’s declare a schema for this:

 1var nameField = new DataField<string>("name");
 2var line1Field = new DataField<string>("line1");
 3var postcodeField = new DataField<string>("postcode");
 4
 5var schema = new ParquetSchema(
 6   nameField,
 7   new ListField("addresses",
 8   new StructField(ListField.ElementName,
 9      line1Field,
10      postcodeField)));

One thing to note - ListField automatically assumes there will be an internal middle level called list so it’s omitted from the schema declaration.

The struct is called "element" which is what ListField.ElementName constant is equal to. Theoretically you can name it anything you want, but common convention is recommended to be followed.

And the final thing is to write data for those 3 columns with their repetition levels:

1await using(ParquetWriter w = await ParquetWriter.CreateAsync(schema, ms)) {
2    using ParquetRowGroupWriter gw = w.CreateRowGroup();
3
4    await gw.WriteAsync(nameField, new string[] { "Joe", "Bob" });
5    await gw.WriteAsync(line1Field, new[] { "Amazonland", "Disneyland", "Cryptoland" }, new[] { 0, 1, 0 });
6    await gw.WriteAsync(postcodeField, new[] { "AAABBB", "CCCDDD", "EEEFFF" }, new[] { 0, 1, 0 });
7}

Congrats, you have saved your first list!

You might have noticed that list schema allows you to specify any Field - and that’s 100% correct. Lists can contain any element type, including other lists! The idea of saving lists of lists is identical to the above.

For more examples or just to run the above, please refer to unit tests in this project.

Maps

Maps are stored as lists of structures, where each structure has two elements - key and value. Theoretically you don’t need maps at all; it’s just a hint to the programming language to deserialize it in a more convenient way.

Untyped Serializer

Untyped serializer gives this library the ability to write and read data without the need to define low-level column data or define classes. This is extremely useful for use case like:

  • Reading data from a file where the schema is not known in advance (i.e. parquet file viewers, generic utilities).
  • Writing parquet file converters, i.e. from parquet to JSON.

The main motivation points to develop untyped serializer are:

  • Single codebase for class serializer and untyped dictionary serializer.
  • Deserialization produces JSON-like structures in memory. These can be written back to JSON files as-is.
  • Row API (seen in V4) is an old legacy that is somewhat buggy and very hard to evolve and fix.

In this API, everything is Dictionary<string, object>. For a simple use-case, with the following schema:

1var schema = new ParquetSchema(
2    new DataField<int>("id"),
3    new DataField<string>("city"));

you can write data like this:

1var data = new List<Dictionary<string, object>> {
2    new Dictionary<string, object> { { "id", 1 }, { "city", "London" } },
3    new Dictionary<string, object> { { "id", 2 }, { "city", "Derby" } }
4};
5
6await ParquetSerializer.SerializeUntypedAsync(schema, data, stream);

To read data back, use the reverse method:

1DeserializationResult<Dictionary<string, object>> result =
2    await ParquetSerializer.DeserializeUntypedAsync(stream);

Untyped serialization is extremely slow and should be avoided when possible.

DataFrame Support

Microsoft.Data.Analysis is supported via additional NuGet package. Due to DataFrame being in general less functional than Parquet, only primitive (atomic) columns are supported at the moment. If DataFrame supports more functionality in future (see related links below), this integration can be extended. When reading and writing, this integration will ignore any columns that are not atomic (primitive). You only need to call WriteAsync() extension method on DataFrame and specify the destination stream to write it to, similarly you can use System.IO.Stream’s extension method to read from parquet stream into DataFrame

1DataFrame df;
2await df.WriteAsync(stream);
3DataFrame dfr = await stream.ReadParquetAsDataFrameAsync();

Utilities

Merging files

Use Parquet.Utils.FileMerger to combine multiple parquet files with the same schema.

  • MergeFilesAsync(...) appends source row groups as-is. Fastest, most efficient, however if you have multiple small row groups it might be better to merge them using the method below.
  • MergeRowGroupsAsync(...) merges all source row groups into one (or more, configurable via rowGroupSize parameter, defaulting to 1 million rows ) destination row group.
 1var files = new[] {
 2    new FileInfo("part-1.parquet"),
 3    new FileInfo("part-2.parquet")
 4};
 5
 6await using var merger = new FileMerger(files);
 7await using var destination = File.Create("merged.parquet");
 8
 9await merger.MergeFilesAsync(destination);
10// or: await merger.MergeRowGroupsAsync(destination, rowGroupSize: 2_000_000);

Support

I’m using GitHub issues to track bugs and feature requests. You can view the list of open issues and submit a new one.

Special thanks

Without these tools development would be really painful.