GitHub - nnnkkk7/go-simdcsv: High-performance SIMD-accelerated CSV parser for Go 1.26+ using simd/archsimd. Drop-in replacement for encoding/csv

SIMD-accelerated CSV parser for Go — drop-in replacement for encoding/csv with AVX-512 optimization.

Experimental: Requires Go 1.26+ with GOEXPERIMENT=simd. The SIMD API is unstable and may change. Not recommended for production use.

Quick Start

import csv "github.com/nnnkkk7/go-simdcsv"

reader := csv.NewReader(strings.NewReader("name,age\nAlice,30\nBob,25"))
records, err := reader.ReadAll()
if err != nil {
    log.Fatal(err)
}
// records: [["name", "age"], ["Alice", "30"], ["Bob", "25"]]

Same API as encoding/csv — just change the import.

Features

Installation

go get github.com/nnnkkk7/go-simdcsv

Usage

Reading

// All at once
records, err := csv.NewReader(r).ReadAll()

// Record by record
reader := csv.NewReader(r)
for {
    record, err := reader.Read()
    if err == io.EOF { break }
    if err != nil { return err }
    // process record
}

// Direct byte parsing (fastest)
records, err := csv.ParseBytes(data, ',')

// Streaming callback
csv.ParseBytesStreaming(data, ',', func(record []string) error {
    return processRecord(record)
})

Writing

writer := csv.NewWriter(w)
writer.WriteAll(records)

Configuration

All standard encoding/csv options are supported:

reader := csv.NewReader(r)
reader.Comma = ';'              // Field delimiter (default: ',')
reader.Comment = '#'            // Comment character
reader.LazyQuotes = true        // Allow bare quotes
reader.TrimLeadingSpace = true  // Trim leading whitespace
reader.ReuseRecord = true       // Reuse slice for performance
reader.FieldsPerRecord = 3      // Expected fields (0 = auto-detect, -1 = variable)

Extended options:

reader := csv.NewReaderWithOptions(r, csv.ReaderOptions{
    SkipBOM: true,  // Skip UTF-8 BOM if present
})

Performance

Benchmarks on AMD EPYC 9R14 with AVX-512 (Go 1.26, GOEXPERIMENT=simd). See Contributing for the CI setup.

ReadAll Throughput

Note: SIMD acceleration benefits unquoted CSV data at scale. Realistic rows reflect real-world CSV with some quoted fields containing commas (10% or 40% of fields quoted); SIMD provides a modest speedup in these cases. Quoted fields currently have overhead from validation passes.

Architecture

┌─────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Raw Input  │ ──▶ │ scanBuffer() │ ──▶ │ parseBuffer()│ ──▶ │buildRecords()│ ──▶ [][]string
│   []byte    │     │  (bitmasks)  │     │  (positions) │     │  (strings)   │
└─────────────┘     └──────────────┘     └──────────────┘     └──────────────┘

Requirements

Without AVX-512

The library still works but falls back to scalar implementation (no speedup). This includes:

• Most CI environments (GitHub Actions ubuntu-latest, etc.)
• Intel CPUs before Skylake-X
• AMD CPUs before Zen 4
• Apple Silicon (ARM64)

Building & Testing

# Build
GOEXPERIMENT=simd go build ./...

# Test
GOEXPERIMENT=simd go test -v ./...

# Benchmark
GOEXPERIMENT=simd go test -bench=. -benchmem

Known Limitations

• Experimental API: simd/archsimd may have breaking changes in future Go releases
• Quoted fields: Currently slower than encoding/csv for heavily quoted content

Contributing

Contributions are welcome! Please open issues or pull requests on GitHub.

CI

This repository runs two GitHub Actions workflows:

Pull request checks use ubuntu-latest, which typically lacks AVX-512. Tests still pass because the library falls back to the scalar path when SIMD instructions are unavailable.

AVX-512 benchmarks and SIMD validation run only on the self-hosted runner via AVX-512 CI. Trigger it manually from the Actions tab when you need real SIMD test and benchmark results.

License

MIT License - see LICENSE file for details.