mmap vs pread in a real Go storage engine

When you build a storage engine in Go, sooner or later you need to answer a very plain question:

“How should the code read bytes from files?”

This sounds too low-level to matter. A database has bigger ideas: partitions, blocks, indexes, filters, compression, compaction, caches, query planning. But all of these ideas end up doing the same simple action many times:

“Read N bytes from file F at offset O.”

We will look at that action through a real codebase: VictoriaLogs , a high-performance log database written in Go, and the shared VictoriaMetrics filesystem layer it uses. The point is not to sell VictoriaLogs. The point is to study a practical Go storage engine and learn why the answer is not just “use mmap” or “use pread”.

The boring answer is:

Use an abstraction. Prefer mmap when the data is already in memory. Fall back to pread when touching mmap memory could block a Go runtime thread on a major page fault.

That answer sounds strange if you are new to filesystems. Let’s build up to this answer together.

Aliaksandr Valialkin, CTO of VictoriaMetrics, wrote about this in “mmap may slow down your Go app”. The article explains that a goroutine touching cold mmap data can occupy an OS thread while the kernel resolves the major page fault, and if enough goroutines do this at once, Go execution can stall. See: valyala.medium.com/mmap-in-go-considered-harmful-d92a25cb161d .

Random reads everywhere

A storage engine does not usually read one giant file from start to end. For a query, it may need:

• a small block of index data
• a small column header
• one bloom filter
• one compressed timestamp block
• one values block for one field
• another values block for another field

Those reads are not always adjacent. They are often small, often random, and often concurrent (happening in many goroutines at once). VictoriaLogs has exactly this shape. A stored data part has multiple files: index data, column headers, timestamps, bloom filters, and values.

We are not here to understand the VictoriaLogs structure; it deserves its own post. The important thing is that the data part keeps these files behind a common random-read interface.

The interface is not “open file and stream it”. It is “read this exact range”. That is why pread and mmap enter our design.

What pread does

pread means “read from this file descriptor at this offset”.

The Linux manual says pread() reads from a file descriptor at a given offset and does not change the file offset. This makes it useful for multithreaded programs because multiple threads can read from the same file without fighting over a shared seek position.

See the Linux manual page for pread(2): man7.org/linux/man-pages/man2/pwrite.2.html .

In Go, os.File.ReadAt gives similar semantics at the Go API level. Under the hood, on Unix-like systems, this maps to positioned file reads. The shape is simple.

1. The program asks the kernel for bytes at an offset.
2. The kernel gets the data from the page cache or from storage.
3. The kernel copies the bytes into the buffer owned by the program.
4. The call returns.

Notice what this involves. The program enters the kernel through a syscall, the kernel copies the bytes into the buffer the program owns, and if the data is not in the page cache the call waits for disk or network storage before it returns.

So pread has visible cost. It is worth putting a number on it, because the number is what makes the whole mmap story matter.

A syscall is not a function call. It is a hardware-mediated crossing from user space (ring 3) into the kernel (ring 0) and back:

• the CPU switches stacks and page tables,
• saves and restores registers,
• and on hardware affected by Spectre and Meltdown runs a set of mitigations that fire unconditionally on every single entry and exit.

Jesús Espino takes that journey apart step by step in System Calls , and it is worth reading to see how much machinery surrounds one read().

None of that machinery is free. A bare mode switch costs a few hundred nanoseconds on modern hardware; once you add the Meltdown/Spectre mitigations and the small copy_to_user that a real read() performs, a single positioned read that hits the page cache lands on the order of one microsecond of CPU time. That is pure overhead, paid before a single useful byte is even looked at.

Now, remember the shape of the workload. A query does not do one big read; it does a flood of small random reads: index blocks, column headers, bloom filters, timestamp blocks, value blocks. A single query in VictoriaLogs or VictoriaMetrics can easily issue more than a million ReadAt calls. Multiply it out:

1,000,000 reads  ×  ~1 µs/read  ≈  1,000,000 µs  ≈  more than 1 second of CPU time

That is a full CPU-second burned on crossing the user/kernel boundary alone, for one query, before any of the real work like decompression, filtering, and merging is even counted.

What mmap does

mmap is different.

Instead of asking the kernel to copy file bytes into your buffer every time, the program asks the kernel to map the file into the process address space.

After that, reading the file looks like reading memory. The program touches an address. If the page is already resident, the load can be very fast. There is no read syscall for every small read, no explicit copy from the kernel into a new buffer.

The Linux manual says mmap() creates a new mapping in the virtual address space of the process. For file mappings, the contents come from the file. The manual also notes that files are mapped in page-size units, and that access beyond the mapped file can raise signals such as SIGBUS.

See: man7.org/linux/man-pages/man2/mmap.2.html .

This is the attractive part of mmap:

• many small reads can become cheap memory copies
• offsets become slice indexes
• the OS page cache still does the caching
• less syscall overhead on the hot path

Now compare the alternative. If those same bytes are already in the page cache and reachable through an mmap-ed region, the read is just a bounds check and a memory copy in user space. No boundary crossing, no mitigations, no copy_to_user. The per-read overhead drops from about a microsecond to a few nanoseconds, and that entire CPU-second largely disappears. This is the real reason the filesystem layer reaches for mmap on the hot path: not because copying bytes is slow, but because crossing into the kernel a million times is slow.

This is why the VictoriaMetrics filesystem layer originally added an mmap optimization for small reads.

mmap hides I/O behind memory access

The clean mental model says mmap turns file I/O into memory access. But the real mental model is more subtle: “mmap turns file I/O into page faults.”

When your code touches a mapped address, one of two things happens:

• The page is already resident in memory. The read is fast.
• The page is not resident. The CPU faults. The kernel must load the page from storage. The goroutine looks like it is just reading memory, but the OS thread is waiting.

This second case is called a major page fault.

But pread also has an important property for Go that mmap does not: the runtime knows this is a syscall.

In the Go runtime, at most GOMAXPROCS threads can execute user-level Go code at the same time. Threads blocked in system calls do not count against this limit.

This matters a lot. If a goroutine blocks inside a filesystem read syscall, not a network syscall, the runtime can detach the blocked OS thread from its processor and start or reuse another OS thread to keep other goroutines moving. The blocked OS thread and the goroutine doing the syscall stay blocked until the syscall returns.

So the takeaway is that if an OS thread is blocked in a normal syscall, Go can still keep other goroutines moving on other OS threads, up to the GOMAXPROCS limit.

For C/C++ code, this can still be acceptable. For Go, there is a special runtime problem. The Go runtime can see blocking syscalls, but it does not see a normal memory load as a blocking operation.

What does that mean?

A normal memory load is just an instruction executed by the CPU. If that load causes a major page fault, the operating system may block the OS thread while it brings the page in from disk, but the Go runtime did not get a syscall entry event. From the runtime scheduler’s point of view, that goroutine was still executing normal Go code when the fault happened.

And if the thread is blocked by a major page fault from a normal memory load, Go usually cannot detach that thread the same way it can detach a thread entering a normal syscall. So if enough OS threads are stuck in major page faults, especially if they reach GOMAXPROCS, the program can stall or lose most of its runnable capacity for a while. Other goroutines may be ready, but there may be no free processor (P) available to run them until some faulting threads return.

In general, Go handles visible blocking syscalls well, but this kind of blocking is below the runtime scheduler’s normal visibility.

Why a storage engine wants both

If mmap can stall Go, why not always use pread?

Because mmap is still useful when the page is hot. A storage engine often reads metadata and small blocks over and over. If the working set is in the page cache, mmap can avoid repeated read syscalls. The read path becomes a bounds check and a memory copy.

If the working set is not in the page cache, mmap can cause major page faults. This can hurt latency beyond the goroutine that touched the cold page.

So the useful question is not: “Should we use mmap or pread?” The useful question is: “Do we know this page is already resident?” If yes, mmap is attractive. If no, pread is safer for the Go runtime.

mincore

Linux has a syscall for checking page residency: mincore.

The Linux manual says mincore() returns a vector that shows whether pages are resident in memory and will not cause disk access if referenced. It also warns that the result is only a snapshot; pages can change after the call returns.

See: man7.org/linux/man-pages/man2/mincore.2.html .

In the current VictoriaMetrics codebase, the reader does this:

1. If mmap is disabled, read via syscall.
2. If mmap is enabled, check whether the target pages are safe for a fast mmap read.
3. If the page check passes, copy from mapped memory.
4. If the page check fails, read via syscall.

The code also caches mincore results for a short time. That matters because calling mincore for every tiny read would add its own overhead. The cached residency information is kept for up to one minute before it is cleared and rechecked.

The ReadAt abstraction

The core of this design is inside two structures: ReaderAt and mmapReader. First, ReaderAt:

type ReaderAt struct {
	path string

	mr     atomic.Pointer[mmapReader]
	mrLock sync.Mutex
	...
}

In the VictoriaMetrics filesystem layer, ReaderAt represents random access to a file. It knows the file path and, after the first real read, it holds the mmap state for that file. An mmap state is an interesting detail that we will open up soon.

A file part is opened once, but many query workers or goroutines may read different byte ranges from that same file at the same time. This is why ReaderAt has both an atomic pointer atomic.Pointer[mmapReader] and a mutex sync.Mutex.

However, the mutex is not for every read. It only protects the first initialization of mmapReader. A storage engine may know about many parts and files, but not all of them will be read immediately. Some parts may be skipped by time range pruning, some blocks may be skipped by filters, some columns may never be needed by the query.

If we opened and mmapped every file immediately, startup would pay the cost for many files that may not be touched soon, or may not be touched at all. That means more file descriptors, more mmap regions, more mmap bookkeeping, more pressure on OS limits, and slower startup.

So the file open operation happens later, when the real read is called for the first time.

func (r *ReaderAt) getMmapReader() *mmapReader {
	mr := r.mr.Load()
	if mr != nil {
		return mr
	}

	r.mrLock.Lock()
	mr = r.mr.Load()
	if mr == nil {
		mr = newMmapReaderFromPath(r.path)
		r.mr.Store(mr)
	}
	r.mrLock.Unlock()

	return mr
}

The atomic load is the fast path. After the mmap reader has already been created, most calls can read the pointer directly and return without taking the mutex.

The mutex is for the slow path. When the pointer is still nil, one goroutine must create the mmap reader. The second atomic load inside the mutex is also needed because another goroutine may have created the mmap reader while the current goroutine was waiting for the lock. So after getting the lock, we check again.

Now, let’s come to the point where we actually read a file, MustReadAt function:

func (r *ReaderAt) MustReadAt(p []byte, off int64) {
	if len(p) == 0 {
		return
	}
	if off < 0 {
		logger.Panicf("BUG: off=%d cannot be negative", off)
	}

	mr := r.getMmapReader()

	if len(mr.mmapData) == 0 {
		mr.mustReadAtViaSyscall(p, off)
	} else {
		if mr.canFastReadViaMmap(off, len(p)) {
			src := mr.mmapData[off:]
			copy(p, src)
		} else {
			mr.mustReadAtViaSyscall(p, off)
		}
	}
}

Everything above asks the same simple thing: fill this buffer from this offset. The caller does not pass a strategy. It does not say mmap or pread. It only gives two facts: where the bytes should go p []byte, and where the bytes start in the file off int64.

Read this slowly, because this is the whole mmap versus pread story in a small form.

First, the reader lazily opens the file and prepares an internal mmapReader. If mmap is disabled, or if there is no mapped data for this file, mr.mmapData is empty. In that case there is no mapped memory to copy from, so the method goes straight to the syscall path:

if len(mr.mmapData) == 0 {
	mr.mustReadAtViaSyscall(p, off)
}

...
func (mr *mmapReader) mustReadAtViaSyscall(p []byte, off int64) {
	n, err := mr.f.ReadAt(p, off)
	if err != nil {
		logger.Panicf("FATAL: cannot read %d bytes at offset %d", len(p), off)
	}
	if n != len(p) {
		logger.Panicf("FATAL: unexpected number of bytes read")
	}
	...
}

That path is just positioned file I/O. In Go it is ReadAt; on Unix-like systems this is the family of pread-style reads. The important detail is that the call is visible to the Go runtime as a blocking file operation.

If mmap is available, we still do not blindly copy from it. We first ask canFastReadViaMmap(off, len(p)). That name is doing real work:

func (r *ReaderAt) MustReadAt(p []byte, off int64) {
	...

	if len(mr.mmapData) == 0 {
		mr.mustReadAtViaSyscall(p, off)
	} else {
		if mr.canFastReadViaMmap(off, len(p)) {
			src := mr.mmapData[off:]
			copy(p, src)
		} else {
			mr.mustReadAtViaSyscall(p, off)
		}
	}
}

This check means “can we read this range through mmap without likely causing a major page fault?” The fast path is very simple. mr.mmapData is a byte slice backed by the mapped file. The offset (off) becomes a slice index and the read becomes copy(p, src). If the page is already resident, this avoids a read syscall for that small range.

But how does it work?

It does that with mincore, the OS check that tells whether a memory page is resident in RAM. If the page is resident, reading through mmap should be cheap. If the page is not resident, this function returns false, so the caller can avoid mmap and use a normal file read path instead.

func (mr *mmapReader) canFastReadViaMmap(off int64, n int) bool {
	...

	end := off + int64(n)
	off -= int64(uint64(off) % pageSize)
	pageIdx := uint64(off) / pageSize
	for off < end {
		wordIdx := pageIdx / 64
		bitIdx := pageIdx % 64
		mask := uint64(1) << bitIdx
		wordPtr := &mincoreBits[wordIdx]
		word := wordPtr.Load()
		if (word & mask) == 0 {
			if !mincore(&mr.mmapData[off]) {
				return false
			}
			for (word&mask) == 0 && !wordPtr.CompareAndSwap(word, word|mask) {
				word = wordPtr.Load()
			}
		}

		off += int64(pageSize)
		pageIdx++
	}

	return true
}

To make sense of this “complicated” code, we need to know that memory is managed in fixed-size chunks called “pages”. A page is often 4 KiB, 8 KiB, or 16 KiB, depending on the system. When a file is mmapped, the file bytes are mapped into virtual memory pages.

So, a page is the unit used to check whether a memory region is in memory or not, and whether reading it can cause a major page fault.

We use a word bitmap, mincoreBits, to store page information. One bit represents one page. A 64-bit word has 64 bits, so it can store 64 yes/no states. Each bit answers one question: “was this page recently checked and found resident?” If the bit is 1, the page is treated as resident in the cache. If the bit is 0, the code does not know yet, so it calls mincore.

We take the requested read range, move the start offset backward to the beginning of its memory page to be aligned, then walk page by page until it covers the whole read. For each page, we compute which cached bit represents that page in mincoreBits: one 64-bit word stores 64 page states, so wordIdx chooses the word and bitIdx chooses the bit inside that word.

If the bit is already set, we assume this page was recently checked and resident in RAM, so we skip the expensive mincore call. If the bit is not set, we call mincore on the mmap address for that page.

• If mincore says the page is not resident, we skip mmap and literally say “do not use mmap for this read.”
• If the page is resident, we set the cached bit, so future reads can skip checking that same page.

If every page in the requested range is either already cached as resident or confirmed resident by mincore, the function returns true, meaning this mmap read is likely safe and fast.

When the read falls back to the syscall path and ReadAt succeeds, the code can also mark the affected pages in mincoreBits. The syscall path also teaches the mmap fast path that these pages have just been read and are likely resident now. Future reads of the same pages may then use mmap without calling mincore again.

func (mr *mmapReader) mustReadAtViaSyscall(p []byte, off int64) {
	...
	
	if len(mr.mmapData) == 0 || !hasMincore() {
		return
	}

	// Mark the just read data as available for fast read via mmap
	mincoreBits := mr.mincoreBits
	pageSize := pageSizeBytes

	end := off + int64(n)
	off -= int64(uint64(off) % pageSize)
	pageIdx := uint64(off) / pageSize
	for off < end {
		wordIdx := pageIdx / 64
		bitIdx := pageIdx % 64
		mask := uint64(1) << bitIdx
		wordPtr := &mincoreBits[wordIdx]
		word := wordPtr.Load()
		for (word&mask) == 0 && !wordPtr.CompareAndSwap(word, word|mask) {
			word = wordPtr.Load()
		}

		off += int64(pageSize)
		pageIdx++
	}
}

However, this is an “opinionated program-level heuristic” and is not supported by the OS natively. What does that mean?

mincore can tell the program that a page is resident at the moment it checks. The cached bit means “we checked this page recently and it was resident then.” It does not mean “this page is guaranteed to stay in RAM.”

The OS is still free to evict file-backed pages under memory pressure, and we do not know when this happens. There is also a small race: a page can be resident when mincore checks it, then become non-resident before the actual mmap read touches it. But it is an edge case, so let’s ignore it.

So we are reducing risk, not removing it completely. To deal with the stale data, the cache is cleaned periodically. The program does not trust old residency information forever. This cleanup happens at the start of this function:

func (mr *mmapReader) canFastReadViaMmap(off int64, n int) bool {
	...
	mincoreBits := mr.mincoreBits
	pageSize := pageSizeBytes

	ct := fasttime.UnixTimestamp()
	nextCleanup := mr.mincoreNextCleanupTimestamp.Load()
	if ct > nextCleanup && mr.mincoreNextCleanupTimestamp.CompareAndSwap(nextCleanup, ct+60) {
		for i := range mincoreBits {
			mincoreBits[i].Store(0)
		}
	}
	...
}

It is a lazy cleanup for the page-residency cache. When a read comes in, we check the clock. If the cache has not been cleaned for more than about 60 seconds, that read triggers cleanup and clears the cached page states. Then we start checking pages again.

All functions here are concurrency-safe. The CompareAndSwap (CAS) makes cleanup ownership a single-winner process. All goroutines can read the old cleanup time, but only one goroutine successfully changes it to the next cleanup time. That goroutine becomes responsible for clearing the cache. The others see that they lost and continue without clearing.

Why 32-bit systems are different

When a process uses mmap, it does not copy the whole file into RAM. That part is important. A mapped file is backed by the OS page cache, and pages are loaded on demand. But the process still needs a virtual address range large enough to represent the mapping. In other words, if the program wants to mmap a 10GB file, it needs a 10GB region in its virtual address space, even if only a few pages are resident in physical memory right now.

On a 64-bit process, this is usually not a big problem. The virtual address space is huge, so mapping large files is practical. The file may be larger than RAM, but the address space can still describe it. That is one reason mmap is attractive on 64-bit systems: it lets the program treat file bytes like memory addresses, while the kernel loads real pages only when they are touched.

A 32-bit process has a much smaller address space. In the simplest mental model, pointers are 32 bits, so the process can only address around 4GB total. That space is not only for one file mapping. It is also used by the program itself, heap, stacks, shared libraries, runtime data, and other mappings. So a data file near or above 4GB is already too large to map as one contiguous region. Even smaller files can become hard to map if the process address space is fragmented or already used.

That is exactly why the default was changed. The original code used mmap by default unless -fs.disableMmap was set. Later, that default was made to depend on pointer size instead of always defaulting to mmap. On 64-bit architectures, mmap stays enabled by default. On 32-bit architectures, mmap is disabled by default, so the reader uses the pread path instead.

The code detects this with a tiny constant:

const is32BitPtr = (^uintptr(0) >> 32) == 0

Then it uses that value as the default for -fs.disableMmap:

var disableMmap = flag.Bool("fs.disableMmap", is32BitPtr, ...)

So the story is: mmap is not “load file into RAM”, but it is “reserve virtual address space for this file mapping”. That is cheap and normal on 64-bit. It is risky or impossible on 32-bit for large database files. Because VictoriaMetrics and VictoriaLogs may work with files larger than a 32-bit process can map, the filesystem layer chooses the safer default: use pread on 32-bit unless the user explicitly enables mmap.

SIGBUS and mmap

SIGBUS means “bus error”. In the mmap story, it usually means: the process touched a virtual memory address that is mapped, but the kernel cannot supply valid bytes for that address from the underlying file.

That is different from normal pread. With pread, if you ask for bytes past the file end, the read returns an error or short read. The failure is part of the syscall result. With mmap, your code is just doing a memory load. If that memory page cannot be backed by the file, the kernel cannot return an ordinary Go error from copy. It sends a signal to the process instead. That signal can be SIGBUS.

Why is this related to our story?

Because mmap changes the failure mode. Near the end of a file, this becomes subtle because files are arbitrary byte lengths, while mmap works in pages.

Imagine the file is 10,003 bytes. Let’s say the OS page size is 4 KiB. The last file page is only partly backed by real file data. If code copies near the end of the mmap region, a highly optimized memory copy may read in machine-sized chunks. Even if the logical Go slice length says “only copy these bytes”, the runtime copy implementation may internally read a little wider for speed. If that wider read crosses into a page that is not safely mapped/backed, the process can get SIGBUS.

A storage engine must care about these edge cases because it often reads small binary ranges near exact offsets.

To deal with this, we made a wrapper for mmap to be SIGBUS-aware:

func mmapFile(f *os.File, size int64) ([]byte, error) {
	if size == 0 {
		return nil, nil
	}
	if size < 0 {
		return nil, fmt.Errorf("got negative file size: %d bytes", size)
	}
	if int64(int(size)) != size {
		return nil, fmt.Errorf("file is too big to be memory mapped: %d bytes", size)
	}
	// Round size to multiple of 4KB pages as `man 2 mmap` recommends.
	// This may help preventing SIGBUS panic at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581
	// The SIGBUS could occur if standard copy(dst, src) function may read beyond src bounds.
	sizeOrig := size
	if size%4096 != 0 {
		size += 4096 - size%4096
	}
	data, err := mmap(int(f.Fd()), int(size))
	if err != nil {
		return nil, fmt.Errorf("cannot mmap file with size %d bytes; already memory mapped files: %d: %w; "+
			"try increasing /proc/sys/vm/max_map_count or passing -fs.disableMmap command-line flag to the application", size, mmappedFiles.Get(), err)
	}
	mmappedFiles.Inc()
	return data[:sizeOrig], nil
}

We map a size rounded up to a 4KB page boundary, but return a slice with the original file size. So normal callers still see the correct file length, while the actual mmap region has extra mapped room at the end.

Later, the close path uses cap(mr.mmapData) for unmap, because the real mapped region may be larger than len(mr.mmapData).

if err := mUnmap(mr.mmapData[:cap(mr.mmapData)]); err != nil {
	logger.Panicf("FATAL: cannot unmap data for file %q: %s", fname, err)
}

To summarize, SIGBUS matters because mmap makes file bytes look like memory, but the file still has boundaries. If a memory read crosses into a part of the mapping the kernel cannot back with file data, the process may receive SIGBUS and panic.

ZFS changed the story again

ZFS is a filesystem.

More precisely, it is both a filesystem and a volume manager. That means it does not only decide how files and directories look. It also manages disks, pools, checksums, snapshots, compression, caching, and data repair.

ZFS has its own in-memory cache called ARC. That means the storage path is no longer just “Linux page cache + disk”. With ZFS, there is another cache layer involved, with its own behavior and bugs.

Unfortunately, using mincore() on mmapped files may trigger a bug related to the ZFS ARC cache. The bad case is especially related to mixing reads from mmapped files and direct disk reads. That mix can corrupt the ZFS ARC cache and lead to data read corruption.

So we added a fs.disableMincore flag to disable mincore on older ZFS filesystems.

That is a very different kind of problem from the earlier Go runtime stall. The earlier problem was performance and scheduling: cold mmap pages could block runtime threads. The ZFS problem is correctness: under some older ZFS setups, the optimization used to avoid stalls can participate in read corruption.

When mincore is disabled, canFastReadViaMmap returns true, so the code no longer uses mincore to avoid cold mmap pages. That means we lose this specific protection against Go runtime stalls from cold mapped pages. But on affected ZFS systems, that trade-off is worth it because correctness is more important than the optimization.