Home/Learn/Operating Systems/Memory-Mapped I/O & Files

Memory-Mapped I/O & Files

Advanced

I/O & Kernel

Memory-mapped files map file contents directly into a process's virtual address space, allowing file I/O to be performed as ordinary memory operations with OS-managed demand paging.

Overview

Normally, reading a file requires two copies: OS reads from disk into kernel page cache, then copies from kernel buffer to user-space buffer (double buffering). Memory mapping eliminates the second copy: the OS maps the file's page cache directly into the process's virtual address space. When the process reads a mapped address, a page fault triggers the OS to load the page from disk into the page cache — which is also the process's mapped memory. This is the same mechanism used to load executable code (.text segment) and shared libraries. Memory-mapped I/O is ideal for large files, inter-process shared memory, and database buffer pools. Java's FileChannel.map() returns a MappedByteBuffer that enables this pattern. Java 21's MemorySegment (Foreign Memory API) provides a safer, more powerful replacement.

MappedByteBuffer vs Traditional FileInputStream

FileInputStream.read() involves a system call, a copy from kernel page cache to JVM heap. MappedByteBuffer maps the page cache directly — no copy, just virtual memory access. The OS pages in data on demand, making it especially efficient for random-access patterns on large files.

Java — FileInputStream vs MappedByteBuffer performance comparison

import java.nio.*;
import java.nio.channels.*;
import java.nio.file.*;
import java.io.*;

Path file = Path.of("/tmp/large.csv");

// ── Traditional: FileInputStream (double-buffered) ────────────────────────
long startTraditional = System.nanoTime();
try (var fis = new FileInputStream(file.toFile());
     var bis = new BufferedInputStream(fis, 65536)) {
    byte[] buf = new byte[65536];
    long totalBytes = 0;
    int n;
    while ((n = bis.read(buf)) != -1) totalBytes += n;
    System.out.println("Traditional: " + totalBytes + " bytes");
}
long traditionalMs = (System.nanoTime() - startTraditional) / 1_000_000;

// ── Memory-Mapped: FileChannel.map() (zero extra copy) ───────────────────
long startMapped = System.nanoTime();
try (FileChannel fc = FileChannel.open(file)) {
    long fileSize = fc.size();

    // Map entire file into virtual address space (READ_ONLY)
    // Mode: READ_ONLY, READ_WRITE, or PRIVATE (copy-on-write)
    MappedByteBuffer mapped = fc.map(
        FileChannel.MapMode.READ_ONLY,
        0,          // starting position in file
        fileSize    // number of bytes to map
    );
    // mapped now lives in virtual address space
    // OS pages in data on demand — no full load upfront

    long totalBytes = 0;
    while (mapped.hasRemaining()) {
        mapped.get();  // direct memory read — no syscall per byte
        totalBytes++;
    }
    System.out.println("Memory-mapped: " + totalBytes + " bytes");

    // Force all pages to be loaded and written to disk (if READ_WRITE)
    mapped.force(); // optional: flush modified mapped region to disk
}
long mappedMs = (System.nanoTime() - startMapped) / 1_000_000;

System.out.printf("Traditional: %dms | Memory-mapped: %dms%n",
    traditionalMs, mappedMs);
// Memory-mapped is typically 2-5x faster for large sequential reads

Shared Memory IPC and Use Cases

Memory-mapped files can be used for inter-process communication (IPC): two processes map the same file and read/write shared memory directly, without any kernel buffering overhead. This is how high-performance message queues (Disruptor, Chronicle Map) achieve sub-microsecond latency. Java 17+ MemorySegment (Foreign Memory API) provides a safer, explicit lifetime-managed alternative.

Java — Memory-mapped IPC and Java 21 MemorySegment

import java.nio.*;
import java.nio.channels.*;
import java.nio.file.*;

// IPC via shared memory-mapped file
// Process A: writes data to mapped region
Path sharedFile = Path.of("/tmp/shared-memory.bin");

try (FileChannel fc = FileChannel.open(sharedFile,
        StandardOpenOption.CREATE,
        StandardOpenOption.READ,
        StandardOpenOption.WRITE)) {

    // Map 4KB as READ_WRITE
    MappedByteBuffer shared = fc.map(FileChannel.MapMode.READ_WRITE, 0, 4096);

    // Write a 64-bit counter (Process A side)
    shared.putLong(0, System.currentTimeMillis()); // write at offset 0
    shared.putInt(8, 42);                          // write int at offset 8
    shared.force(); // flush to disk (visible to other process)

    System.out.println("Wrote timestamp: " + shared.getLong(0));
    System.out.println("Wrote value:     " + shared.getInt(8));

    // Process B would open the same file, map it, and read:
    // long timestamp = shared.getLong(0);
    // int value      = shared.getInt(8);
}

// Java 21 Foreign Memory API (safer alternative):
// import java.lang.foreign.*;
// try (Arena arena = Arena.ofShared()) {
//     MemorySegment segment = arena.allocate(4096);
//     segment.set(ValueLayout.JAVA_LONG, 0, System.currentTimeMillis());
//     long val = segment.get(ValueLayout.JAVA_LONG, 0);
// } // explicit lifetime management — no GC pressure

// Key use cases for memory-mapped I/O:
// 1. Large file processing (log parsing, CSV analytics)
// 2. IPC via shared memory (Chronicle Map, Aeron messaging)
// 3. Database buffer pools (mmap-based: SQLite, LMDB)
// 4. Loading executables and shared libraries (JVM does this for .jar files)

Key Points to Remember

1Memory-mapped files eliminate double-buffering by mapping the OS page cache directly into the process virtual address space.
2The OS loads pages on demand via page faults — the entire file is never loaded at once, saving memory.
3FileChannel.map() returns a MappedByteBuffer; use MapMode.READ_ONLY, READ_WRITE, or PRIVATE (copy-on-write).
4Memory-mapped I/O is ideal for large files with random access patterns; sequential small files may not benefit.
5MappedByteBuffer.force() is equivalent to msync() — it flushes modified mapped pages to disk.
6High-performance Java libraries (Chronicle Map, Aeron, Disruptor) use memory-mapped files for sub-microsecond IPC.

Interview Questions

What is memory-mapped I/O and how does it differ from traditional buffered file I/O?

MediumGoogle

What is double-buffering and how does memory mapping eliminate it?

MediumAmazon

How would you use memory-mapped files for inter-process communication in Java?

HardUber

What are the risks of using MappedByteBuffer in Java and how do you safely unmap it?

HardNetflix

Ask Aria about Memory-Mapped I/O & Files

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

System Calls & Kernel Mode