Understanding Computer Memory Architecture and SSD Internals

Memory is a hardware component that stores binary information in blocks. Every memory device stores data as a sequence of binary digits (1s and 0s). Memory often appears as one continuous address space, but physical computer systems are built from multiple memory technologies arranged in a hierarchy. Each technology has its own form factor and involves trade-offs among capacity, access latency, bandwidth, power consumption, and manufacturing cost. Memory elements retain information over time, allowing digital systems to preserve state and execute sequential operations.

Every piece of information processed inside a computer eventually resides in some form of memory. Program instructions, variables, images, cryptographic keys, network packets, and intermediate computation results are all represented as binary values stored within a memory device.

Common Characteristics of Memory

• Zero access latency
• Non-volatile data retention
• Low power consumption
• Unlimited storage capacity

Memory Structure

A memory is represented as a large rectangular area with width and depth. The width represents the number of bits that can be allocated to a single memory block index, and depth defines the number of data-block locations.

For example: If a data-bit is 8-bit wide and stored in 128 locations, then the total memory bits are 128 x 8 = 1024 bits.

Memory capacity is determined by WIDTH x DEPTH

Memory Operations

Every memory device supports two fundamental operations

• Read: retrieves the data stored at a specified memory address.
• Write: stores new data at a specified memory address.

Common memory interfaces

• Address Bus
• Data Bus
• Read Enable
• Write Enable
• Clock (for synchronous memories)

Memory Hierarchy

In the previous section, I have described memory as a rectangular storage structure organized by its width and depth. This representation is useful for understanding how data is stored but it does not accurately describe how memory is organized within a modern computer system.

A general-purpose computer does not contain a single memory device. Instead, it consists of multiple memory technologies, each designed to satisfy different requirements for performance, capacity, access latency, and manufacturing cost. These memory technologies are organized into a memory hierarchy, where each level balances storage capacity and access speed differently.

When a processor executes a program, it continuously requests instructions and data from memory. The processor first checks the smallest, fastest memory closest to the execution units rather than accessing main memory directly. If the requested data is not available, the request is forwarded to the next level of the hierarchy. These memories are arranged into a memory hierarchy, where each level serves specific speeds and sizes.

CPU Cache Architecture

One of the simplest ways to improve a computer's performance is to use a faster processor. Every new processor generation introduces higher clock frequencies, improved instruction pipelines, wider execution units, and more sophisticated heterogeneous computing architecture. It is therefore natural to assume that replacing a processor with a faster one should proportionally improve the entire system's performance.

A processor rarely performs computations in isolation. Every instruction executed by the processor depends upon data stored in memory. Instructions themselves must be fetched before they can be decoded and executed, operands must be read before arithmetic operations can begin, and computation results are frequently written back to memory. Whenever the processor needs an instruction or data value, it issues a memory request to main memory and waits for the requested information to return.

If the processor executes instructions faster than the memory can supply them, the processor simply waits.

A processor running at a higher clock frequency completes arithmetic operations in less time, but the access latency of the main memory does not decrease at the same rate. Eventually, the processor spends a considerable portion of its execution time waiting for memory rather than performing useful computation.

Suppose we have designed a processor running at 100 MHz, and the system performs exactly as expected. A few years later, a newer version of the same processor becomes available, capable of operating at 500 MHz. At first glance, upgrading the processor seems like the easiest way to improve system performance. But increasing the processor clock frequency doesn’t proportionally reduce main memory latency. Although the arithmetic units can execute billions of operations per second, they remain idle while waiting for memory.

Why not simply build a faster main memory?

Fast memories exist, but they are significantly more expensive and still cannot eliminate every delay within the memory subsystem. Address decoding, memory buses, controllers, refresh operations, and other supporting circuitry also add latency. A different strategy is to create a small, high-speed memory called cache, designed to be accessed frequently by keeping it close to the processor. The cache acts as an intermediate storage layer between the processor and the main memory. When the processor requests an instruction or a data value, it first checks whether a copy already exists in the cache.

Memory Interleaving

To understand memory interleaving, a useful analogy comes from the firing tactics used by infantry during the 16th, 17th, and 18th centuries. At that time, a musket required a relatively long sequence of operations before it could be fired again: pouring gunpowder, inserting the projectile, ramming it into the barrel, cleaning the air-locks, and preparing to shoot. If every soldier fired and reloaded simultaneously, the battlefield would experience long periods during which no shots could be fired.

Instead, armies organized soldiers into multiple rows. While the front row fired their muskets, the rows behind them were already reloading. Once the front row stepped back to reload, the next prepared row advanced and fired. By alternating between rows, the army maintained a nearly continuous stream of fire even though each individual musket required a significant amount of time to reload.

Memory interleaving applies the same principle

Instead of placing all memory locations in a single memory bank, the memory is divided into multiple independent banks.

The individual access time of each memory bank does not become faster. Instead, every memory address is mapped to a specific memory bank using a deterministic addressing rule.

For a memory system containing N independent banks, this simple mapping distributes consecutive memory addresses across multiple banks.

Bank Number = Address mod Number_of_Banks

Bank Address = Address / Number_of_Banks

Consider a memory subsystem consisting of four independent memory banks.

Address      Bank
0            Bank 0
1            Bank 1
2            Bank 2
3            Bank 3
4            Bank 0
5            Bank 1
6            Bank 2
7            Bank 3

Consecutive addresses never access the same memory bank. Every bank is still busy for exactly the same amount of time as before.

Memory Interleaving is fundamentally an address mapping technique. The memory controller does not dynamically choose a bank for each request, rather the bank is determined directly from the memory address. By distributing consecutive addresses across independent banks, the memory subsystem increases its effective bandwidth while the latency of each individual memory access remains unchanged.

What are SIMM and DIMM?

SIMM (Single In-line Memory Module) was an earlier memory module introduced during the 1980s and widely used in Intel 286, 386, and 486 computer systems. The memory module had electrical contacts on both sides of the printed circuit board (PCB), these contacts were electrically connected together, forming a single set of signal connections.

A standard 72-pin SIMM provided a 32-bit data path, requiring two identical modules to satisfy the Pentium’s 64-bit memory bus. On systems supporting parity or ECC (Error Correction Code), each SIMM typically provided an additional 4 check bits, resulting in a 36-bit module (32-bit data + 4 check bits). When installed as a matched pair, the memory subsystem presented a 64-bit data bus together with 8 parity/ECC bits for error detection.

SIMM is essentially obsolete as a current memory module standard. It was widely used in older systems, especially 30-pin and 72-pin modules, before DIMM became the major form factor.

A Dual In-line Memory Module (DIMM) is the physical memory module that consists of multiple DRAM chips mounted on a printed circuit board (PCB). The processor accesses the DRAM chips through these memory modules, making the DIMM the primary form factor used for installing main memory in desktop computers, workstations, and servers. DIMM has independent 64-bit data channels on both sides, and the gold-plated pins are not interconnected. This design allows data to be transmitted independently at lower voltage and with greater capacity.

Server DIMMs: Hynix 4GB DDR3 ECC Registered RAM

Form Factor: 240-Pin DIMM

Memory Technology: DDR3 SDRAM

Speed: 1333 MHz

Error Correction: ECC Registered (RDIMM)

Desktop DIMMs: Transcend 256MB DDR2 Non-ECC RAM

Form Factor: 240-Pin DIMM

Memory Technology: DDR2 SDRAM

Speed: 667 MHz

Error Correction: Non-ECC

SO-DIMM

The Small Outline DIMM (SO-DIMM) is a compact form factor designed specifically for laptops and mini-PCs. It has a max of 262 pins on DDR5 SO-DIMM.

SO-DIMMs: Transcend 1GB DDR2 RAM

Form Factor: 200-Pin SO-DIMM

Memory Technology: DDR2 SDRAM

Speed: 667 MHz

Error Correction: Non-ECC

Memory Access in Early Processor (386) Systems

Jim Handy’s book on The Cache Memory explains that early 386 processor systems do not communicate directly with the DRAM devices. Instead, an entire communication path lies between the processor and the memory subsystem, consisting of address buffers, data buffers, a shared system bus, and the DRAM connected to its DMA (Direct Memory Access) devices.

These buffers consume a small amount of current from the CPU to drive the address inputs of many DRAMs in the system and to isolate the CPU from main memory. Memory performance is limited by the latency of the entire processor-to-memory system bus, not just by DRAM chips.

Processor clock frequencies and DRAM access times improved over successive generations, but bus buffers and setup and hold timing introduced fixed delays. As processors became faster, these fixed delays consumed more clock cycles, making external memory the primary bottleneck. That’s why Cache memory was introduced: not to eliminate these delays, but to avoid traversing the external memory path for memory access.

Registered DIMMs (RDIMMs)

Registered DIMMs (RDIMMs) are memory modules primarily designed for servers and enterprise workstations, where memory capacity, signal integrity, and long-term reliability matter more than just minimizing access latency.

Registered DIMMs have two operating modes

• Register Clock Driver
• Phase-Locked Loop (PLL)

Register Clock Driver

The register sits between the processor’s memory controller and the SDRAM (Synchronous Dynamic Random Access Memory) devices. Instead of driving every SDRAM chip directly, the memory controller first sends the address, command, and control signals to the register. The register temporarily latches the signals, then forwards them simultaneously to all SDRAM chips on the next clock edge. This additional stage significantly reduces electrical loading on the memory controller, allowing the system to maintain signal integrity by supporting more DIMMs.

Phase-Locked Loop (PLL)

SDRAM devices are synchronous memories, meaning every read and write operation is triggered by the rising or falling edge of a clock signal. For reliable operation, every SDRAM chip on the DIMM must observe the same clock edge at nearly the same instant.

An RDIMM uses a Phase-Locked Loop (PLL), which receives the clock signal from the motherboard, reduces clock jitter, compensates for signal skew, and redistributes a clean, phase-aligned clock to every SDRAM device on the module.

Unlike a UDIMM (Unbuffered Dual In-Line Memory Module), where the memory controller drives the SDRAM directly, an RDIMM adds an extra synchronization stage through the Registered Clock Driver. During one clock cycle, the register captures the incoming address, command, and control signals from the memory controller. On the next rising clock edge, synchronized by the PLL, these signals are forwarded simultaneously to all SDRAM chips on the module.

The additional register stage introduces one extra clock cycle of latency for every memory transaction. However, because every SDRAM device receives synchronized control signals on the same clock edge, setup and hold timing requirements become significantly easier to satisfy.

The one-clock-cycle delay introduced by an RDIMM is not simply a penalty. It results from pipelining the memory interface. By registering the address and command signals before they reach the SDRAM devices, the memory subsystem gains an additional clock period for signal stabilization and synchronization.

Buffered Memory is usually expensive and available at very low capacity. Because additional circuitry is required to design the tiny storage device. Buffers electrically isolate the processor from the rest of the memory subsystem.

A pass-through electrical interface that strengthens and isolates signals between hardware components, similar to an elevator or tunnel, it provides a reliable path for signals to travel but does not retain data.

Static Random Access Memory (SRAM)

In the early days of processor designs, processors spent a considerable amount of time waiting for instructions and data to arrive from the main memory. Although processor clock frequencies continued to increase with each propagation, the latency of the external memory subsystem, including the shared system bus, address decoding, DRAM access, and signal propagation, did not improve at the same rate.

Static Random Access Memory (SRAM) is one of the fastest semiconductor memory technologies available today. Unlike Dynamic Random Access Memory (DRAM), which stores information as electrical charge inside a capacitor, an SRAM cell stores a bit using a small bistable circuit known as a latch. Once a logic value is written into the latch, it maintains the state as long as power is supplied, eliminating the need for periodic refresh operations.

This simple characteristic gives SRAM two significant advantages:

• First, SRAM can be accessed within only a few processor clock cycles, making it significantly faster than DRAM.
• Second, because the latch continuously maintains the stored value, the memory controller does not need to interrupt normal operation to refresh the memory contents.

A typical SRAM cell uses 6 MOSFET transistors.

Four transistors form two cross-coupled CMOS inverters that continuously reinforce one another, creating a stable storage element that can hold either logic 0 or 1.

The other two additional access transistors connect the storage cell to the bit lines during read and write operations.

Although this design provides extremely fast access, it comes at a high cost. Every stored bit requires six transistors, consuming considerably more silicon area than other memory technologies. Also, SRAM chips are expensive to manufacture, occupy a large die area, consume more static power, and provide relatively low storage density.

For these reasons, modern processors contain only a relatively small amount of SRAM, typically a few megabytes, which is reserved for the processor’s L1, L2, and L3 cache memories, where the lowest possible latency has the greatest impact on overall system performance. The remaining system memory uses a much denser and less expensive technology known as Dynamic Random Access Memory (DRAM), which I will explain in the next section.

Dynamic Random Access Memory (DRAM)

SRAM is expensive, physically large, and impractical for constructing gigabytes of main memory. Unlike SRAM, which stores information using a bistable latch, Dynamic Random Access Memory (DRAM) stores each bit as a tiny electrical charge in a microscopic capacitor.

Because each memory cell requires only a single transistor and capacitor, millions or even billions of cells can be integrated onto a single semiconductor chip. This compact design provides significantly higher storage density than SRAM while dramatically reducing the manufacturing cost per bit.

But this design has a caveat: unlike the bistable latch used in SRAM, a capacitor cannot hold electrical charge indefinitely. Leakage currents within the semiconductor gradually discharge the capacitor, eventually causing the stored information to disappear.

For this reason, DRAM is known as Dynamic Random Access Memory. The memory contents remain valid only if every storage cell is periodically refreshed. During a refresh operation, the memory controller reads each row's contents and immediately writes them back, restoring the electrical charge in every capacitor before it decays.

Why is DRAM arranged into rows and columns?

A memory chip containing one billion storage cells would require one billion individual address connections, making the device physically impossible to manufacture. The possible solution to organize DRAM memory cells into a two-dimensional matrix consisting of rows and columns. The memory address now becomes the combination of a row address and column address. This row-and-column organization forms the foundation of modern DRAM architecture. Once the required row has been activated, the memory controller selects the appropriate column to access the requested data.

Row Address Strobe (RAS) and Column Address Strobe (CAS)

The Row Address Strobe (RAS) is asserted first to activate the required row (Word Line), making every memory cell in that row accessible. The Column Address Strobe (CAS) is asserted to select the required column (Bit Line), allowing the processor to read from or write to the desired memory cell.

For example: Addressing a 1024 x 1024 grid (1 MB) directly would require 20 address pins.

By multiplexing the address lines, the chip requires only 10 physical address pins: the memory controller sends 10 bits for the row address as signals RAS, and then sends 10 bits for the column address on the exact same pins as signaling CAS.

Address Decoding using RAS and CAS

A processor views memory as one continuous address space.

For example, a processor may request the following memory location:

Address = 0x12345678

However, a DRAM chip does not interpret this as a single linear address. Instead, the memory controller divides the address into two parts

• Row Address
• Column Address

The memory controller first places the row address on the address bus and asserts the Row Address Strobe (RAS) signal. Inside the DRAM chip, the row decoder receives this address and activates the corresponding Word Line. Once the selected row becomes active, the memory controller places the column address on the same address bus and asserts the Column Address Strobe (CAS) signal. The column decoder then selects the required Bit Line, allowing a read or write operation.

A DRAM array can be thought of as a crossword puzzle. Every memory cell lies at the intersection of exactly one row and one column. As a single letter contributes to both a horizontal and a vertical word, in the same way a single DRAM cell belongs simultaneously to one Word Line and one Bit Line. Identifying both the row and column uniquely determines the location of that memory cell. The processor doesn’t send the row and column simultaneously. Instead, it first sends the row address (RAS), followed by the column address (CAS), reusing the same address pins for both operations.

SDRAM and DDR

Synchronous DRAM

The DRAM architecture described so far is commonly referred to as Asynchronous DRAM. Memory operations such as row activation, column selection, and refresh are initiated through control signals, but they are not synchronized to a continuous system clock. As processor frequencies increased, coordinating these independent operations became increasingly difficult, limiting overall memory performance.

To overcome this limitation, the industry introduced Synchronous Dynamic Random Access Memory (SDRAM). SDRAM synchronizes all memory operations to a common clock signal. This synchronization allows both the processor and memory controller to operate predictably, enabling pipelined memory access and significantly higher operating frequencies. Instead of waiting for one memory operation to completely finish before beginning the next, SDRAM can schedule multiple commands in a deterministic sequence, improving overall memory bandwidth.

Double Data Rate (DDR SDRAM)

In the early days, SDRAM transferred one unit of data using each clock cycle on the rising edge of the clock signal. DDR SDRAM increases memory bandwidth by transferring data on both the rising and falling edges of the same clock.

Prefetch Architecture

Although Double Data Rate (DDR) transfers data on both the rising and falling edges of the system clock, the internal DRAM storage cells themselves do not operate at these high frequencies. Power consumption increases significantly with higher operating frequencies, making it difficult to operate at normal rates.

Instead, modern DDR memories employ a prefetch architecture. Rather than retrieving a single data word from the memory array for every bus transaction, the DRAM core reads multiple consecutive words into a small internal prefetch buffer. These buffered words are then transferred to the processor over the external memory bus at a much higher rate.

DDR2, DDR3, DDR4, and DDR5 do not introduce a new type of memory cell technology. All of them continue to use the same fundamental 1T1C DRAM cell design. Improvements come primarily from advances in the memory interface, signaling techniques, prefetch architecture, and memory controller design.

Low Power DDR (LPDDR)

LPDDR (Low-Power Double Data Rate) is a variant of DDR SDRAM designed to provide high memory bandwidth with lower power consumption. Like conventional DDR, LPDDR is volatile DRAM and is used as working memory. Its lower operating voltage and power-saving modes make it particularly useful in systems where power and physical space are constrained.

LPDDR is widely used in mobile and embedded SoCs, but it is also an important memory technology for FPGA-based systems. Modern FPGAs may interface directly with external LPDDR devices when the internal FPGA block RAM is insufficient for applications requiring larger amounts of high-bandwidth memory. FPGA design may use it as external working memory for large buffers, frame data, lookup tables, and computational workloads that exceed internal FPGA memory capacity.

Flash Memory

Flash Memory is a non-volatile memory technology that can retain stored information even when power is removed. It does not require continuous refresh. Over the last few decades, Flash memory has become increasingly dense and cheaper, enabling significant storage capacity in very small physical form factors. The core design paradigm for underlying key architectural memory systems has been mainly driven by two trends:

Filling more cells into a smaller area.

Storing multiple bits in a single cell using Multi-level cell programming.

What is a Flash Cell?

Flash memory stores data using the threshold voltage (Vth) of a non-volatile memory cell called a flash cell. The basic storage element is a transistor containing a floating gate (FG) and a control gate (CG). The floating gate is electrically isolated from the surrounding structures. An inter-poly oxide layer separates it from the control gate, and a tunnel oxide layer separates it from the underlying semiconductor. These insulating layers electrically isolate the floating gate, allowing electrons stored on it to remain trapped even when the Flash memory is powered off.

The amount of charge stored on the floating gate changes the transistor’s threshold voltage, which is used to represent the stored data.

Bits per Cell: SLC, MLC, and TLC

A flash cell does not have to represent only a single binary value. By programming the cell to different threshold-voltage (Vth) ranges, a single physical cell can represent multiple bits. The number of bits stored in each cell determines how many distinct voltage states the cell must support.

NAND and NOR Flash Block Architecture

NAND and NOR Flash memory both organize large numbers of Flash cells into a two-dimensional array of rows and columns. Each horizontal row is controlled by a common Wordline (WL), while vertical connections provide the electrical paths used to access and read the cells.

The fundamental difference between NAND and NOR Flash lies in how the cells are electrically connected.

NOR Flash Block Design

NOR Flash uses a more parallel array organization. Individual memory cells connect more directly to the array circuitry through contacts, allowing a selected cell to be accessed without passing through a long series of other Flash cells.

This more direct connection design gives NOR Flash its strong random-access performance, making it suitable for applications such as firmware, program storage, and code execution. The trade-off is that parallel architecture requires more contacts and circuit area per memory cell. As a result, NOR Flash generally has a larger effective cell size and a higher cost per stored bit compared with NAND Flash.

NAND Flash Block Design

NAND Flash design provides direct contacts to every individual memory cell. Multiple Flash cells are connected in series to form a NAND string, allowing them to share contacts, bitline, and peripheral circuitry, the cells can be arranged in a dense, highly regular memory array. This reduces the physical area required per stored bit, allowing much larger contiguous storage regions to be integrated onto a single chip.

NAND string shares only a small number of contacts and access circuitry across many Flash cells. From the book Inside NAND Flash Memories by Rino Micheloni • Luca Crippa • Alessia Marelli, it is described that 32 memory cells are placed between select transistors and accessed through only two contacts.

Because the NAND cells are connected in series, the string is accessed through two contacts using the String Select Line (SSL) side toward the bit line and the Ground Select Line (GSL) side toward the source line, rather than requiring a separate bit-line contact on every cell.

The bitline is like a shared electrical path, while the sense amplifier acts as the detector at its end. The Flash cell influences the electrical behavior of that path, and the sense amplifier determines whether the response corresponds to the expected threshold-voltage range.

In NAND Flash, a page is formed by logically grouping the same bit position from all cells connected to a single Wordline.

Multi-level Cell: Each cell stores two bits. The Least-Significant Bits (LSBs) of all cells on the same Wordline form an LSB page, while the Most-Significant Bits (MSBs) form an MSB page. Therefore, one Wordline corresponds to two logical pages.

Triple-level Cell: Each cell stores three bits. The Least-Significant Bits (LSBs), Center-Significant Bits (CSBs), and Most-Significant Bits (MSBs) of all cells on the same Wordline form separate LSB, CSB, and MSB pages. Therefore, one Wordline corresponds to three logical pages.

A Flash block contains many such pages. MLC Flash block contains approximately 256–1024 flash pages, with individual pages typically around 8–16 KB in size.

Solid-State Drive (SSD): A NAND Flash Memory Storage Form-Factor

Solid-State Drives combine multiple NAND Flash chips with a dedicated controller and supporting memory to create a reliable storage system. Compared with traditional magnetic hard drives, SSDs provide much higher random-access I/O performance, consume less static power, and can be built in a much smaller form factor.

Learning from Academic Research

I have thoroughly studied the SSD architecture and reliability mechanisms discussed in this article, using the research paper “Errors in Flash-Memory-Based Solid-State Drives: Analysis, Mitigation, and Recovery” by Yu Cai, Saugata Ghose, Erich F. Haratsch, Yixin Luo, and Onur Mutlu. The paper provides a detailed research findings of SSD error mechanisms, error correction, data recovery, flash management, and lifetime-related design trade-offs.

Manufacturing Process Scaling and Storage Density

One of the major reasons NAND Flash became suitable for mass storage is its highly regular memory-array architecture. Flash cells can be arranged repeatedly in a dense two-dimensional structure of Wordlines and Bitlines. This regular structure allows a large number of memory cells to be integrated within a relatively small silicon area. As semiconductor manufacturing processes continue to improve, the dimensions of Flash cells and surrounding circuitry could also shrink, allowing more cells to be placed on the same NAND die.

Memory Organization inside an SSD

An SSD does not contain a single large Flash memory device. Its storage capacity is distributed across multiple NAND Flash chips, accessed in parallel through independent memory channels.

NAND flash storage is organized in a hierarchy to scale capacity across many independent memory structures. SSD contains multiple NAND flash chips connected to the SSD controller through physical memory channels. Each channel provides an independent communication path, allowing the controller to access multiple NAND devices in parallel.

A flash chip contains one or more dies. A die is an individual piece of silicon that operates independently from other dies. This creates flexibility in another level of parallelism inside the SSD.

Planes

Each die contains one to four planes. A plane contains its own collection of flash blocks and can perform flash operations in parallel with other planes. But planes within the same die share data and control buses, which means operations can be coordinated and pipelined across them.

Blocks and Pages

A plane contains hundreds or thousands of flash blocks. A block is a two-dimensional array of NAND flash cells containing hundreds of rows, around 256–1024 rows. These rows form the underlying physical structure used to organize stored data.

Within a block, data is organized into pages

• A page is typically 8–16 kB.
• Read and program/write operations occur at page granularity.
• A block contains hundreds of pages.
• Pages within a block are numbered sequentially.

Superblocks

The SSD controller firmware extends the physical organization into a larger logical structure. Blocks with the same block ID across multiple planes and/or flash chips can be grouped together as a superblock.

Superpages

Inside a superblock, pages with the same page number/id are grouped into a superpage. By writing a superpage, the SSD can exploit the internal parallelism available across multiple planes and flash chips. Instead of programming only one physical page at a time, the controller can initiate operations across several NAND structures simultaneously.

Memory Channel

A memory channel is an independent communication path between the SSD controller and NAND flash memory. Each channel has its own data bus and control connections, allowing the SSD controller to communicate with flash memory independently and in parallel across multiple channels. The channel typically uses an 8-bit or 16-bit wide bus to transfer both data and flash commands between the SSD controller and the NAND flash memory chips.

Along with the data bus, the channel includes several control signals that indicate how the information on the bus should be interpreted:

• ALE — Address Latch Enable: Indicates that the controller is sending an address.
• CLE — Command Latch Enable: Indicates that the controller is sending flash commands.
• WE — Write Enable: On every rising edge of this signal, the controller indicates that the flash memory should write the data sent on the bus.
• RE — Read Enable: On every rising edge of this signal, the controller indicates that the flash memory should send the next piece of data on the bus.

Each flash memory die has its own Chip Enable (CE) signal, which the controller uses to select a die to communicate over a channel. The die whose CE signal is active reads the requested operation and executes it.

SSD Controller

SSD controller is the central control panel and processing unit for the SSD. The physical layer sits between the host interface and the underlying NAND flash chips to manage the data processing operations. The controller handles host I/O requests and efficiently processes them into the underlying NAND flash memory.

The controller executes firmware to perform these operations, commonly referred to as the Flash Translation Layer (FTL). The FTL is embedded in one or more processors inside the controller. The FTL maintains the connections between the logical address exposed to the host and the physical locations where data is stored in NAND flash memory. The controller uses DRAM to store controller metadata such as logical-to-physical address mappings and to cache frequently accessed SSD pages.

The SSD controller can be viewed as the bridge between the hosts’ logical view of storage and the physical organization of NAND Flash.

List SSD using the macOS diskutil command line

Cylinder Head Sector (CHS) vs Logical Block Addressing (LBA)

Cylinder-Head-Sector is an older addressing scheme developed for magnetic hard disk drives (HDD), where data was physically arranged as tracks on rotating platters.

• Cylinder (Cyl): identifies a set of tracks at the same radial position across the platters.
• Head (Hd): identifies the disk surface being accessed.
• Sector (Sec): identifies the data block on a particular track.

CHS reflects the physical geometry of HDDs. SSDs do not contain cylinders, heads, or rotating platters. However, the MBR partition-table format retains CHS fields for legacy compatibility. Modern storage devices such as SSDs use Logical Block Addressing (LBA). LBA treats the storage medium as a linear sequence of logical sectors.

SSD Filesystem Structure

After identifying the partitions on the SSD, we can inspect the storage structure each partition contains. On the Transcend SSD370S, the inspection shows several ext4 filesystems and one Linux swap area.

Ext4 (Fourth Extended Filesystem) is a filesystem commonly used between the Linux Virtual File System and the generic block layer. It maintains the hierarchy needed to structure files and locate them as needed. In the current SSD370S inspection, several partitions are identified as ext4 filesystems, each copying its own range of logical sectors.

More info: https://en.wikipedia.org/wiki/Ext4

Flash Translation Layer (FTL)

The Flash Translation Layer is part of the SSD controller’s firmware and manages the mapping between the logical addresses used by the host and the physical addresses in the underlying flash memory. This separation allows the SSD controller to move data internally without notifying the host. When existing data is overwritten or moved during operations such as garbage collection, the old data is marked invalid, and new data is written to a new page in a flash block open for writes.

FTL also performs wear leveling, which distributes Program or Erase (P/E) cycles across the NAND blocks. When the current write block becomes full, the controller selects new blocks from the free list by utilizing the wear-leveling algorithm. The block selection mechanism prefers blocks with the lowest number of P/E cycles to minimize wearout-amount.

Wear Leveling

Wear leveling is the FTL mechanism that distributes P/E cycles across the NAND flash blocks instead of repeatedly using the same blocks. The wearout amount is a metric that tracks how much of a storage drive’s total lifespan has been used up due to writing and erasing data.

Data Protection and Recovery in SSDs

Error-Correcting Codes (ECC)

The SSD controller uses Error-Correcting Codes (ECC) to detect and correct raw bit errors in NAND flash memory. When the host writes a page of data, the SSD controller divides the data into smaller chunks. For each chunk, it generates a codeword containing both the original data and additional error-correction information.

The strength of protection offered by ECC is determined by the coding rate, which is the chunk size divided by the codeword size.

Coding Rate = Data Chunk Size / Codeword Size

A higher coding rate means a larger fraction of the codeword is actual data, so it consumes less space. This improves storage efficiency but provides weaker error-correction capability.

A lower coding rate uses more redundancy and therefore provides stronger protection, at the cost of additional storage space.

Overall error-correction capability depends on the ECC algorithm, codeword length, and coding rate. Common ECC algorithms used in SSDs include BCH and LDPC.

More technical detail on these algorithms is out of scope for this article.

BCH: https://en.wikipedia.org/wiki/BCH_code

LDPC: https://en.wikipedia.org/wiki/Low-density_parity-check_code

Contemporary SSDs have ECC engines that can tolerate relatively high raw bit error rates while still reducing the error rate presented to the host to levels required by storage reliability standards.

ECC protection is not enough by itself

Another important step occurs in the read path. Each ECC codeword also contains CRC (Cyclic Redundancy Check) information. This is necessary because an ECC decoder can, in some cases, incorrectly conclude that an error has been successfully corrected even though some errors remain. The SSD controller therefore performs a CRC check after ECC processing to verify that the reconstructed data is actually correct before returning it to the host.

Data Path Protection

ECC primarily protects data against raw bit errors in NAND flash memory. The SSD controller, however, also contains SRAM and DRAM structures through which data and metadata travel, and these memories can themselves experience errors. The paper refers to the mechanism used to protect these paths collectively known as data path protection.

Paper: “Errors in Flash-Memory-Based Solid-State Drives: Analysis, Mitigation, and Recovery” by Yu Cai, Saugata Ghose, Erich F.Haratsch, Yixin Luo, and Onur Mutlu

https://arxiv.org/abs/1711.11427

When data arrives from the host, the controller first places it into the host FIFO buffer (HFIFO). But before writing to the buffer, the data is appended with two forms of protection:

• HFIFO parity protects against errors while the data resides in the SRAM-based FIFO.
• Memory Protection ECC (MPECC) protects the data later while it is stored in DRAM.

1. Write Data Path

When data arrives from the host, the SSD controller adds MPECC and HFIFO parity before placing it in the Host FIFO buffer. HFIFO parity protects the data against SRAM errors while it resides in the FIFO, and MPECC is retained to protect the data later when it is stored in DRAM.

When the data reaches the head of the Host FIFO, the controller uses HFIFO parity to correct any errors, discards the parity information, and sends the data with its MPECC information to the DRAM manager. The DRAM Manager stores the data in DRAM and tracks its location. When the data is ready to be written to NAND, it is retrieved from DRAM, and the controller uses MPECC to detect and correct any DRAM errors, after which the MPECC information is discarded.

The corrected data is then encoded into a NAND ECC codeword, and CRC parity is generated for that codeword. The ECC codeword and its CRC parity are placed in the NAND Flash FIFO Buffer. The controller then checks the codeword using the CRC information. After this check, the data is sent through the NAND Flash Interface and written to NAND flash memory.

2. Read Data Path

When data is read from NAND, the controller first receives the stored ECC codeword together with its CRC parity through the NAND Flash Interface. The CRC information checks the codeword's integrity, and the ECC decoder then processes the NAND data to detect and correct raw bit errors. MPECC protects the data while it resides in DRAM. When the data leaves DRAM, the controller uses MPECC to detect and correct any errors while the data was stored there, then discards the MPECC information.

The recovered data is then passed to the Host FIFO Buffer, where HFIFO parity protects the data against errors while it resides in the SRAM-based FIFO. The controller checks the HFIFO parity, corrects any detected SRAM errors, discards the HFIFO parity, and sends the verified data through the Host interface to the host.

Bad Block Management

A small number of NAND flash blocks may have higher raw bit-error rates because of manufacturing defects or uneven wear-out. The SSD controller removes these blocks from normal use and marks them as bad blocks.

The paper distinguishes two types:

• Original Bad Blocks (OBBs): These are defective due to manufacturing issues.
• Growth Bad Blocks (GBBs): These fail during runtime.

Original Bad Blocks

Few NAND flash blocks are already defective when the flash chip is manufactured. These are called Original Bad Blocks (OBBs). The flash manufacturers perform extensive bad-block scanning to identify these defective blocks. Initially, the NAND blocks are kept in the erased state, where each byte contains 0xFF. The bad-block scanning process identifies the OBB and writes a specific value 0x00, into a designated block-status location inside that block.

Good Block: 0xFF

Bad Block: 0x00

When the SSD controller powers up for the first time, it scans the NAND blocks and reads these block status locations. A block whose status value is not 0xFF is identified as an Original Bad Block and recorded in the controller’s bad-block table. The controller then avoids using these blocks for normal storage.

A small number of reserved blocks are kept for replacement, and the bad-block table maps operations intended for an OBB to an available reserved block.

Replacement is preferably done within the same plane so the SSD can preserve its internal parallelism. The paper highlights that less than 2% of blocks are expected to be original bad blocks.

Growth Bad Blocks

A block can also fail during normal operation. After a program or erase operation, the SSD checks the NAND status information. If the operation fails, the controller marks that block as a Growth Bad Block (GBB). At that point, the SSD preserves the data that was stored in the affected block. The paper describes using superpage-level parity, data is recovered, then the remaining data is copied to another superblock, and then the affected block is retired.

Data Protection and Recovery with RAID Parity

Why does parity exist?

Parity is a mathematical technique that allows a storage array to tolerate a drive failure without losing data. The core operation is XOR (exclusive OR), a bitwise operation that combines data from multiple blocks to generate a parity block. If one data block becomes unavailable, the missing data can be reconstructed from the remaining data blocks and the parity.

Standard RAID (Redundant Arrays of Independent Disks) levels are configurations for combining multiple storage drives into a larger, more reliable storage system. They use three fundamental techniques: striping, mirroring, and parity.

Common levels include RAID 0 for striping, RAID 1 for mirroring, RAID 5 for distributed parity, and RAID 6 for dual parity. RAID levels can also be combined or nested, such as RAID 10, which stripes (RAID 0) data across mirrored (RAID 1) drive groups.

The Storage Networking Industry Association (SNIA) standardizes RAID levels and their associated data formats through the Common Disk Drive Format (DDF) standard. The numbers assigned to RAID levels are simply identifiers, they do not indicate performance, reliability, generation, or hierarchy.

RAID 0

RAID 0, also called a stripe set, distributes data evenly across two or more drives using striping without parity, redundancy, or fault tolerance. Data is divided into fixed-size stripes and written across the drives, allowing multiple drives to be accessed in parallel and potentially increasing read and write throughput. However, because each drive contains part of the data, the failure of any one drive makes the entire RAID 0 array unusable.

RAID 0 is therefore used primarily when performance matters more than redundancy.

RAID 1

RAID 1 stores an exact copy, or mirror, of the same data on two or more drives. This provides redundancy and fault tolerance, since the array can continue operating as long as at least one member drive remains available. However, because each drive stores the same data, there is no striping or parity, and the usable capacity is limited to that of the smallest drive.

RAID 1 is therefore mainly used when reliability is more important than storage capacity or write performance.

In RAID 1, any member drive can serve a read request, allowing read performance to benefit from parallel access depending on the workload. However, write performance remains at the level of a single drive and is limited by the slowest drive when the members have different speeds.

RAID 2

RAID 2 distributes data at the bit level across multiple synchronized drives and uses Hamming-code-based error correction to provide redundancy and detect/correct errors. RAID 2 can transfer data in parallel but typically handles one I/O request at a time. Since modern drives already provide internal error correction, RAID 2 offers little practical benefit and is rarely used today.

RAID 3

RAID 3 distributes data at the byte level across multiple drives and stores parity on a dedicated parity drive. A single data block is spread across all members, an I/O operation generally involves every drive, limiting the ability to service multiple requests simultaneously and requiring synchronized operation. RAID 3 is therefore suited to workloads dominated by large sequential reads and writes. The requirement for synchronized operation added complexity without a significant advantage over other RAID levels, so RAID 3 had little practical adoption. RAID 5 later became a more widely used alternative.

RAID 4

RAID 4 distributes data at the block level across multiple drives and stores parity on a dedicated parity drive. This layout provides good random-read performance, because different data blocks can be read from different drives independently. Random writes can be slower because updates require corresponding parity updates on the single dedicated parity drive.

RAID 5

RAID 5 uses block-level striping with distributed parity, storing parity information across all drives rather than on a dedicated parity drive as in RAID 4. It requires at least three drives and can continue operating after the failure of any one drive, the missing data can be reconstructed from the remaining data and distributed parity.

Because parity is spread across all members, the parity workload is distributed rather than concentrated on a single drive, improving write performance compared with RAID 4. However, parity still adds overhead, so RAID 5 does not provide the same write efficiency as RAID 0. Small writes can also cause write amplification, because updating a small amount of data may require reading the old data and parity, recalculating the parity, and then writing both the updated data and parity.

How does RAID 5 parity work?

RAID 5 uses XOR parity to recover data when one drive fails. XOR produces 0 when two bits are the same and 1 when they differ, and because XOR is its own inverse, a missing data block can be reconstructed from the remaining data blocks and the parity block.

Write penalty with RAID 5 layout?

RAID 5 introduces a write penalty because parity must be calculated and updated whenever data is changed. This additional processing and I/O can reduce write performance, although a hardware RAID controller can help mitigate the impact at the cost of greater complexity and cost.

Another issue with RAID 5 is the longer rebuild time after a drive failure. Rebuilding a large array can take many hours/days, or even longer, and it places additional load on the remaining drives. During the rebuild, the drive is vulnerable to failure, which can cause data loss and create a ripple effect that can cause multiple disks to develop bad sectors or bad blocks. For reliable data protection, RAID should be used alongside a separate backup and disaster-recovery strategy.

I found this very detailed article on how raid parity works. Please check this link from Rossman Group.
https://rossmanngroup.com/technical-reference/how-raid-parity-works

How does Software RAID work in macOS?

Before diving into RAID 6 and dual-parity mathematics, let’s see practically how a RAID setup works in macOS using a few budget SATA SSDs and standard USB enclosures. While enterprise systems use dedicated hardware RAID cards, macOS includes a native software RAID engine called appleRAID.

Hardware Test Bench Setup

To evaluate performance, capacity limits, and spare drive handling, I will show you a demo with three different cases.

• 2 x 256GB EVM SATA III SSDs
• 1 x 64GB Transcend SSD370S
• USB 3.0 Dual Bay SATA Docking Station.
• USB 3.0 HDD SSD SATA I/II/III Docking Station.

Case 1: Symmetrical RAID 0 (2x 256GB SSDs)

1. Stripe Array Creation

I have striped two identical 256GB EVM drives into a single APFS (Apple File System) volume:

List the SSD drives:

Create RAID using the appleRAID diskutil tool

Usable Capacity: ~511.43 GB (256 GB + 256 GB)

3. Key Takeaway

Software striping across dedicated USB ports bumped real-world transfer speeds up to ~339 MB/s write / ~390 MB/s read. Theoretical bandwidth could reach ~900 MB/s combined; real-world throughput is capped near ~400 MB/s due to the single-threaded dd utility queue limit.

Case 2: Asymmetrical RAID 0 (256GB EVM + 64GB Transcend)

1. Stripe Array Creation

This is another test, but a bit different, to test the RAID 0 striping with two SSDs of different storage sizes. To understand how macOS handles mismatched drive capacities.

Usable Capacity: ~127.36 GB (2 x 64 GB)

3. Key Takeaway

There are two issues identified with Asymmetrical RAID 0

• The usable space is truncated to twice the size of the smallest drive (64 GB x 2 = 128 GB), leaving ~192 GB on the EVM SSD drive completely unused.
• Sequential read speed remained high (~ 394 MB/s), but write speed was cut in half (~163 MB/s) because writes were slower due to the legacy flash controller on the Transcend SSD370S.

Case 3: How to utilize all three disks?

Option-1: 3-Disk Striped Array (3-Drive RAID 0)

Usable Capcity: 64 GB x 3 = 191.04 GB (truncated to 3 x smallest drive)

Option-2: 3-Disk Mirrored Array (3-Drive RAID 1)

Usable Capacity: Total size is capped at the smallest drive (63.68 GB)

Option-3: Hot Spare Setup

Create a 2-drive RAID 1 mirror using the two 256GB EVM drives, and assign the 64GB Transcend drive as a hot spare.

Usable Capacity: ~255.72 GB (2 x EVM 256 GB drives)

Key Takeaway

macOS enforces strict hot spare sizing: A spare drive must be >= the active member slice size. Any extra storage on a larger spare drive is simply left unallocated.

RAID 6

While RAID 5 can only survive a single drive failure using simple XOR parity (P), RAID 6 introduces a second independent parity block (Q) per stripe, allowing the array to tolerate two simultaneous drive failures without data loss. This two-failure tolerance is increasingly critical for modern high-capacity drive arrays.

With a RAID 5 rebuild, the intense I/O workload significantly raises the possibility of an Unrecoverable Read Error (URE), or a second disk failure can cause a cascading failure on the remaining drives, resulting in catastrophic array loss.

Dual Parity in RAID 6

RAID 6 requires a minimum of 4 drives (2 data drives, 2 parity drives), creating a net usable capacity of (N — 2) x Drive Size.

To reconstruct two arbitrary lost data blocks (D1 and D2), the array generates two parity syndromes (P and Q) that are mathematically independent.

P Parity (Standard Bitwise XOR):

P = D0 ⊕ D1 ⊕ D2 ⊕ ….. ⊕ Dn-1

Q Parity (Reed-Solomon Code over GF(2⁸))

Q = (g⁰ x D0) ⊕ (g¹ x D1) ⊕ (g² x D2) ⊕ (g³ x D3) ⊕ …. ⊕ (g^n-1 . Dn-1)

Where g = 0x02 is the primitive element of GF(2⁸), and the exponent corresponds directly to the drive slot index i. Each data block is scaled by a distinct coefficient power g^i. P and Q form a system of two linearly independent equations with two unknowns, guaranteeing exact reconstruction through matrix inversion over GF(2⁸).

Superpage-Level Parity

As I have briefly described in the previous section, SSDs use RAID-like parity to protect against failures that ECC cannot correct, such as a failure within a NAND chip or plane. A superpage spans across multiple dies and planes, with one die dedicated as a parity die while the remaining dies store user data. After writing the data, the SSD controller XORs the corresponding data from each plane and stores the resulting parity in the parity die. Thus, the parity is generated at the superpage level, while ECC continues to protect individual codewords within each page.

Superpage-level parity is invoked when an access to a logical block (LB) fails, either because the data resides in an undetected growth bad block (GBB) or because one or more ECC codewords within the LB cannot be corrected. The controller then reads the corresponding LBs from the other dies in the same plane, including the parity LB, and XORs them to reconstruct the failed LB.

Because this reconstruction requires all the remaining LBs to be read correctly, recovery fails when more than one LB in the superpage is unavailable.

In this way, dedicating one die for parity allows the SSD to tolerate complete loss of the superpage data in one die without data loss during an LB access. The organization is similar in principle to RAID 4, but the parity is implemented internally across NAND dies within a single SSD rather than across separate storage drives.