Section 3: CPU Memory Hierarchy

The Big Picture: The Problem This Solves

RAM is fast compared to storage, but it is still far too slow for a modern CPU core. A CPU at 4 GHz can run an instruction about every 0.25 nanoseconds (a nanosecond is one billionth of a second). A round trip to RAM takes about 60–100 nanoseconds. If every instruction had to wait for RAM, the CPU would spend most of its life doing nothing.

The fix: put a hierarchy (a ladder) of smaller, faster memories between the CPU's execution units and RAM.

Analogy: You are cooking. Salt is in your hand (registers). Spices are on the counter (L1). Other ingredients are in the nearby cupboard (L2 and L3). The rest is in the supermarket (RAM). You keep what you use often close by.

┌─────────────────────────────────────────────────────────┐
│  REGISTERS        a few dozen        ~0.25 ns   (fastest)│
│  L1 CACHE         32–64 KB           ~1 ns               │
│  L2 CACHE         256 KB–1 MB        ~4 ns                │
│  L3 CACHE         4–16 MB (shared)   ~15 ns               │
│  MAIN RAM         8–64+ GB           ~60–100 ns  (slowest)│
└─────────────────────────────────────────────────────────┘
      SMALLER & FASTER  ◄──────────────►  BIGGER & SLOWER

3.1 Registers

What they are: A small set of storage spots built directly into the CPU core. They are the fastest storage in a computer.

• Typical count: a few dozen general-purpose registers (the exact number depends on the design)
• Size: each holds one value, 32 bits or 64 bits depending on the CPU mode (a bit is a single 0 or 1)
• Speed: used in the same cycle as the instruction that needs them, so effectively instant

Registers hold what the CPU is working on right now: the numbers for the current instruction, loop counters and pointers. Every arithmetic operation reads from and writes to registers.

3.2 L1 Cache

What it is: The first, smallest and fastest cache (a small, fast memory that keeps copies of data you will probably need again). L1 is private to each core.

• Size: typically 32–64 KB, often split into an L1 instruction cache (for fetched instructions) and an L1 data cache (for data being used)
• Speed: roughly 4 CPU cycles to access. A few times slower than a register, but far faster than RAM
• Scope: one per core. Core 0's L1 cache is invisible to Core 1

3.3 L2 Cache

What it is: A larger, slightly slower cache, also usually private to each core.

• Size: typically 256 KB to 1 MB per core
• Speed: roughly 10–12 cycles
• Purpose: catches data that did not fit in L1, before the CPU has to go to L3 or RAM

3.4 L3 Cache

What it is: A large cache shared by all cores on the chip.

• Size: typically 4–16 MB (some server and workstation CPUs have much more, tens of MB)
• Speed: roughly 30–40 cycles. Much slower than L1/L2, still far faster than RAM
• Purpose: a shared pool. If one core loads data that another core also needs, it does not need to go to RAM again. It also helps threads that work together share data.

What it looks like: Look back at the labelled chip in Figure 2.3 (Section 2). Cache takes a lot of space on the chip, which shows how important it is. The green blocks are the L2 cache (2 MB per P-core) and the dark red blocks are the L3 cache (3 MB per P-core). The E-cores have their own orange L3 cache and an L2 cache of 4 MB per cluster of 4 cores.

3.5 Main RAM

What it is: The big, slower pool covered in Chapter 3. It holds gigabytes, but is hundreds of times slower than L1 cache.

• Size: typically 8–64+ GB in modern systems
• Speed: roughly 200+ CPU cycles. This is the "worst case" the cache ladder exists to avoid

3.6 Cache Hierarchy Diagram — How a Read Actually Happens

When the CPU needs data, it checks each level in order, fastest first, and stops as soon as it finds it:

CPU needs data
      │
      ▼
   L1 Cache?  ──── HIT ────► use it (fastest path)
      │ MISS
      ▼
   L2 Cache?  ──── HIT ────► use it, copy into L1
      │ MISS
      ▼
   L3 Cache?  ──── HIT ────► use it, copy into L2/L1
      │ MISS
      ▼
   Main RAM   ──────────────► fetch it, copy into L3/L2/L1

Finding the data in a cache is a cache hit. Not finding it, and having to check the next level, is a cache miss. Missing every cache and going all the way to RAM is the slowest, most expensive result.

3.7 Why Caching Matters (The Speed Differential)

The whole idea works because of locality of reference. In plain words: programs tend to reuse the same data, and data stored close by, again and again in short bursts (loops, repeated function calls, reading an array in order). Caches keep recently used and nearby data close to the CPU.

To make the scale feel real, here is the same comparison scaled up to everyday distances:

Key idea: A good program is not only "correct". It is arranged so the data it needs keeps landing in cache instead of falling back to RAM. This is why two programs that do the same job can run at very different speeds.

Summary

Hit = found in cache (fast). Miss = not found, go down a level (slow).