Section 5: Advanced CPU Concepts

The Big Picture: Doing More Per Tick

These are the tricks that let a modern CPU finish more than one instruction per clock cycle (one "tick" of the CPU's clock). They do not change the basic Fetch → Decode → Execute → Memory → Writeback cycle from Section 4. They just run many copies of it at the same time, in clever ways.

5.1 Pipelining (Overlapping Stages)

The problem: Suppose the CPU waited for one instruction to finish all five stages before starting the next. Most of the hardware would sit idle most of the time. While instruction #1 is in Execute, the Fetch hardware has nothing to do.

The solution: Overlap the stages, like an assembly line or a laundry day. While the first load is in the dryer, the second load is already in the washer. Here, while instruction #1 is being decoded, instruction #2 can already be fetched. While #1 executes, #2 decodes and #3 fetches.

Without pipelining (one instruction fully finishes before the next starts):
Instr 1: [F][D][E][M][W]
Instr 2:                [F][D][E][M][W]
Instr 3:                               [F][D][E][M][W]

With pipelining (stages overlap):
Instr 1: [F][D][E][M][W]
Instr 2:    [F][D][E][M][W]
Instr 3:       [F][D][E][M][W]
Instr 4:          [F][D][E][M][W]

(F = Fetch, D = Decode, E = Execute, M = Memory, W = Writeback.)

Result: Once the pipeline is full, the CPU can finish about one instruction per clock cycle, even though each instruction still takes five cycles to pass through all its stages. Modern CPUs go further with much deeper pipelines (14–20+ stages). Each instruction takes a little longer, but far more get done overall.

The catch: pipeline hazards. Overlapping works smoothly only if instructions are independent. If instruction #2 needs the result of #1 before it can run, or if #1 is a branch (a point where the program may jump somewhere else) that might change the Program Counter, the pipeline can stall or have to throw away work it started wrongly. Branch prediction and out-of-order execution exist to reduce exactly this problem.

5.2 Branch Prediction

The problem: A pipeline works by fetching upcoming instructions ahead of time. But at an IF statement or a loop condition (a branch) the CPU does not yet know which way the program will go. The pipeline still needs to be fetching something.

The solution: The CPU guesses. A branch predictor looks at how that same branch behaved in the past (and at general patterns) to guess which path is more likely. It keeps fetching and decoding down that guessed path.

if (x > 0) {
    A();   ← predictor guesses this branch is usually taken
} else {
    B();
}

Modern predictors are very accurate (often well above 90%), because most branches in real code (like loop conditions) behave the same way nearly every time.

5.3 Out-of-Order Execution

The problem: Instructions are written in a set order. But running them strictly in that order wastes time whenever one instruction is stuck (for example, waiting on a slow memory load) while a completely unrelated later instruction is ready.

The solution: The CPU may run instructions in a different order than the program lists them, as long as the final result is the same as running them in order. Think of a cook who puts the pasta on to boil, and while waiting chops the vegetables instead of staring at the pot.

Program order:
1. LOAD  R1, [memory]     ← slow, waiting on RAM
2. ADD   R2, R3, R4        ← doesn't depend on R1, ready immediately
3. MUL   R1, R1, 2         ← depends on instruction 1, must wait

Out-of-order execution:
Instruction 2 runs first (while instruction 1 is still waiting on memory)
Instruction 1 completes
Instruction 3 runs (now that R1 is ready)

The CPU keeps internal notes, called a reorder buffer, to remember the original program order. Results are still committed (made official, written back) in the right order. Software never notices the shuffling. It only notices the speed-up.

5.4 Speculative Execution

What it is: A bolder mix of branch prediction and out-of-order execution. The CPU does not just guess which branch will be taken. It actually starts running the instructions down the guessed path before it knows the guess is right.

Branch predicted "taken" → CPU starts executing that path speculatively
   ├── Prediction correct   → keep the results, huge time saved
   └── Prediction incorrect → throw away all speculative work, restart

Why it is powerful: In a deeply pipelined, out-of-order CPU there is often spare execution capacity. Speculative execution uses that spare capacity for "probably useful" work instead of leaving it idle.

Why it is also famous for a bad reason: Speculative execution is the root cause of the Spectre and Meltdown CPU security flaws (2018). An attacker could watch tiny side effects (cache timing) of speculatively run instructions and use them to work out secret data, even though the speculative results themselves had been correctly thrown away. It is a good real-world example of a speed trick creating a surprise security problem.

5.5 Instruction-Level Parallelism (ILP)

What it is: The umbrella name for everything in this section. Pipelining, branch prediction, out-of-order execution and speculative execution are all ways to get parallelism from inside a single instruction stream. One core works on several instructions at different stages at the same time.

This is a different kind of parallelism from multi-core (Section 2), which runs completely separate instruction streams side by side:

Modern CPU speed comes from both working together: many cores (thread-level), each one squeezing as much instruction-level parallelism as it can out of its code.

Summary