The Big Picture: Many Tenants, Locked Doors
Your PC runs programs written by thousands of different people. Some are careless. A few are hostile. You still want to browse the web, run a game and keep your tax file safe, all on the same machine, at the same time.
The OS makes this possible with isolation: each program lives in its own sealed space and may only act through doors the OS controls. Think of an apartment building with locked flats and one doorman who checks every request.
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Program A│ │ Program B│ │ Program C│ USER MODE
│ own memory│ │ own memory│ │ own memory│ (limited power)
└─────┬────┘ └─────┬────┘ └─────┬────┘
│ │ │
═════▼════════════▼════════════▼═════ <-- the only doors:
system calls, checked by the OS
═══════════════════════════════════════
┌───────────────────────────────────────┐
│ KERNEL (full power) │ KERNEL MODE
├───────────────────────────────────────┤
│ HARDWARE │
└───────────────────────────────────────┘
Security has two sides: isolation (keep programs apart) and protection (check who is allowed to do what). Both rely on hardware help from the CPU and MMU (Chapters 2 and 3).
6.1 User/Kernel Separation
What it is: The CPU runs code in two main modes. Kernel mode can do anything. User mode is restricted. Programs run in user mode. Only the OS core runs in kernel mode. Section 7 explains this in depth. Here is why it matters for security.
In user mode, a program cannot:
- Talk to hardware directly
- Change the page tables or the MMU settings
- Turn interrupts off
- Run special CPU instructions that control the system
If it tries, the CPU raises an exception and the OS stops the program. The program can only ask the OS to act, through a system call, and the OS checks the request first.
A simple picture: a user-mode program is a visitor at a bank counter behind glass. The kernel is the bank staff behind the glass with access to the vault. The visitor can fill in a form (system call), and the staff decide whether to act.
The hardware makes it real. This is not just a polite rule. The CPU itself refuses forbidden instructions in user mode (Chapter 2: privilege rings). Software alone could not hold the line, because a hostile program could simply ignore it.
6.2 Process Isolation
What it is: Each process runs in its own sandbox: its own address space, its own handles, its own identity. One process cannot reach into another.
How the OS builds the walls:
| Wall | How it works |
|---|---|
| Private memory | Separate page tables (Section 4): process A's address 0x1000 and B's 0x1000 are different real memory |
| Private handles | A handle number (like 4) means something only inside its own process |
| Process crash containment | If A crashes, the OS cleans up A. B keeps running |
| Separate security identity | Every process carries a token (a Windows badge) saying who it runs as |
Doors that are allowed. Processes sometimes must cooperate, so the OS provides controlled doors, called inter-process communication (IPC): pipes, shared memory (only if both agree), sockets, messages. The OS checks permission before connecting two processes.
Going further: sandboxes. Modern programs shrink their own power. A web browser runs each tab, or group of tabs, in a separate low-privilege process, so a malicious web page that breaks one tab still cannot read your files. Windows offers features such as AppContainer (a locked-down environment for Store apps) and Windows Sandbox (a throw-away mini-PC window, on some Windows editions). A virtual machine goes even further: a whole separate OS in a box (think of it as the same isolation idea applied to a whole computer; later parts of the course return to it).
Not perfect. Isolation can have bugs. In 2018 the Spectre and Meltdown CPU flaws let one program guess at other memory by measuring timing, which showed that even hardware-backed walls can leak. The fix was new OS and CPU updates. The lesson: isolation is a strong defence, not a magic shield.
6.3 Memory Access Protection
What it is: The detailed checks on every memory access. These are the same page-table bits from Section 4.3, seen now as a security tool.
Every time a program reads or writes memory, the MMU checks:
Program touches virtual address V
│
▼
Is V in a page the process owns? no ──► fault → OS ends program
│ yes
Is this access allowed (R/W/X)? no ──► fault → OS ends program
│ yes
Is it a kernel page, and are we in user mode?
yes ─► fault → OS ends program
│ no
▼
Access proceeds to real RAM
This check happens in hardware, on every access, at full speed. A program cannot dodge it.
Extra protections layered on top:
| Protection | What it does |
|---|---|
| NX / DEP | Data pages cannot run as code |
| ASLR | Random layout stops attackers from knowing addresses |
| Guard pages | An unusable page placed at the end of a stack so overflows trigger a fault |
| SMEP / SMAP (on modern CPUs) | Kernel mode cannot accidentally run or read user-supplied memory, which blocks some kernel attacks |
| Kernel DMA Protection | The IOMMU stops plug-in devices reading RAM they should not (Chapter 4) |
Why the kernel's memory is special. The kernel half of every process's address space is mapped, but marked "supervisor only". User code touching it faults. That is why the kernel can be fast (no switching of memory maps for every system call) and still safe.
6.4 Resource Quotas
What it is: Quotas are limits. Isolation keeps programs from reading each other. Quotas keep them from starving each other. Without limits, a runaway program could use every byte of RAM, fill the disk, or open millions of files, and take the whole PC down.
| Resource | Typical limit the OS can apply |
|---|---|
| CPU time | Priorities and fair sharing; some systems cap CPU share for a group |
| Memory | A cap on how much a process (or group) may commit |
| Open handles / files | A limit on how many a process can open |
| Disk space | Disk quotas per user on NTFS |
| Network | Bandwidth limits on some systems |
| Number of processes | Stops a "fork bomb" (a program that copies itself endlessly) |
On Windows, a Job object lets a group of processes share one set of limits. Linux uses cgroups (control groups) for the same job. Containers (a topic for later chapters) are built from these tools plus isolation.
Kernel memory and "pools". The OS also limits how much memory drivers can take for themselves, because a driver that leaks memory could eat the whole system over days.
Fairness without hard limits. Even without quotas, the scheduler (Section 3) makes sure each thread gets some CPU, and the memory manager (Section 4) takes pages from idle programs first.
6.5 Access Control
What it is: Access control decides who may do what to which thing. Isolation protects processes from each other. Access control protects files, devices and settings from users and programs that should not touch them.
Three questions every access check answers:
| Question | Windows word |
|---|---|
| Who is asking? | The user account and groups, stored in the process token |
| What are they trying to touch? | A securable object: a file, folder, registry key, process, etc. |
| What do they want to do? | Read, write, run, delete, change permissions... |
The object carries an ACL (Access Control List), a list of rules such as "Alice: read and write; Everyone: read only; Guests: no access". The kernel compares the token with the ACL, and says yes or no.
Process (token: user = Alice) ──asks──► "Write to report.docx"
│
Kernel reads the file's ACL:
Alice → Modify allowed? ──► YES, proceed
Bob → Read only ──► NO, "Access denied"
Authentication vs authorization. Two words people mix up:
- Authentication: proving who you are (password, PIN, fingerprint, face).
- Authorization: deciding what you may do once the OS knows who you are.
Least privilege. A golden rule: give every user and program only the power it needs. Windows helps with:
| Feature | What it does |
|---|---|
| Standard vs Administrator accounts | Daily work uses a normal account |
| UAC (User Account Control) | The pop-up asking "Do you want to allow this app to make changes?" It raises a program to administrator power only with your consent |
| Windows Defender / Security features | Extra checks for malware |
| Secure Boot | Only trusted code starts the OS (Chapter 5) |
On Windows: Right-click any file → Properties → Security to see its ACL. In Command Prompt, whoami /groups shows the groups in your token, and icacls "C:\Windows" lists the permissions on a folder. Try opening a system file for editing: the OS says "Access denied" because your token fails the ACL check.
Summary
| Topic | Remember this |
|---|---|
| User/kernel separation | Programs run in restricted user mode; only the kernel has full power; the CPU enforces it |
| Process isolation | Private address space, handles and token per process; a crash stays inside one process |
| Memory access protection | MMU checks every access: ownership, read/write/execute, user vs kernel; NX, ASLR and guard pages add more |
| Resource quotas | Limits on CPU, memory, handles, disk and process count stop one program from starving the rest |
| Access control | Tokens plus ACLs decide who may do what; UAC and least privilege reduce risk |