Section 4: System Calls

The Big Picture: The Bank Counter

In a bank, customers wait on one side of a glass wall. The staff and the vault are on the other side. A customer cannot reach over and take money. Instead the customer hands a slip through a small slot. A clerk behind the glass reads the slip, checks the customer's identity, fetches the money and slides it back.

A system call is that slot in the glass. Your program stands on the user-mode side. Windows and the hardware are on the kernel-mode side. The system call is the only approved way to pass a request through the glass.

 USER MODE (ring 3)        │glass│        KERNEL MODE (ring 0)
 Your program              │     │        Windows kernel, drivers
  "please read this file"  ─slot─►        checks, then does the work
                           ◄─slot─        result slides back

4.1 The Transition from User Mode to Kernel Mode

What it is: A system call (or syscall) is a request from user-mode code that asks the kernel to do something only the kernel is allowed to do. Reading a disk, creating a process and allocating memory all need one.

Why it is needed (a reminder from Chapter 2 and Chapter 7):

So the CPU needs a controlled way to switch from ring 3 to ring 0. Not "jump anywhere in the kernel." A jump to one fixed, safe entry point chosen by Windows when it started, which a program cannot change.

What happens during the switch:

Why it costs a little time. Each switch needs saving state and checking. A system call is much cheaper than a disk read, but costs more than a normal function call. This is why good programs avoid making tiny system calls thousands of times when one larger call will do. For example, reading a file in big blocks beats reading one byte at a time.

You can feel this cost: in PowerShell, Measure-Command { 1..100000 | ForEach-Object { } } against a loop that makes lots of file calls will show a big difference. (The exact times depend on your PC.)

Not every API call makes a system call. Many Win32 functions are handled entirely in user mode (for example some string and math helpers, or GetLastError). Only the ones that need kernel services go down.

4.2 The System Service Dispatch Table (SSDT)

What it is: Once the CPU enters the kernel, the kernel has to answer: "Which service did the program ask for?" The program put a service number in a register. The kernel uses that number to look up the right function in a table.

That table is called the System Service Dispatch Table, or SSDT. (In some places you may see the System Service Descriptor Table, which is the same idea.) Think of it as a phone book: the service number is the line number, and the entry holds where the matching kernel code is.

 service number   function that handles it (inside ntoskrnl.exe)
 ───────────────  ─────────────────────────────────────────────
      ...                   ...
      55      ────►   NtCreateFile (kernel version)
      56      ────►   (another service)
      ...                   ...

(The numbers above are made up for illustration. Real numbers differ between Windows versions, which is exactly why the Native API is not a safe thing for programs to depend on.)

The path in full:

 NtCreateFile (in ntdll.dll, user mode)
      │ puts service number in a register
      ▼
   syscall  ───── CPU switches to ring 0 ─────►  kernel entry point
                                                      │
                                                      ▼
                                              look up number in the
                                              dispatch table (SSDT)
                                                      │
                                                      ▼
                                         kernel's real NtCreateFile code

Important details (kept simple):

Hedge: The SSDT is an internal structure. Its layout is not documented and has changed over time. For this course, you only need to know what it is for.

Before the lookup, the kernel checks. Before running any service, the kernel verifies that pointers given by the program point into the program's own memory. This stops a program from tricking the kernel into reading or writing the kernel's memory.

4.3 The syscall and sysenter Instructions

What it is: The CPU has special instructions built for the user-to-kernel jump. They are much faster than the older method.

For modern 64-bit Windows, you only meet syscall. A matching instruction sysret returns to user mode.

How syscall works:

In plain words: the CPU is the gatekeeper. It will only open the gate for the syscall instruction, and it always sends you to the same place.

              ┌────────────────────────────────────────────┐
  CPU core    │  ring 3: your code  ──► executes "syscall" │
  (see Ch. 2) │                                │           │
              │            ring switch ◄───────┘           │
              │  ring 0: kernel entry ──► SSDT ──► service │
              │                                │           │
              │  "sysret" ──► back to ring 3 ◄─┘           │
              └────────────────────────────────────────────┘

What NtCreateFile looks like in machine code. On 64-bit Windows 10 and 11, the stub is only a few instructions. Roughly:

mov r10, rcx           ; copy the first input to a safe register
mov eax, <number>      ; the service number for NtCreateFile
syscall                ; CPU goes to ring 0
ret                    ; back in user mode with the result

(The <number> depends on the Windows version. Some Windows builds also add a small check around syscall. The idea is the same.) A program called a debugger can show this. You do not need one in this course.

Figure 23.1 — A real CPU die. The ring change from user mode to kernel mode, and the syscall instruction that triggers it, are built into every core you see here. Photo: JmsDoug and Fritzchens Fritz, CC0, via Wikimedia Commons.

4.4 What the Kernel Does Next

What it is: After the SSDT lookup, the kernel runs the real service inside ntoskrnl.exe, the Windows kernel file. Most services are carried out by a group of kernel managers called the Executive. (Chapter 7 introduces them.) Each manager owns one job:

A kernel service always works the same way:

1. Validate the inputs and the handle.
2. Check security (does this user have the right?).
3. Do the work, perhaps asking a driver.
4. Return an NTSTATUS (Section 3.4).

4.5 Seeing System Calls on Your PC (Safe)

You cannot see raw syscall instructions without a debugger, but you can watch their effects:

In the Task Manager CPU graph, light blue is the total and dark blue is the kernel part. A program that makes a great many system calls pushes up the dark portion.

Windows-specific tip. When an antivirus or security product watches a PC, it often watches these crossings from user mode into the kernel, because every important action must pass them. It is also why Microsoft protects the kernel tables so strictly.

Summary