Section 9: Thread Pools

The Big Picture: A Taxi Rank

Imagine a town where, for every passenger, a taxi company must build a brand-new taxi, drive the passenger, then scrap the taxi. Crazy! A smarter company keeps a row of taxis waiting at a rank. A passenger arrives, takes the first free taxi, and when the ride ends, the taxi returns to the rank for the next passenger.

A thread pool is that taxi rank. The taxis are threads. The passengers are tasks (small jobs).

   Tasks arrive           QUEUE              POOL OF THREADS
   ─────────────►     ┌──────────┐         ┌───────────────┐
   (small jobs)       │ job job  │ ──────► │ thread 1 busy │
                      │ job job  │ ──────► │ thread 2 busy │
                      │ job      │ ──────► │ thread 3 idle │
                      └──────────┘         │ thread 4 idle │
                                           └───────────────┘
                      Threads are reused. They are not destroyed after each job.

9.1 What a Thread Pool Is

What it is: A thread pool is a ready-made group of worker threads that take jobs from a queue, run them one by one, then go back for more. Your program says what to do. The pool decides which thread does it and when.

How it works:

 1. The program SUBMITS a task (a function plus its data) to the pool.
 2. The task goes into a QUEUE.
 3. A free worker thread picks the task, runs it.
 4. When finished, the worker does NOT exit. It looks at the queue again.
 5. If the queue is empty, the worker sleeps (state Waiting, no CPU use).
 6. If work piles up, the pool may add threads (up to a limit).
    If threads stay idle for a long time, the pool may remove some.

Terms:

Typical uses:

• A web server handling many short requests
• Asynchronous I/O completions: "the file read finished, run this code"
• Timers: "run this function in 5 seconds"
• Splitting a big job into many small parts (for example, processing 1,000 images)

9.2 Why Thread Pools Exist (Reuse Overhead)

What it is: The pool exists because creating and destroying a thread for each small job is wasteful.

What one CreateThread costs (see Section 2.3):

If a task only takes 0.1 ms, but creating the thread takes 0.1 ms or more, then half your time is overhead. With a pool, the creation cost is paid once and shared by thousands of tasks.

The bigger danger: too many threads.

 A server with 1,000 simultaneous requests, one new thread each:
   - 1,000 stacks of memory (about 1 GB of address space)
   - Only 8 logical processors available
   - 1,000 threads fight for 8 cores ► heavy context switching (Section 6)
   - Everything gets slower for everyone

A pool limits the number of active threads (near the number of cores for CPU-heavy work), so extra tasks wait their turn in the queue instead of overwhelming the CPU.

Benefits in a table:

Trade-offs and cautions:

9.3 Windows Thread Pool API

What it is: Windows includes a built-in pool for every process. Since Windows Vista it is the modern thread pool API, run by ntdll and the kernel. You do not have to build your own.

Main object types (all begin with Create...Threadpool...):

Tiny example (shape only):

VOID CALLBACK MyTask(PTP_CALLBACK_INSTANCE inst, PVOID ctx, PTP_WORK work) {
    // runs on a pool thread
}
 
PTP_WORK w = CreateThreadpoolWork(MyTask, NULL, NULL);
SubmitThreadpoolWork(w);               // queue it
WaitForThreadpoolWorkCallbacks(w, FALSE);   // wait until it is done
CloseThreadpoolWork(w);

The old API. Before Vista there was QueueUserWorkItem. It still works, and uses the same pool underneath. Many programs still use it.

Higher-level wrappers you may see:

How the pool sizes itself: Windows watches how busy the pool is. If tasks wait too long and threads are all busy, it can add threads gradually. If threads are idle for a while, it removes them. For CPU-heavy work, you should choose a size near the number of logical processors. For I/O-heavy work, a larger pool makes sense.

Rule of thumb:

See it: In Task Manager or Process Explorer, a busy program often shows several threads whose start address is in ntdll.dll (names like TppWorkerThread). These are the pool's worker threads. Programs such as browsers, antivirus and File Explorer have dozens.

Summary