Section 4: The Rendering Pipeline

The Big Picture: From Shapes to Coloured Dots

A 3D game scene is not stored as a picture. It is stored as a list of shapes in 3D space: a character, a wall, a car. Your screen, however, is a flat grid of coloured dots. Something has to turn "3D shapes" into "2D coloured dots".

That something is the rendering pipeline: a fixed series of steps inside the GPU. Data goes in at one end (shapes), and a finished picture comes out at the other end. Like a car factory line, each station does one job, and many cars are on the line at once.

Everything is built from triangles. Why triangles? Three points always form a flat surface, so the GPU can handle every triangle in the same simple way. A smooth-looking character is really thousands of tiny triangles joined together.

Here is the whole pipeline in one picture. The rest of the section explains each station.

 3D shapes (triangles made of vertices)
        │
        ▼
 ┌────────────────────────────────────────┐
 │ 1. Vertex shader                       │  moves each corner point to
 │                                        │  the right place on the screen
 └──────────────────┬─────────────────────┘
                    ▼
 ┌────────────────────────────────────────┐
 │ 2. Primitive assembly                  │  joins corner points into
 │                                        │  triangles
 └──────────────────┬─────────────────────┘
                    ▼
 ┌────────────────────────────────────────┐
 │ 3. Rasterization                       │  works out which screen pixels
 │                                        │  each triangle covers
 └──────────────────┬─────────────────────┘
                    ▼
 ┌────────────────────────────────────────┐
 │ 4. Pixel (fragment) shader             │  works out the colour of
 │                                        │  each pixel
 └──────────────────┬─────────────────────┘
                    ▼
 ┌────────────────────────────────────────┐
 │ 5. Depth / stencil test                │  throws away pixels hidden
 │                                        │  behind others
 └──────────────────┬─────────────────────┘
                    ▼
 ┌────────────────────────────────────────┐
 │ 6. Blending and output merger          │  mixes colours, writes the
 │                                        │  pixel to the render target
 └──────────────────┬─────────────────────┘
                    ▼
 The finished picture in the frame buffer (VRAM)

Two words you need:

• A shader is a small program that runs on the GPU. The GPU runs the same shader on thousands of vertices or pixels at once (Section 1).
• A render target is the block of memory the GPU is drawing into. For the final picture, it is the back buffer (Section 7).

4.1 Vertex Processing

What it is: A vertex (plural vertices) is a corner point of a shape: it has a position in 3D space, and often extra data such as a colour or a texture position. Three vertices make one triangle.

The vertex shader runs once for every vertex. Its main job is to answer: "Where does this corner appear on the screen?" The maths takes into account the object's own position and rotation, and where the camera is looking from. Think of it as taking a photograph of the 3D scene: each point of the world must be placed on the flat photo.

After this, the GPU does primitive assembly: it joins the vertices back into triangles (a triangle is one kind of primitive, meaning a basic shape). It also removes triangles that are completely outside the picture, since nobody can see them.

4.2 Rasterization

What it is: The screen is a grid of pixels, but a triangle is a smooth shape. Rasterization decides which pixels each triangle covers. The fixed-function rasterizer block (Section 1) does this in hardware.

   A triangle on a pixel grid             The pixels it covers

   . . . . . . . . . .                    . . . . . . . . . .
   . . . . . /\ . . . .                   . . . . . # . . . .
   . . . . / . \ . . .                    . . . . # # # . . .
   . . . / . . . \ . .                    . . . # # # # # . .
   . . / . . . . . \ .                    . . # # # # # # # .
   . ./____________\ .                    . # # # # # # # # #

Each covered pixel becomes a fragment: a "candidate pixel" that might end up on screen. It carries information such as its position, its depth (distance from the camera) and values smoothly interpolated (blended) from the three corners. For example, a triangle with a red corner and a blue corner gives fragments that shade from red to blue.

Rasterization is where "3D shapes" officially become "2D pixels".

4.3 Pixel (Fragment) Shading

What it is: The pixel shader (also called the fragment shader) runs once for every fragment and decides its colour. This is where most of the beauty of a game scene is made, and where most GPU time goes.

Typical jobs of a pixel shader:

Because millions of fragments need this, this is exactly where the GPU's thousands of cores earn their keep (Section 1).

A word on ray tracing: A newer technique called ray tracing works out light by following imaginary rays of light through the scene. It gives very realistic reflections and shadows, and modern GPUs have special hardware for it. It is added on top of this traditional pipeline in many games, and is not covered further here.

4.4 Output Merging into the Frame Buffer

What it is: The last stage decides what happens to each coloured fragment. The ROP (Render Output Unit) hardware does this. It has three tasks.

Task 1 — Depth test (and stencil test). What if two objects overlap, such as a tree in front of a house? Each fragment carries its depth (how far from the camera). The GPU keeps a second picture, the depth buffer, that remembers the nearest depth drawn so far at each pixel. A new fragment that is farther away is thrown out. A closer one is kept and updates the depth buffer. The stencil test is a similar test that uses a mask, for example to draw only inside a window shape.

   Pixel at position (50, 20)

   House fragment: depth 40   -> kept (nothing is there yet)
   Tree  fragment: depth 10   -> closer -> kept, replaces the house
   Bush  fragment: depth 70   -> farther -> thrown away

Task 2 — Blending. Some things are see-through, like glass or smoke. Blending mixes the new fragment's colour with the colour already stored at that pixel, based on how transparent it is.

Task 3 — Write the result. The final colour is written into the render target in VRAM. When the whole scene has been drawn, the render target holds the finished picture: the frame (Section 7).

The pipeline is programmable and parallel

Each stage works on many items at once, and the stages overlap like a factory line. While some triangles are being rasterized, others are still being moved by the vertex shader.

Summary