4. Coordinate Spaces

Written by Caroline Begbie

Everything you see on your device’s screen, from the characters you type to the digital art you admire, is all math under the surface.

We’d all love to be math geniuses, but some of us lost the opportunity early in life. Fortunately, to use math, you don’t always have to know what’s under the hood. In this chapter, you’re going to become a matrix master, and you’ll learn what matrices can do for you and how to manipulate them painlessly.

You’ll start by learning how to move, scale and rotate a triangle using matrices. Once you’ve mastered one triangle, it’s a cinch to rotate thousands of triangles at once. Then, you’ll upgrade your train project from the previous chapter and understand why you use matrices and what coordinate spaces are.

Transformations

In this picture, the vector image editor, Affinity Designer, was used to translate, scale and rotate a cat through a series of affine transformations.

Instead of individually working out each position, Affinity Designer creates a transformation matrix that holds the combination of the transformations. It then applies the transformation to each element.

Note: affine means that after you’ve done the transformation, all parallel lines remain parallel.

Scaling cats is difficult because they can bite, so instead, you’ll try out translating, scaling and rotating vertices in a Playground.

Translation

Open the starter project playground located in the starter folder for this chapter. Run the playground, and you’ll get a blank cream screen. The starter playground is set up ready for drawing but has nothing to draw yet.

In this playground, you won’t be importing models like you’ve been doing — you’ll be sending vertices to the GPU as a simple array rather than having to set up vertex descriptors. You’ll start by setting up the vertex and fragment functions to draw these single vertices.

Vertex function

Inside the Resources folder for the playground, open Shaders.metal.

Before the existing vertex function, add the struct that will contain the values you’ll return from the vertex function:

struct VertexOut {
  float4 position [[position]];
  float point_size [[point_size]];
};

Here, you set the [[position]] attribute which tells the rasterizer which value contains the position. You also create a property with the [[point_size]] attribute. Points are tiny; the size of a pixel. On a retina screen, you wouldn’t see the point, so you’ll make it larger. The value of this property tells the GPU the size of the point that it will draw.

Replace the vertex function with:

// 1
vertex VertexOut 
       vertex_main(constant float3 *vertices [[buffer(0)]],
// 2
  uint id [[vertex_id]])
{
  // 3
  VertexOut vertex_out {
    .position = float4(vertices[id], 1),
    // 4
    .point_size = 20.0
  };
  return vertex_out;
}

Going through this code:

1. Initially you’ll just draw a single point, but shortly you’ll be sending to the GPU the three vertices of a triangle. These will be an array of float3s containing the xyz position of the vertex. On the Swift side, you’ll set the vertices up in buffer index 0.

   constant tells the GPU to use constant address space. This is optimized for accessing the same variable over several vertex functions in parallel. Device address space, keyword device, is best for when you access different parts of a buffer over the parallel functions. You would use device when using a buffer with points and color data interleaved, for example.

2. The attribute [[vertex_id]] informs the vertex function of the current id of the vertex. It’s the index into the array.

3. Extract out the vertex position from the array and turn it into a float4.

4. Set the point size. You can make this larger or smaller as you prefer.

Fragment function

Now replace the fragment function with this code:

fragment float4 
         fragment_main(constant float4 &color [[buffer(0)]]) 
{
  return color;
}

You’ll send the fragment function the color that it should draw. You’ll put this color value in buffer index 0 on the Swift side.

Set up the data

Back in the main playground page, you’ll draw two vertices. You’ll hold one vertex point in an array and create two buffers: one containing the original point, and one containing the translated point.

In the playground page, under:

// drawing code here

Add this code:

var vertices: [float3] = [[0, 0, 0.5]]

This is the vertex position which is at the center of the screen (in Normalized Device Coordinates). You load it into an array so you can create a triangle with two more vertices later.

Note: float3 is a typealias of type SIMD3<Float>. This is defined for you in Utility.swift as a convenience to match the MSL float3 type.

Create the Metal buffer:

let originalBuffer = device.makeBuffer(bytes: &vertices, 
      length: MemoryLayout<float3>.stride * vertices.count, 
      options: [])

Here, you create an MTLBuffer containing vertices. The length is the number of vertices multiplied by the size of a float3 .

Set up the buffers for the vertex and fragment functions:

renderEncoder.setVertexBuffer(originalBuffer, 
                              offset: 0, index: 0)
renderEncoder.setFragmentBytes(&lightGrayColor,
                    length: MemoryLayout<float4>.stride, 
                    index: 0)

Here, you assign the MTLBuffer containing vertices to index 0 and then assign the color, light gray, to index 0 for the fragment function. lightGrayColor is defined in Utility.swift.

Draw

Do the draw call using the primitive type .point:

renderEncoder.drawPrimitives(type: .point, vertexStart: 0, 
                             vertexCount: vertices.count)

Because you’re not using submesh indices, you use drawPrimitives instead of drawIndexedPrimitives to handle the draw.

Run the playground, and the GPU renders the vertex point at the center of NDC (Normalized Device Coordinates). You will see a gray square at the center of the playground’s live view.

You’ll now move the vertex right and down and create a second buffer with the new values. Add this code under the previous code:

vertices[0] += [0.3, -0.4, 0]
var transformedBuffer = device.makeBuffer(bytes: &vertices, 
       length: MemoryLayout<float3>.stride * vertices.count, 
       options: [])

Here, you added a displacement value to the original vertex and created a new MTLBuffer that holds these new values.

Set up the vertex function with the new buffer, color the point red and draw.

renderEncoder.setVertexBuffer(transformedBuffer, 
                              offset: 0, index: 0)
renderEncoder.setFragmentBytes(&redColor,
                       length: MemoryLayout<float4>.stride, 
                       index: 0)
renderEncoder.drawPrimitives(type: .point, vertexStart: 0, 
                       vertexCount: vertices.count)

This is the same code as with the original buffer.

Note: Why do you need two buffers to draw the same point? Logically, it might seem that you’ve created and drawn the first point, so you can change the vertex and assign it to the same originalBuffer for the second draw call. However, the GPU does not do the draw immediately when drawPrimitives(type:vertexStart:vertexCount:) is called. It does the draw some time after the command buffer presents the drawable. If you want to overwrite the vertex array before drawing it, you need to have two separate data buffers.

The translation code here is straightforward: aside from all the Metal code, you changed the vertex value, just as if you were moving a view’s frame around using CGPoints. Run the playground and see the two points - the gray one in the original position, and the red one in the translated position.

Generally, as well as translation, you’ll need to turn your model around or make it larger or smaller, and this involves rotation and scaling. A transformation encapsulates translation, scaling and rotation all in one, and you can represent a transformation in a matrix.

Vectors and matrices

You can describe your previous translation as a displacement vector of [0.3, -0.4, 0]. You moved the vertex 0.3 units in the x-direction, and -0.4 in the y-direction from its starting position.

In this image, the blue arrows are vectors.

The left blue arrow is a vector with a value [-1, 2]. Coincidentally, the right blue arrow, partially behind the cat, is a vector, also with a value [-1, 2].

Positions (points) are locations in space, whereas vectors are displacements in space; in other words, a vector contains the amount and direction to move.

If you were to displace the cat by the blue vector, it would end up at point (2, 4). That’s the cat’s position (3, 2) plus the vector [-1, 2].

This 2D vector is a 1x2 matrix. It has one column and two rows.

Note: Matrices can be ordered by rows or by columns. Metal matrices are constructed in column-major order, which means that columns are contiguous in memory.

A matrix is a two-dimensional array. Even the single number 1 is a 1×1 matrix. In fact, the number 1 is unique in that when you multiply a number by 1, the answer is always that number. All square matrices — where the array width is the same as the array height — have a matrix with this same property. It’s called the identity matrix. Any vector or matrix multiplied by an identity matrix returns the same value.

A 4×4 identity matrix looks like this (all zeros, except for the diagonal 1s):

A 3D transformation matrix has four rows and four columns. It holds scaling and rotation information in the upper left 3×3 matrix, with the translation information in the last column. When you multiply vectors and matrices, the number of columns of the left side matrix or vector must equal the number of rows of the right side. For example, you can’t multiply a float3 by a float4×4. Shortly, you’ll see that you have to add an extra w dimension to your 3D points.

Included in this chapter’s Resources folder is a matrix calculator app named MatrixCalculator. This calculator app will help you visualize the mathematics of how matrices work. Open up the project and run the app.

On the right-hand side, change Matrix to Vector. The matrix on the left will hold the transformation information and is currently an identity matrix. The vector on the right represents your point in 3D space. The left matrix multiplies the right vector (or matrix) to produce a Result.

Column 3 of the transformation matrix holds a vector that will displace a point.

Compute the cat displacement example from above using a matrix. Change the vector on the right to the cat’s position [3, 2, 0, 1]. In the transformation matrix, change column 3 to [-1, 2, 0, 1]. That’s the two-dimensional vector with an extra z and w value.

The app multiplies the vector by the matrix, and the result is [2, 4, 0, 1], which is the same result as above.

As you go through the following examples, experiment with the matrix calculator to see how your original point changes with the various matrices.

Now you’ll translate your vertex in your playground in the same way. To set up a transformation matrix, add this line of code after declaring the vertices array:

var matrix = matrix_identity_float4x4

Here, you set up a 4×4 identity matrix.

Set the last column of the matrix to be your displacement vector. Replace:

vertices[0] += [0.3, -0.4, 0]

With:

matrix.columns.3 = [0.3, -0.4, 0, 1]

Following on from that code, process each vertex and multiply by the transformation matrix.

vertices = vertices.map {
  let vertex = matrix * float4($0, 1)
  return [vertex.x, vertex.y, vertex.z]
}

Remember that you can only multiply similar matrices, where the number of columns of the left matrix or vector is equal to the number of rows of the one on the right. Here, you convert the float3 from the vertices array into a float4, adding the extra w component to your vertex so that you can then multiply the vertex by the 4×4 matrix.

Run the playground, and you get the same result as you did before using the matrix.

Matrices on the GPU

You may have noticed that this vertex processing code is taking place on the CPU. This is serial processing, which is much more inefficient compared to parallel processing. There’s another place where each vertex is being processed — the GPU. You can pass the GPU your transformation matrix and multiply every vertex in the vertices array by the matrix in the vertex shader. The GPU is optimized for matrix calculation.

Replace the code:

vertices = vertices.map {
  let vertex = matrix * float4($0, 1)
  return [vertex.x, vertex.y, vertex.z]
}

With:

renderEncoder.setVertexBytes(&matrix, 
     length: MemoryLayout<float4x4>.stride, index: 1)

Here, you’re sending the matrix to the GPU.

In Shaders.metal, change the vertex function definition to:

vertex VertexOut 
        vertex_main(constant float3 *vertices [[buffer(0)]],
                    constant float4x4 &matrix [[buffer(1)]],
                    uint id [[vertex_id]]) 

This will receive your matrix into your vertex function via buffer index 1.

Replace:

.position = float4(vertices[id], 1),

With:

.position = matrix * float4(vertices[id], 1),

This is where you multiply each vertex by the transformation matrix.

Because you’ve changed the vertex function, you’ll also need to send the matrix for drawing the first point. In the playground page, add this before the first draw call:

renderEncoder.setVertexBytes(&matrix, 
           length: MemoryLayout<float4x4>.stride, index: 1)

Run the playground, and you should still get the same result. This time though, the multiplication is happening on the GPU in parallel and is more efficient.

Scaling

Translating a single vertex is useful, but you’ll want to scale and rotate your models to fit inside your scene.

Instead of a single vertex, you’ll now draw and manipulate a triangle with three vertices.

Change:

var vertices: [float3] = [[0, 0, 0.5]]

To:

var vertices: [float3] = [
  [-0.7,  0.8,   1],
  [-0.7, -0.4,   1],
  [ 0.4,  0.2,   1]
]

You’re now sending three vertices at a time to the GPU, and when you run your playground, it should display three gray points and the three points transformed by the matrix. To display a solid triangle, change the two draw calls to render triangles instead of points. Change both draw calls from:

renderEncoder.drawPrimitives(type: .point, vertexStart: 0, 
                             vertexCount: vertices.count)

To:

renderEncoder.drawPrimitives(type: .triangle, vertexStart: 0, 
                             vertexCount: vertices.count)

The original untranslated triangle displays in light gray, and the translated triangle displays in red.

You’ll now scale the red triangle. Later, you’ll use pre-made functions to create these transformation matrices, but just so that you can get a feel for what’s happening under the hood, you’ll set up the matrices manually here.

Remove this line of code:

matrix.columns.3 = [0.3, -0.4, 0, 1]

And replace it with:

let scaleX: Float = 1.2
let scaleY: Float = 0.5
matrix = float4x4(
  [scaleX, 0, 0, 0],
  [0, scaleY, 0, 0],
  [0,      0, 1, 0],
  [0,      0, 0, 1]
)

Without going into the mathematics too much, this is how you set up a scale matrix.

The vertex function will process all vertices. The following result for the top-left vertex shows that you’ll scale the x vector value by 1.2 and the y vector value by 0.5.

Run the playground. You’ve set up the matrix maths already on the GPU, so the scale transformation happens to the red triangle just by sending the correct scale matrix.

The following screen capture demonstrates a scaled x coordinate.

Rotation

You perform rotation in a similar way to scaling. Replace:

let scaleX: Float = 1.2
let scaleY: Float = 0.5
matrix = float4x4(
  [scaleX, 0, 0, 0],
  [0, scaleY, 0, 0],
  [0,      0, 1, 0],
  [0,      0, 0, 1]
)

With:

let angle = Float.pi / 2.0
matrix.columns.0 = [cos(angle), -sin(angle), 0, 0]
matrix.columns.1 = [sin(angle), cos(angle), 0, 0]

Instead of setting up the entire matrix, you have the option of only setting the changed matrix columns. Here you set a rotation around the z axis of the angle in radians.

Note: Float.pi / 2.0 is the same as 90º. If you’re not sure how to convert between radians and degrees, the project for this chapter includes an extension on Float that will convert degrees to radians in MathLibrary.swift. Radians is the standard unit in computer graphics.

Run the playground, and the red triangle is rotated.

Notice that the rotation takes place around the center of the screen. The center of the screen is the world’s origin. Each vertex rotates around the world origin.

Matrix concatenation

You may want the rotation to take place around a point other than the world origin. You’ll now rotate the triangle around its right-most point.

To do this, you’ll work out the distance between the world origin and the right-most point. You’ll translate all the vertices by this distance, rotate, and then translate back again.

Change your previous rotation code:

matrix.columns.0 = [cos(angle), -sin(angle), 0, 0]
matrix.columns.1 = [sin(angle), cos(angle), 0, 0]

To:

var distanceVector = float4(vertices.last!.x,
                            vertices.last!.y,
                            vertices.last!.z, 1)
var translate = matrix_identity_float4x4
translate.columns.3 = distanceVector
var rotate = matrix_identity_float4x4
rotate.columns.0 = [cos(angle), -sin(angle), 0, 0]
rotate.columns.1 = [sin(angle), cos(angle), 0, 0]

Here, you set up two separate matrices: one for translation and one for rotation. The translation matrix is using the right-most point of the triangle.

Now for the magic! You can multiply several matrices together to get one final matrix that holds all of the transformations.

Remember the steps. Step 1 was to translate all the other vertices by the distance from the world origin. You can achieve this by setting a matrix to the vertex’s vector value and using the translate matrix’s inverse.

Run the playground after each of the following steps to follow what the matrix multiplication does. Add this after the previous code:

matrix = translate.inverse 

You moved the triangle so that the right-most point is at the world origin.

Change the code you just entered to:

matrix = rotate * translate.inverse

The triangle rotates by 90º around the world origin.

Note: Matrix multiplication order is important. Try changing the previous code to:

matrix = translate.inverse * rotate

You get a completely different (and incorrect in this instance) result. Generally, the order of multiplication is TRS or translate * rotate * scale * point. The matrix operations work backward — you first scale the point, then rotate the result, then finally translate that result.

Change the code you just entered to:

matrix = translate * rotate * translate.inverse

Now you’re doing all the steps of translating each vertex by the distance of the right-most vertex from the world origin; then rotating it; then translating it back again.

Run the playground, and you’ll see the final 90º rotation around the right-most triangle point.

You now know how to translate, rotate and scale points and triangles using transformation matrices.

Coordinate spaces

Now that you know about matrices, you’ll be able to convert models and entire scenes between different coordinate spaces. Coordinate spaces map different coordinate systems, and just by multiplying a vertex by a particular matrix, you convert the vertex to a different space.

A vertex on its trip through the pipeline will pass through (usually) six spaces:

• Object space
• World space
• Camera space
• Clip space
• NDC (Normalized Device Coordinate) space
• Screen space

This is starting to sound like a description of Voyager leaving our solar system, so take a closer conceptual look at each space.

Object space

You may be familiar with Cartesian coordinates from your graphing days. This image is a 2D grid showing possible vertices mapped out in Cartesian coordinates.

The positions of these vertices are in relation to the origin of the dog, which is at (0, 0). They are in what’s called object space (or local or model space).

World space

In the following picture, the direction arrows mark the origin. This is the center of world space at (0, 0, 0). In world space, the dog is at (1, 0, 1) and the cat is at (-1, 0, -2).

However, cats are always the center of their universe, so in cat space, the cat thinks that he is at (0, 0, 0). This would make the dog’s position relative to the cat (2, 0, 3). When the cat moves, in his universe he is always at (0, 0, 0) and the position of everything else in the world changes relative to the cat.

Note: Cat space is not recognized as a traditional 3D coordinate space, but mathematically, you can create your own space and use any position in the universe as the origin. Every other point in the universe is now relative to that origin. In a later chapter, you’ll discover other spaces besides the ones described here.

Camera space

For the dog, the center of his universe is the person holding the camera behind the picture. In camera space, the camera is at (0, 0, 0), and the dog is approximately at (-3, -2, 7). When the camera moves, it stays at (0, 0, 0), but the positions of the dog and cat move relative to the camera.

Clip space

The main reason for doing all this math is to turn a three-dimensional scene with perspective into a two-dimensional scene. Clip space is a cube that is ready for flattening.

Note: If you want to render engineering drawings, for example, you might use orthographic or isometric projection instead of perspective projection.

NDC space

Projection into clip space creates a half cube of w size. During rasterization, the GPU will convert the w into normalized coordinate points between -1 and 1 for the x-axis and y-axis and 0 and 1 for the z-axis.

Screen space

Now that the GPU has a normalized cube, it will flatten clip space and convert into screen coordinates ready to display on the device’s screen.

In this image, the dog is the same size as the cat, but it’s further away from the camera.

Converting between spaces

You may have already guessed it: you use transformation matrices to convert from one space to another.

For example, in the following image, the vertex on the dog’s ear, which was (-1, 4, 0) in object space, is now, looking at the picture, at about (0.75, 1.5, 1) in world space.

To move the dog vertices from object space to world space, using a transformation matrix, you’d translate (move) them, and also scale them down.

There are four spaces that you control, so there are three corresponding matrices that you’ll shortly construct:

• model matrix: between object and world space
• view matrix: between world and camera space
• projection matrix: between camera and clip space

Coordinate systems

Different graphics APIs use different systems. You already found out that Metal’s NDC (Normalized Device Coordinates) use 0 to 1 on the z-axis. You may already be familiar with OpenGL, which uses 1 to -1 on the z-axis.

In addition to being a different size, OpenGL’s z-axis points in the opposite direction from Metal’s z-axis. OpenGL’s system is called a right-handed coordinate system, and Metal’s is a left-handed coordinate system.

Both systems use x to the right and y as up. Blender uses a different coordinate system again, where z is up, and y is into the screen.

If you are consistent with your coordinate system and create matrices accordingly, it doesn’t matter what coordinate system you use. In this book, we chose to use Metal’s left-handed coordinate system, but we could equally have decided to use a right-handed coordinate system with different matrix creation methods.

Upgrade the engine

Open the starter app for this chapter, named Matrices. This app is the same as at the end of the previous chapter, with the addition of MathLibrary.swift. This utility file contains methods that are extensions on float4x4 for creating the translation, scale and rotation matrices that you created in your 3DTransforms playground.

Currently, your train takes up the whole screen, is stretched to fill the window, resizes when you resize the window and has no depth perspective.

You can decouple the train’s vertex positions from the window size by taking the train into other coordinate spaces. It’s the vertex function that is responsible for converting the model vertices through these various coordinate spaces, and that’s where you will perform the matrix multiplications that do the conversions between different spaces.

Uniforms

Constant values that are the same across all vertices or fragments are generally referred to as uniforms. You’ll create a uniform struct to hold the conversion matrices and then apply them to every vertex.

Both the shaders and the code on the Swift side will access these uniform values. If you were to create a struct in Renderer and a matching struct in Shaders.metal, it’s effortless to forget to keep them synchronized. The easiest method is to create a bridging header that both C++ and Swift can access.

Create a new file in the Matrices group, using the macOS Header File template. Name it Common.h.

In the Project navigator, click the main Matrices project folder.

Select the Project Build Settings. In the search bar, type bridg to narrow down the settings. Double click the Objective-C Bridging Header value and enter Matrices/Common.h.

This tells Xcode to use this file for both the Metal Shading Language and Swift.

In Common.h, before the final #endif, import the simd framework. This framework provides types and functions for working with vectors and matrices. Add this code:

#import <simd/simd.h>

Model matrix

Add the uniforms struct to Common.h:

typedef struct {
  matrix_float4x4 modelMatrix;
  matrix_float4x4 viewMatrix;
  matrix_float4x4 projectionMatrix;
} Uniforms;

These three matrices with four rows and four columns, will hold the necessary conversion between spaces as mentioned earlier.

Your train vertices are currently in object space. You’ll now use modelMatrix to convert them to world space. Simply by changing modelMatrix you’ll easily be able to translate, scale and rotate your train.

In Renderer.swift, add the new struct to Renderer:

var uniforms = Uniforms()

Common.h is the bridging header file, so Swift is able to pick up the Uniforms type.

At the bottom of init(metalView:) add:

let translation = float4x4(translation: [0, 0.3, 0])
let rotation = 
    float4x4(rotation: [0, 0, Float(45).degreesToRadians])
uniforms.modelMatrix = translation * rotation

Here, you set modelMatrix to have a translation of 0.3 units up and a counterclockwise rotation of 45 degrees.

In draw(in:), replace:

timer += 0.05
var currentTime: Float = sin(timer)
renderEncoder.setVertexBytes(&currentTime, 
             length: MemoryLayout<Float>.stride, index: 1)

With:

renderEncoder.setVertexBytes(&uniforms, 
             length: MemoryLayout<Uniforms>.stride, index: 1)

That’s set up the uniform matrix values on the Swift side. In Shaders.metal, import the bridging header file below setting the namespace:

#import "Common.h"

Change the vertex function to:

vertex float4 vertex_main(const VertexIn vertexIn [[stage_in]],
                    constant Uniforms &uniforms [[buffer(1)]])
{
  float4 position = uniforms.modelMatrix * vertexIn.position;
  return position;
}

Here, you receive the Uniforms struct as a parameter and multiply all the vertices by the model matrix. Build and run the app.

You should see the following:

All the vertices are translated by 0.3 units in the y-direction and then rotated. Notice how the train is skewed. Make your window square to see the train rotated correctly.

You’ll fix this skewing shortly.

View matrix

To convert between world space and camera space, you’ll set a view matrix. Depending on how you want to move the camera in your app, you can construct the view matrix appropriately. The view matrix you’ll create here is a simple one that is best for FPS (First Person Shooter) style games.

In Renderer.swift, at the end of init(metalView:), add this code:

uniforms.viewMatrix = float4x4(translation: [0.8, 0, 0]).inverse

Remember that all the objects in the scene should move in the opposite direction to the camera. inverse does an opposite transformation. For example, as the camera moves to the right, everything in the world will move 0.8 units to the left. You set the camera as you want it in world space, and then add .inverse on the end so that all objects will react relatively to the camera.

In Shaders.metal, change:

float4 position = uniforms.modelMatrix * vertexIn.position;

To:

float4 position = uniforms.viewMatrix * uniforms.modelMatrix 
                      * vertexIn.position;

Build and run the app, and the train moves to the left. Later, you’ll navigate through a scene using the keyboard and just changing the view matrix will update all the objects in the scene around the camera.

The last matrix you’ll set will prepare the vertices to move from camera space to clip space. It will also allow you to use unit values instead of the -1 to 1 NDC (Normalized Device Coordinates) that you’ve been using up to now.

Rotate the train on the y-axis to demonstrate why this is necessary. In Renderer.swift, in draw(in:), just above:

renderEncoder.setVertexBytes(&uniforms, 
       length: MemoryLayout<Uniforms>.stride, index: 1)

Add this code:

timer += 0.05
uniforms.viewMatrix = float4x4.identity()
uniforms.modelMatrix = float4x4(rotationY: sin(timer))

Here, you reset the camera and replace the translation matrix with a rotation around the y-axis.

Build and run the app.

You can see that when the train rotates, any vertices that are greater than 1.0 on the z-axis are being clipped. Remember that any vertex outside Metal’s NDC will be clipped.

Projection

So far you haven’t applied any perspective to your render. Perspective is where close objects appear bigger than objects that are farther away.

When you render a scene, you’ll have to take into account:

• How much of that scene will fit on the screen. Your eyes have a field of view of about 200º, and within that field of view, your computer screen takes up about 70º.
• How far you can see by having a far plane. Computers can’t see to infinity.
• How close you can see by having a near plane.
• The aspect ratio of the screen. Currently, your train changes size when the screen size changes. When you take into account the width and height ratio, this won’t happen.

The image above shows all these. The shape created from the near to the far plane is a cut-off pyramid called a frustum. Anything in your scene that is located outside the frustum will not render.

Compare the rendered image again to the scene setup. The rat in the scene does not render because he is in front of the near plane.

MathLibrary.swift provides a projection method that returns the matrix to project objects within this frustum into clip space ready for conversion to NDC coordinates.

Projection Matrix

Open Renderer.swift, and at the end of init(metalView:) set up the projection matrix:

let aspect = Float(metalView.bounds.width) / Float(metalView.bounds.height)
let projectionMatrix =
  float4x4(projectionFov: Float(45).degreesToRadians,
           near: 0.1,
           far: 100,
           aspect: aspect)
uniforms.projectionMatrix = projectionMatrix

You’re using a field of view of 45º; a near plane of 0.1, and a far plane of 100 units.

In Shaders.metal, in the vertex function, change the position matrix calculation to:

float4 position = 
      uniforms.projectionMatrix * uniforms.viewMatrix
      * uniforms.modelMatrix * vertexIn.position;

Build and run the app. The z-coordinates measure differently now, so you’re zoomed in on the train.

In Renderer.swift, in draw(in), replace:

timer += 0.05
uniforms.viewMatrix = float4x4.identity()
uniforms.modelMatrix = float4x4(rotationY: sin(timer))

With:

uniforms.viewMatrix = float4x4(translation: [0, 0, -3]).inverse

This moves the camera back into the scene by three units.

In init(metalView:), change:

let rotation = 
    float4x4(rotation: [0, 0, Float(45).degreesToRadians])

To:

let rotation = 
    float4x4(rotation: [0, Float(45).degreesToRadians, 0])

This changes the model’s rotation from around the z axis to around the y axis.

In init(metalView:), change the projection matrix’s projectionFOV parameter to 70º: the train appears smaller because the field of view is wider, and more objects horizontally will fit into the rendered scene.

Note: Experiment with the projection values in init(metalView:). Set translation’s z value to a distance of 97: the front of the train is just visible. At z = 98, the train is no longer visible. (The projection far value is 100 units, and the camera is back 3 units.) If you change the projection’s far parameter to 1000, the train is visible again.

Perspective divide

Now that you’ve converted your vertices from object space through world space through camera space to clip space, the GPU takes over to convert to NDC coordinates (that’s -1 to 1 in the x and y directions and 0 to 1 in the z direction). The ultimate aim is to scale all the vertices from clip space into NDC space, and by using the fourth w component, this becomes easy.

To scale a point such as (1, 2, 3), you can have a fourth component: (1, 2, 3, 3). Divide by that last w component to get (1/3, 2/3, 3/3, 1). The xyz values are now scaled down. These coordinates are called homogeneous. (Homogeneous means of the same kind.)

The projection matrix projected the vertices from a frustum to a cube in the range -w to w. After the vertex leaves the vertex function along the pipeline, the GPU will perform a perspective divide and divide the x, y and z values by their w value. The higher the w value, the further back the coordinate is. The result of this is that all visible vertices will now be within NDC.

Note: To avoid a divide by zero, the projection near plane should always be a value slightly more than zero.

In the following picture, the dog and the cat are the same height — perhaps a y value of 2 for example. With projection, as the dog is further back, it should appear smaller in the final render.

After projection, the cat might have a w value of ~1 and the dog a w value of ~8. Dividing by w would give the cat a height of 2 and the dog a height of 1/4 which will make the dog appear smaller.

NDC to screen

Lastly, the GPU converts from normalized coordinates to whatever the device screen size is. You may already have done something like this at some time in your career when converting between normalized coordinates and screen coordinates.

To convert Metal NDC (Normalized Device Coordinates) which are between -1 and 1 to a device, you could use code something like this:

converted.x = point.x * screenWidth/2  + screenWidth/2
converted.y = point.y * screenHeight/2 + screenHeight/2

However, you can also do this with a matrix by scaling half the screen size and translating by half the screen size. The clear advantage of this is that you can set up a transformation matrix once and multiply any normalized point by the matrix to convert it into the correct screen space using code like this:

converted = matrix * point

As you can see in MatrixCalculator, this transformation matrix transforms a normalized point to iPhone XS (375×812) space:

The rasterizer on the GPU takes care of this matrix calculation for you.

Update screen dimensions

Currently, when you rotate your iOS device or rescale the macOS window, the train stretches with the size of the window. You’ll need to update the aspect ratio for the projection matrix whenever this happens. Fortunately MTKViewDelegate gives you a method whenever the view’s drawable size changes.

In Renderer.swift, add the following to mtkView(_:drawableSizeWillChange:):

let aspect = Float(view.bounds.width) / Float(view.bounds.height)
let projectionMatrix =
  float4x4(projectionFov: Float(70).degreesToRadians,
           near: 0.001,
           far: 100,
           aspect: aspect)
uniforms.projectionMatrix = projectionMatrix

In init(metalView:), replace the projection code similar to the above with:

mtkView(metalView, 
        drawableSizeWillChange: metalView.bounds.size)

Build and run the app; now the projection matrix updates every time you resize the window.

Where to go from here?

You’ve covered a lot of mathematical concepts without diving too far into the underlying mathematical principles. To get started in computer graphics, you can fill your transform matrices and continue multiplying them at the usual times, but to be sufficiently creative, you will need to understand some linear algebra.

A great place to start is Grant Sanderson’s Essence of Linear Algebra at https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab. This treats vectors and matrices visually.

You’ll also find some further references in references.markdown in the Resources for this chapter.

Even though your train is rendering in three dimensions, it still looks two dimensional. Some lighting and fake shadows will help improve the render in the next chapter.