7. Maps & Materials

Written by Caroline Begbie

In the previous chapter, using Model I/O, you imported and rendered a simple house with a flat color texture. But if you look at the objects around you, you’ll notice how their basic color changes according to how light falls on them. Some objects have a smooth surface, and some have a rough surface. Heck, some might even be shiny metal!

In this chapter, you’ll find out how to use material groups to describe a surface, and how to design textures for micro detail. This is also the final chapter on how to render still models.

Normal maps

The following example best describes normal maps:

On the left, there’s a lit cube with a color texture. On the right, there’s the same low poly cube with the same color texture and lighting, however, it also has a second texture applied to it, called a normal map.

With the normal map, it looks as if the cube is a high poly cube with all of the nooks and crannies modeled into the object. But this is just an illusion!

For this illusion to work, it needs a texture, like this:

All models have normals that stick out perpendicular to each face. A cube has six faces, and each face’s normal points in a different direction. Also, each face is flat. If you wanted to create the illusion of bumpiness, you need to change a normal in the fragment shader.

In the following image, on the left is a flat surface with normals in the fragment shader. On the right, you see perturbed normals. The texels in a normal map supply the direction vectors of these normals through the RGB channels.

Take a look at this single brick split out into the red, green and blue channels that make up an RGB image.

Each channel has a value between 0 and 1, and you generally visualize them in grayscale as it’s easier to read color values. For example, in the red channel, a value of 0 is no red at all, while a value of 1 is full red. When you convert 0 to an RGB color (0, 0, 0), that results in black. On the opposite spectrum, (1, 1, 1) is white, and in the middle you have (0.5, 0.5, 0.5) which is mid-gray. In grayscale, all three RGB values are the same, so you only need refer to a grayscale value by a single float.

Take a closer look at the edges of the red channel’s brick. Look at the left and right edges in the grayscale image. The red channel has the darkest color where the normal values of that fragment should point left (-X, 0, 0), and the lightest color where it should point right (+X, 0, 0).

Now look at the green channel. The left and right edges have equal value but are different for the top and bottom edges of the brick. The green channel in the grayscale image has darkest for pointing down (0, -Y, 0) and lightest for pointing up (0, +Y, 0).

Finally, the blue channel is mostly white in the grayscale image because the brick — except for a few irregularities in the texture — points outward. The edges of the brick are the only places where the normals should point away.

Note: Normal maps can be either right-handed or left-handed. Your renderer will expect positive y to be up, but some apps will generate normal maps with positive y down. To fix this, you can take the normal map into Photoshop and invert the green channel.

The base color of a normal map — where all normals are “normal” (orthogonal to the face) — is (0.5, 0.5, 1).

This is an attractive color but was not chosen arbitrarily. RGB colors have values between 0 and 1, whereas a model’s normal values are between -1 and 1. A color value of 0.5 in a normal map translates to a model normal of 0.

The result of reading a flat texel from a normal map should be a z value of 1 and the x and y values as 0. Converting these values (0, 0, 1) into the colorspace of a normal map results in the color (0.5, 0.5, 1). This is why most normal maps appear bluish.

Creating normal maps

To create successful normal maps, you need a specialized app. In the previous chapter, you learned about texturing apps, such as Substance Designer and Mari. Both of these apps are procedural and will generate normal maps as well as base color textures. In fact, the brick texture in the image at the start of the chapter was created in Substance Designer.

Sculpting programs, such as ZBrush, 3D-Coat, Mudbox and Blender will also generate normal maps from your sculpts. You sculpt a detailed high poly mesh; then, the app looks at the cavities and curvatures of your sculpt, and bakes a normal map. Because high poly meshes with tons of vertices aren’t resource-efficient in games, you should create a low poly mesh and then apply the normal map to this mesh.

Photoshop CC (from 2015), CrazyBump and Bitmap2Material can generate a normal map from a photograph or diffuse texture. Because these apps look at the shading and calculate the values, they aren’t as good as the sculpting or procedural apps, but it can be quite amazing to take a photograph of a real-life, personal object, run it through one of these apps, and render out a shaded model.

Here’s a normal map that was created using Allegorithmic’s Bitmap2Material:

Tangent space

You send the normal map to the fragment function in the same way as a color texture, and you extract the normal values using the same UVs. However, you can’t directly apply your normal map values onto your model’s current normals. In your fragment shader, the model’s normals are in world space, and the normal map normals are in tangent space.

Tangent space is a little hard to wrap your head around. Think of the brick cube with all its six faces pointing in different directions. Now think of the normal map with all the bricks the same color on all the six faces.

If a cube face is pointing toward negative x, how does the normal map know to point in that direction?

Using a sphere as an example, every fragment has a tangent — that’s the line that touches the sphere at that point. The normal vector in this tangent space is thus relative to the surface. You can see that all of the arrows are at right angles to the tangent. So if you took all of the tangents and laid them out on a flat surface, the blue arrows would point upward in the same direction. That’s tangent space!

The following image shows a cube’s normals in world space.

To convert the cube’s normals to tangent space, you create a TBN matrix - that’s a Tangent Bitangent Normal matrix that’s calculated from the tangent, bitangent and normal value for each vertex.

In the TBN matrix, the normal is the perpendicular vector as usual; the tangent is the vector that points along the horizontal surface; and the bitangent is the vector — as calculated by the cross product — that is perpendicular to both the tangent and the normal.

Note: the cross product is an operation that gives you a vector perpendicular to two other vectors.

The tangent can be at right angles to the normal in any direction; however, to share normal maps across different parts of models, and even entirely different models, there are two standards:

1. The tangent and bitangent will represent the directions that u and v point, respectively, defined in model space.
2. The red channel will represent curvature along u, and the green channel, along v.

You could calculate these values when you load the model; however, with Model I/O, as long as you have data for both the position and texture coordinate attributes, Model I/O can calculate and store these tangent and bitangent values at each vertex for you.

Finally some code! :]

Using normal maps

Open up the starter project for this chapter. Note there are a few changes from the previous chapter’s final code:

• pipelineState moved to Submesh from Model. Materials may need different rendering requirements, so each material needs its own pipeline state.

Note: if you are in control of the asset pipeline and know that all models are textured consistently across all models, then you can centralize your pipeline state object (PSO) more. Holding the PSO on the submesh allows for the times when part of the model is textured and part of it isn’t.

• The model is now a textured cottage, and the textures are in Textures.xcassets.
• There is additional code in Submesh.swift and Shaders.metal to read the normal texture in the same way as you read the diffuse color texture in the previous chapter.

Open up Submesh.swift. You’ll find a new property, normal, in Textures, with the corresponding loading of the texture file in Submesh.Textures, in init(material:) using the Model I/O material property, .tangentSpaceNormal.

Open cottage1.mtl located in the Models folder. The map_tangentSpaceNormal property is what Model I/O is expecting for the normal texture. cottage-normal is a texture in Textures.xcassets.

You’ll deal with multiple textures that hold data instead of color. Looking at these textures in a photo editor, you’d think they are color, but the trick is to regard the RGB (0, 0, 0) values as numerical data instead of color data.

In Renderer.swift, in draw(in:), you’re sending submesh.textures.normal in the same way as you did the base color texture in the previous chapter. Similarly, in Shaders.metal, the fragment function receives the normal texture in texture index 1; but aside from normalizing the value, the fragment function doesn’t do anything with the texture value.

Build and run, and you’ll see a quaint cartoon cottage.

It’s a bit plain, but you’re going to add a normal map to help give it some surface details.

The first step is to apply the normal texture to the cottage as if it were a color texture.

Open Shaders.metal. In fragment_main, after reading normalValue from normalTexture, add this:

return float4(normalValue, 1);

This is only temporary to make sure the app is loading the normal map correctly, and that the normal map and UVs match.

Build and run to verify the normal map is providing the fragment color.

You can see all the surface details the normal map will provide. There are scattered bricks on the wall, wood grain on the door and windows and a shingle-looking roof.

Excellent! You tested that the normal map loads, so it’s time to remove the previous line of code. Remove this from the fragment function:

return float4(normalValue, 1);

You may have noticed that in the normal map’s bricks, along the main surface of the house, the red seems to point along negative y, and the green seems to map to positive x. You might expect that red (1, 0, 0) maps to x and green (0, 1, 0) maps to y. This is because the UV island for the main part of the house is rotated 90 degrees counterclockwise.

Not to worry; the mesh’s stored tangents will map everything correctly! They take UV rotation into account.

Don’t celebrate just yet! You have several tasks ahead of you. You still need to:

1. Load tangent and bitangent values using Model I/O.
2. Tell the render command encoder to send the newly created MTLBuffers containing the values to the GPU.
3. In the vertex shader, change the values to world space — just as you did normals — and pass the new values to the fragment shader.
4. Calculate the new normal based on these values.

1. Load tangents and bitangents

Open VertexDescriptor.swift and look at defaultVertexDescriptor. Model I/O is currently reading into the vertex buffer the normal values from the .obj file. And you can see that you’re telling the vertex descriptor that there are normal values in the attribute named MDLVertexAttributeNormal.

So far, your models have normal values included with them, but you may come across odd files where you have to generate normals. You can also override how the modeler smoothed the model. For example, the house model has smoothing applied in Blender so that the roof, which has very few faces, does not appear too blocky.

Smoothing recalculates vertex normals so that they interpolate smoothly over a surface. Blender stores smoothing groups in the .obj file, which Model I/O reads in and understands. Notice how the edges of the above sphere are unchanged. Smoothing only changes the way the renderer evaluates the surface. Smoothing does not change the geometry.

Try reloading vertex normals and overriding the smoothing. Open Model.swift and in init(name:), replace:

let (mdlMeshes, mtkMeshes) = 
     try! MTKMesh.newMeshes(asset: asset,
                            device: Renderer.device)

With:

var mtkMeshes: [MTKMesh] = []
let mdlMeshes = 
     asset.childObjects(of: MDLMesh.self) as! [MDLMesh]
_ = mdlMeshes.map { mdlMesh in
  mdlMesh.addNormals(withAttributeNamed: 
                        MDLVertexAttributeNormal,
                     creaseThreshold: 1.0)
  mtkMeshes.append(try! MTKMesh(mesh: mdlMesh, 
                                device: Renderer.device))
}

You’re now loading the MDLMeshes first and changing them before initializing the MTKMeshes. You ask Model I/O to recalculate normals with a crease threshold of 1. This crease threshold, between 0 and 1, determines the smoothness, where 1.0 is unsmoothed.

Build and run, and notice that the cottage is now completely unsmoothed, and you can see all of its separate faces. If you were to try a creaseThreshold of zero, where everything is smoothed, you’d get a lot of rendering artifacts because of the surfaces rounding too far. When dealing with smoothness remember this: Smoothness is good, but use it with caution. The artist needs to set up the model with smoothing in mind.

Remove the line you just added that said:

mdlMesh.addNormals(withAttributeNamed: MDLVertexAttributeNormal,
                   creaseThreshold: 1.0)

and replace it with this:

mdlMesh.addTangentBasis(forTextureCoordinateAttributeNamed: 
            MDLVertexAttributeTextureCoordinate,
          tangentAttributeNamed: MDLVertexAttributeTangent,
          bitangentAttributeNamed: MDLVertexAttributeBitangent)

All of the models have normals provided by Blender, so this new code loads the vertex tangent and bitangent values directly. Model I/O does a few things behind the scenes:

• Add two named attributes to mdlMesh’s vertex descriptor: MDLVertexAttributeTangent and MDLVertexAttributeBitangent.
• Calculate the tangent and bitangent values.
• Create two new MTLBuffer’s to contain them.
• Update the layout strides on mdlMesh’s vertex descriptor to match the two new buffers.

With the addition of these two new attributes, you should change the vertex descriptor. Each mdlMesh provides a vertex descriptor, so for convenience, you’ll hold this as a class property. Add this new property to Model:

static var vertexDescriptor: MDLVertexDescriptor =
  MDLVertexDescriptor.defaultVertexDescriptor

Inside the map closure just after where you added the tangent and bitangent attributes, update the Model vertex descriptor:

Model.vertexDescriptor = mdlMesh.vertexDescriptor

In Submesh.swift, in makePipelineState(textures:), change vertexDescriptor to use this new vertex descriptor:

let vertexDescriptor = Model.vertexDescriptor

You’ve completed the necessary updates to the model’s vertex layouts, and now you’ll update the rendering code to match.

2. Send tangent and bitangent values to the GPU

In Renderer.swift, in draw(in:), locate // render multiple buffers and these lines of code:

let vertexBuffer = mesh.mtkMesh.vertexBuffers[0].buffer
renderEncoder.setVertexBuffer(vertexBuffer, offset: 0,
                      index: Int(BufferIndexVertices.rawValue))

This only sends one MTLBuffer to the GPU per mesh. The buffer contains the position, normal and UV data — all interleaved and unpacked by the vertex function according to the vertex descriptor attributes.

However, you now have two more buffers added by Model I/O for the tangent and bitangent values, with two more attributes to deal with, so you need to update the code.

Replace the two lines of code that follow // replace the following two lines with:

for (index, vertexBuffer) in 
      mesh.mtkMesh.vertexBuffers.enumerated() {
  renderEncoder.setVertexBuffer(vertexBuffer.buffer,
                                offset: 0, index: index)
}

Now, you’re enumerating all of the mesh’s vertex buffers and sending them all to the GPU.

Build and run, and you’ll get the dreaded clear screen!

In the code you just added, you’re sending three MTLBuffers, with three different argument table indices. With the earlier code, you specified that you would send uniforms in argument table index 1. With the new code, you’re overwriting the table with the new MTLBuffers and the vertex function. Since it’s expecting uniforms in index 1, it thinks it has incorrect values.

You can easily fix this by changing the index numbers in Common.h.

In Common.h, change:

typedef enum {
  BufferIndexVertices = 0,
  BufferIndexUniforms = 1,
  BufferIndexLights = 2,
  BufferIndexFragmentUniforms = 3
} BufferIndices;

To:

typedef enum {
  BufferIndexVertices = 0, 
  BufferIndexUniforms = 11,
  BufferIndexLights = 12,
  BufferIndexFragmentUniforms = 13
} BufferIndices;

Here, you’re leaving a gap so that you can assign indices 0 to 10 for various MTLBuffers. Build and run, and your cottage should be back.

Click the Capture GPU Frame button to see the buffers on the GPU.

The tangent and bitangent values are in argument table indices 1 and 2, and uniforms is now in argument table index 11.

3. Convert tangent and bitangent values to world space

Just as you converted the model’s normals to world space, you need to convert the tangents and bitangents to world space in the vertex function.

In Common.h, add this to enum Attributes:

Tangent = 3,
Bitangent = 4

In Shaders.metal, add these new attributes to struct VertexIn:

float3 tangent [[attribute(Tangent)]];
float3 bitangent [[attribute(Bitangent)]];

Add new properties to struct VertexOut so that you can send the values to the fragment function:

float3 worldTangent;
float3 worldBitangent;

In vertex_main, after calculating out.worldNormal, add this:

.worldTangent = uniforms.normalMatrix * vertexIn.tangent,
.worldBitangent = uniforms.normalMatrix * vertexIn.bitangent,

This moves the tangent and bitangent values into world space.

4. Calculate the new normal

Now that you have everything in place, it’ll be a simple matter to calculate the new normal.

Before doing the normal calculation, consider the normal color value that you’re reading. Colors are between 0 and 1, and normal values range from -1 to 1.

Still in Shaders.metal, in fragment_main, after reading the normal from the texture, but before normalizing it, add:

normalValue = normalValue * 2 - 1;

This redistributes the normal value to be within the range -1 to 1.

Further down in the function, replace:

float3 normalDirection = normalize(in.worldNormal);

With:

float3 normalDirection = float3x3(in.worldTangent, 
                                  in.worldBitangent, 
                                  in.worldNormal) * normalValue;
normalDirection = normalize(normalDirection);

Build and run to see the normal map applied to the cottage.

As you rotate the cottage, notice how the lighting affects the small cavities on the model, especially on the door and roof where the specular light falls — it’s almost like you created new geometry but didn’t! :]

Normal maps are almost like magic, and great artists can add amazing detail to simple low poly models.

Other texture map types

Normal maps are not the only way of changing a model’s surface. There are other texture maps:

• Roughness: Describes the smoothness or roughness of a surface. You’ll add a roughness map shortly.
• Metallic: White for metal and black for dielectric. Metal is a conductor of electricity, whereas a dielectric material is a non-conductor.
• Ambient Occlusion: Describes areas that are occluded; in other words, areas that are hidden from light.
• Reflection: Identifies which part of the surface is reflective.
• Opacity: Describes the location of the transparent parts of the surface.

In fact, any value (thickness, curvature, etc.) that you can think of to describe a surface, can be stored in a texture. You just look up the relevant fragment in the texture using the UV coordinates and use the value recovered. That’s one of the bonuses of writing your own renderer. You can choose what maps to use and how to apply them.

You can use all of these textures in the fragment shader, and the geometry doesn’t change.

Note: A displacement or height map can change geometry. You’ll read about displacement in Chapter 11, “Tessellation and Terrains.”

Materials

Not all models have textures. For example, the train you rendered earlier in the book has different material groups that specify a color instead of using a texture.

Take a look at cottage1.mtl in the Models folder. This is the file that describes the visual aspects of the cottage model. In the previous chapter, you loaded the diffuse texture using map_Kd from LowPolyHouse.mtl, and you experimented with material groups in Blender in Chapter 2, “3D Models.”

Each of the groups here has values associated with a property. You’ll be extracting these properties from the file and using them in the fragment shader.

• Ns: Specular exponent (shininess)
• Kd: Diffuse color
• Ks: Specular color

Note: You can find a full list of the definitions at http://paulbourke.net/dataformats/mtl/.

The current .mtl file loads the color using map_Kd, but for experimentation, you’ll switch the rendered cottage file to one that gets its color from the material group and not a texture. Open cottage2.mtl and see that none of the groups, except the glass group, has a map_Kd property. This map_Kd on the glass group is an error by the texture artist that you’ll highlight later in the chapter.

In Renderer.swift, in init(metalView:), change the model from "cottage1.obj" to "cottage2.obj".

The diffuse color won’t be the only material property you’ll be reading. Add a new struct in Common.h to hold the material values:

typedef struct {
  vector_float3 baseColor;
  vector_float3 specularColor;
  float roughness;
  float metallic;
  vector_float3 ambientOcclusion;
  float shininess;
} Material;

There are more material properties available, but these are the most common. For now, you’ll read in baseColor, specularColor and shininess.

Open up Submesh.swift and create a new property to hold the materials:

let material: Material

Note: Your project won’t compile until you’ve initialized material.

At the bottom of Submesh.swift, create a new Material initializer:

private extension Material {
  init(material: MDLMaterial?) {
    self.init()
    if let baseColor = material?.property(with: .baseColor),
      baseColor.type == .float3 {
      self.baseColor = baseColor.float3Value
    }
  }
}

In Submesh.Textures, you read in string values for the textures’ file names from the submesh’s material properties. If there’s no texture available for a particular property, you want to check whether there is a single float value instead. For example, if an object is solid red, you don’t have to go to the trouble of making a texture, you can just use float3(1, 0, 0) to describe the color. You’re reading in the material’s base color.

Add the specular and shininess values to the end of Material’s init(material:):

if let specular = material?.property(with: .specular),
  specular.type == .float3 {
  self.specularColor = specular.float3Value
}
if let shininess = material?.property(with: .specularExponent),
  shininess.type == .float {
  self.shininess = shininess.floatValue
}

In Submesh, in init(mdlSubmesh:mtkSubmesh:), initialize material:

material = Material(material: mdlSubmesh.material)

You’ll now send this material to the shader. This sequence of coding should be familiar to you by now.

In Common.h, add another index to BufferIndices:

BufferIndexMaterials = 14

In Renderer.swift, in draw(in:), add the following below // set the materials here:

var material = submesh.material
renderEncoder.setFragmentBytes(&material,
              length: MemoryLayout<Material>.stride,
              index: Int(BufferIndexMaterials.rawValue))

This sends the materials struct to the fragment shader.

In Shaders.metal, add the following as the second parameter of fragment_main:

constant Material &material [[buffer(BufferIndexMaterials)]],

You’ve now passed the model’s material properties to the fragment shader.

In fragment_main, temporarily comment out the baseColor assignment from the texture file, and add the following:

float3 baseColor = material.baseColor;

Finally, replace the assignments of materialShininess and materialSpecularColor with:

float3 materialSpecularColor = material.specularColor;
float materialShininess = material.shininess;

Build and run, and you’re now loading cottage2 with the colors coming from the Kd values instead of a texture.

As you rotate the cottage, you can see the roof, door and window frames are shiny with strong specular highlights. Open cottage2.mtl, and in both the roof and wood groups, change:

• Ns: 1.0
• Ks: 0.2 0.2 0.2

These changes eliminate the specular highlights for those two groups.

You can now render either a cottage with a color texture, or a cottage without a color texture simply by changing the baseColor assignment in the fragment shader.

As you can see, models have various requirements. Some models need a color texture; some models need a roughness texture; and some models need normal maps. It’s up to you to check conditionally in the fragment function whether there are textures or constant values. You also don’t want to be sending spurious textures to the fragment function if your models don’t use textures. You can fix this dilemma with function constants.

Function specialization

Over the years there has been much discussion about how to render different materials. Should you create separate short fragment shaders for the differences? Or should you have one long “uber” shader with all of the possibilities listed conditionally? Function specialization deals with this problem, and allows you to create one shader that the compiler turns into separate shaders.

When you create the model’s pipeline state, you set the Metal functions in the Metal Shading library, and the compiler packages them up. At this stage, you can create booleans to indicate whether your model’s submesh material requires particular textures.

You can then pass these booleans to the Metal library when you create the shader functions. The compiler will then examine the functions and generate specialized versions of them.

In the shader file, you reference the set of booleans by their index numbers.

Open Submesh.swift and locate the extension where you create the pipeline state.

You’ll first create a set of function constant values that will indicate whether the textures exist for the current submesh material. Create a new method in the extension where you create the pipeline state:

static func makeFunctionConstants(textures: Textures) 
                                  -> MTLFunctionConstantValues {
  let functionConstants = MTLFunctionConstantValues()
  var property = textures.baseColor != nil
  functionConstants.setConstantValue(&property, 
                                     type: .bool, index: 0)
  property = textures.normal != nil
  functionConstants.setConstantValue(&property, 
                                     type: .bool, index: 1)
  return functionConstants
}

MTLFunctionConstantValues is a set that contains two boolean values depending on whether the two textures exist. You defined boolean values here, but the values can be any type specified by MTLDataType. On the GPU side, you’ll soon create boolean constants using the same index values; and in the functions that use these constants, you can conditionally perform tasks.

You’ll use this set when creating the fragment function. At the top of makePipelineState(textures:), add this:

let functionConstants = 
      makeFunctionConstants(textures: textures)

Change the assignment to fragmentFunction to:

let fragmentFunction: MTLFunction?
do {
  fragmentFunction = 
         try library?.makeFunction(name: "fragment_main",
                            constantValues: functionConstants)
} catch {
  fatalError("No Metal function exists")
}

Here, you tell the compiler to create a library of functions using the function constants set. The compiler creates multiple shader functions and optimizes any conditionals in the functions. This makeFunction method throws an exception, so you check to see if the function exists in the project.

Now for the GPU side! Open Shaders.metal. After the import statement, add these constants:

constant bool hasColorTexture [[function_constant(0)]];
constant bool hasNormalTexture [[function_constant(1)]];

These match the constants you just created in the MTLFunctionConstantValues set.

In fragment_main, replace:

float3 baseColor = material.baseColor;

With:

float3 baseColor;
if (hasColorTexture) {
  baseColor = baseColorTexture.sample(textureSampler,
                       in.uv * fragmentUniforms.tiling).rgb;
} else {
  baseColor = material.baseColor;
}

Similarly, change the normalValue assignment to:

float3 normalValue;
if (hasNormalTexture) {
  normalValue = normalTexture.sample(textureSampler,
                       in.uv * fragmentUniforms.tiling).rgb;
  normalValue = normalValue * 2 - 1;
} else {
  normalValue = in.worldNormal;
}
normalValue = normalize(normalValue);

Depending on the constant value, you either read the texture value or the base value. Generally, you should try to avoid conditional branching in GPU functions, but the compiler removes these conditionals when it creates the specialized functions.

One more thing: you don’t want to receive a texture into the function if it doesn’t exist.

Change the function parameters for baseColorTexture and normalTexture to:

texture2d<float> baseColorTexture [[texture(BaseColorTexture), 
                          function_constant(hasColorTexture)]],
texture2d<float> normalTexture [[texture(NormalTexture), 
                          function_constant(hasNormalTexture)]],

Here, you’re telling the shader to check the function constant value and only load the texture if the constant value is true.

Build and run, and you’ll see that your green cottage still renders as it used to using the base material colors.

In Renderer.swift, render cottage1 again and notice that the textures render now. You can use this for error checking too. Render cottage2, and place this at the top of the fragment function:

if (hasColorTexture) {
  return float4(1, 0, 0, 1);
}
return float4(0, 1, 0, 1);

This highlights that cottage2 erroneously has a texture map in the glass group.

Remove that error checking code before continuing.

Physically based rendering

To achieve spectacular scenes, you need to have good textures, but lighting plays an even more significant role. In recent years, the concept of physically based rendering (PBR) has become much more popular than the simplistic Phong shading model. As its name suggests, PBR attempts physically realistic interaction of light with surfaces. Now that Augmented Reality has become part of our lives, it’s even more important to render your models to match their physical surroundings.

The general principles of PBR are:

• Surfaces should not reflect more light than they receive.
• Surfaces can be described with known, measured physical properties.

The Bidirectional Reflectance Distribution Function (BRDF) defines how a surface responds to light. There are various highly mathematical BRDF models for both diffuse and specular, but the most common are Lambertian diffuse; and for the specular, variations on the Cook-Torrance model (presented at SIGGRAPH 1981). This takes into account:

• microfacet slope distribution: You learned about microfacets and how light bounces off surfaces in many directions in Chapter 5, “Lighting Fundamentals.”
• Fresnel: If you look straight down into a clear lake, you can see through it to the bottom, however, if you look across the surface of the water, you only see a reflection like a mirror. This is the Fresnel effect, where the reflectivity of the surface depends upon the viewing angle.
• geometric attenuation: Self-shadowing of the microfacets.

Each of these components have different approximations, or models written by many clever people. It’s a vast and complex topic! In the Resources folder for this chapter, references.markdown contains a few places where you can learn more about physically based rendering and the calculations involved. You’ll also learn some more about BRDF and Fresnel in Chapter 20, “Advanced Lighting.”

Artists generally provide some textures with their models that supply the BRDF values. These are the most common:

• Albedo: You already met the albedo map in the form of the base color map. Albedo is originally an astronomical term describing the measurement of diffuse reflection of solar radiation, but it has come to mean in computer graphics the surface color without any shading applied to it.
• Metallic: A surface is either a conductor of electricity — in which case it’s a metal; or it isn’t a conductor — in which case it’s a dielectric. Most metal textures consist of 0 (black) and 1 (white) values only: 0 for dielectric and 1 for metal.
• Roughness: A grayscale texture that indicates the shininess of a surface. White is rough, and black is smooth. If you have a scratched shiny surface, the texture might consist of mostly black or dark gray with light gray scratch marks.
• Ambient Occlusion: A grayscale texture that defines how much light reaches a surface. For example, less light will reach nooks and crannies.

Included in the starter project is a fragment function that uses a Cook-Torrance model for specular lighting. It takes as input the above textures, as well as the color and normal textures.

Note: The PBR fragment function is an abbreviated version of a function from Apple’s sample code LODwithFunctionSpecialization. This is a fantastic piece of sample code to examine, complete with a gorgeous fire truck model. It uses function constants for creating different levels of detail depending on distance from the camera. As a challenge, you can import the sample’s fire truck into your renderer to see how it looks. Remember, though, that you haven’t yet implemented great lighting and reflection.

PBR workflow

First, change the fragment function to use the PBR calculations. In Submesh.swift, in makePipelineState(textures:), change the name of the referenced fragment function from "fragment_main" to "fragment_mainPBR".

Open PBR.metal. In the File inspector, add the file to the macOS and iOS targets.

Examine fragment_mainPBR. It starts off similar to your previous fragment_main but with a few more texture parameters in the function header. The function extracts values from the textures and calculates the normals the same as previously. For simplicity, it only processes the first light in the lights array.

fragment_mainPBR calls render(Lighting) that works through a Cook-Torrance shading model to calculate the specular highlight. The end of fragment_mainPBR adds the diffuse color — which is the same calculation as your previous shader — to the specular value to produce the final color.

To add all of the PBR textures to your project is quite long-winded, so you’ll only add roughness, however, you can choose to add the others as a challenge, if you’d like.

Open Submesh.swift and create a new property for roughness in Textures:

let roughness: MTLTexture?

In the Submesh.Textures extension, add this to the end of init(material:):

roughness = property(with: .roughness)

In addition to reading in a possible roughness texture, you need to read in the material value too. At the bottom of Material’s init(material:), add:

if let roughness = material?.property(with: .roughness),
  roughness.type == .float3 {
  self.roughness = roughness.floatValue
}

Now you need to change makeFunctionConstants() so that it sets up all the texture function constants. Add this code to the end of the function, before return functionConstants:

property = textures.roughness != nil
functionConstants.setConstantValue(&property, 
                                   type: .bool, index: 2)
property = false
functionConstants.setConstantValue(&property, 
                                   type: .bool, index: 3)
functionConstants.setConstantValue(&property, 
                                   type: .bool, index: 4)

Here, you tell the function constants set whether there is a roughness texture, and you set two constants for metallic and ambient occlusion to false since you’re not reading in either of those textures. On the GPU-side, the function constants are already set up for you in PBR.metal.

In Renderer.swift, in draw(in:), locate where you send the base color and normal textures to the fragment function, then add this afterward:

renderEncoder.setFragmentTexture(submesh.textures.roughness,
                                 index: 2)

Still in Renderer.swift, this time in init(metalView:), change the rendered model to “cube.obj”. Also, remove the rotation from the model.

Find where you set up the camera property, and change the camera distance and target to:

camera.distance = 3
camera.target = [0, 0, 0]

Build and run to see a cube with only an albedo texture applied. This texture has no lighting information baked into it. Textures altering the surface will change the lighting appropriately. Open cube.mtl in the Models group, and remove the # in front of map_tangentSpaceNormal cube-normal. The # is a comment, so the texture won’t load.

Build and run to see the difference when the normal texture is applied.

Again, open cube.mtl, and remove the # in front of map_roughness cube-roughness. Previously, the roughness value in the .mtl file was 1.0, which is completely rough. Open Textures.xcassets and select cube-roughness. Select the image and press the spacebar to preview it. The dark gray values will be smooth and shiny (exaggerated here for effect), and the white mortar between the bricks will be completely rough (not shiny).

Compare the roughness map to the cube’s color and normal maps to see how the model’s UV layout is used for all the textures.

Build and run to see the PBR function in action. Admire how much you can affect how a model looks just by a few textures and a bit of fragment shading.

Channel packing

Later on, you’ll again be using the PBR fragment function for rendering. Even if you don’t understand the mathematics, understand the layout of the function and the concepts used.

When loading models built by various artists, you’re likely going to come up against a variety of standards. Textures may be a different way up; normals might point in a different direction; sometimes you may even find three textures magically contained in a single file, a technique called channel packing. Channel packing is an efficient way of managing external textures.

To understand how it works, open PBR.metal and look at the code where the fragment function reads single floats: roughness, metallic and ambient occlusion. When the function reads the texture for each of these values, it’s only reading the red channel. For example:

roughness = roughnessTexture.sample(textureSampler, in.uv).r;

Available within the roughness file are green and blue channels that are currently unused. As an example, you could use the green channel for metallic and the blue channel for ambient occlusion.

Included in the Resources folder for this chapter is an image named channel-packed.png.

If you have Photoshop or some other graphics application capable of reading individual channels, open this file and inspect the channels.

A different color channel contains each of the words. Similarly, you can load your different grayscale maps to each color channel. If you receive a file like this, you can split out each channel into a different file by hiding channels and saving the new file.

If you’re organizing your maps through an asset catalog, channel packing won’t impact the memory consumption and you won’t gain much advantage. However, some artists do use it for easy texture management.

Challenge

In the Resources folder for this chapter is a fabulous treasure chest model from Demeter Dzadik at Sketchfab.com. Your challenge is to render this model! There are three textures that you’ll load into the asset catalog. Don’t forget to change Interpretation from Color to Data, so the textures don’t load as sRGB.

If you get stuck, you’ll find the finished project in the challenge folder.

The challenge project can also render USDZ files with textures. When Model I/O loads USDZ files, the textures are loaded as MDLTextures instead of string filenames. In the challenge project there is an additional method in Texturable to cope with this. To load these textures, you also have to preload the asset textures when you load the asset in Model, by using asset.loadTextures().

You can download USDZ samples from https://developer.apple.com/augmented-reality/quick-look/ to try. The animated models, such as the toy robot, still won’t work until after you’ve completed the next chapter, but the static models, such as the car, should render well once you scale the model down to [0.1, 0.1, 0.1].

Where to go from here?

The sky’s the limit! Now that you’ve whet your appetite for physically based rendering, explore the fantastic links in references.markdown which you’ll find in the Resources folder. Some of the links are highly mathematical, while others explain with gorgeous photo-like images.

Since you now know how to render almost any model that you can export to .obj or .usd, try downloading models from http://www.sketchfab.com in the glTF format and convert them from glTF to .obj using Blender 2.8.

In Chapter 12, “Environment”, you’ll explore Image Based Lighting with reflection from a skycube texture. You’ll revisit rendering metals with metallic textures at that point.

In good games you generally interact with interesting characters, so in the next chapter, you’ll level up your skills by making your models come to life with animation!