# 11. Maps & Materials Written by Marius Horga & Caroline Begbie

In the previous chapter, you set up a simple Phong lighting model. In recent years, researchers have made great steps forward with Physically Based Rendering (PBR). PBR attempts to accurately represent real-world shading, where the amount of light leaving a surface is less that the amount the surface receives. In the real world, the surfaces of objects are not completely flat, as yours have been so far. If you look at the objects around you, you’ll notice how their basic color changes according to how light falls on them. Some objects have a smooth surface, and some have a rough surface. Heck, some might even be shiny metal!

In this chapter, you’ll find out how to use material groups to describe a surface, and how to design textures for micro detail.

## Normal Maps

The following example best describes normal maps:

On the left, there’s a lit cube with a color texture. On the right, there’s the same low-poly cube with the identical color texture and lighting. The only difference is that the cube on the right also has a second texture applied to it known as a normal map. This normal map makes it appear as if the cube is a high-poly model with lots of nooks and crannies. In truth, these high-end details are just an illusion.

For this illusion to work, the model needs a texture, like this:

All models have normals that stick out perpendicular to each face. A cube has six faces, and the normal for each face points in a different direction. Also, each face is flat. If you wanted to create the illusion of bumpiness, you’d need to change a normal in the fragment shader.

Look at the following image. On the left is a flat surface with normals in the fragment shader. On the right, you see perturbed normals. The texels in a normal map supply the direction vectors of these normals through the RGB channels.

Now, look at this single brick split out into the red, green and blue channels that make up an RGB image.

Each channel has a value between 0 and 1, and you generally visualize them in grayscale as it’s easier to read color values. For example, in the red channel, a value of 0 is no red at all, while a value of 1 is full red. When you convert 0 to an RGB color `(0, 0, 0)`, the result is black. On the opposite spectrum, `(1, 1, 1)` is white. And in the middle, you have `(0.5, 0.5, 0.5)`, which is mid-gray. In grayscale, all three RGB values are the same, so you only need to refer to a grayscale value by a single float.

Take a closer look at the edges of the red channel’s brick. Look at the left and right edges in the grayscale image. The red channel has the darkest color where the normal values of that fragment should point left `(-X, 0, 0)`, and the lightest color where they should point right `(+X, 0, 0)`.

Now look at the green channel. The left and right edges have equal value but are different for the top and bottom edges of the brick. The green channel in the grayscale image has darkest for pointing down `(0, -Y, 0)` and lightest for pointing up `(0, +Y, 0)`.

Finally, the blue channel is mostly white in the grayscale image because the brick — except for a few irregularities in the texture — points outward. The edges of the brick are the only places where the normals should point away.

Note: Normal maps can be either right-handed or left-handed. Your renderer will expect positive `y` to be up, but some apps will generate normal maps with positive `y` down. To fix this, you can take the normal map into Photoshop and invert the green channel.

The base color of a normal map — where all normals are “normal” (orthogonal to the face) — is `(0.5, 0.5, 1)`.

This is an attractive color but was not chosen arbitrarily. RGB colors have values between 0 and 1, whereas a model’s normal values are between -1 and 1. A color value of 0.5 in a normal map translates to a model normal of 0. The result of reading a flat texel from a normal map should be a `z` value of 1 and the `x` and `y` values as 0. Converting these values `(0, 0, 1)` into the colorspace of a normal map results in the color `(0.5, 0.5, 1)`. This is why most normal maps appear bluish.

### Creating Normal Maps

To create successful normal maps, you need a specialized app. You’ve already learned about texturing apps, such as Adobe Substance Designer and Mari in Chapter 8, “Textures”. Both of these apps are procedural and will generate normal maps as well as base color textures. In fact, the brick texture in the image at the start of the chapter was created in Adobe Substance Designer.

### Tangent Space

To render with a normal map texture, you send it to the fragment function in the same way as a color texture, and you extract the normal values using the same UVs. However, you can’t directly apply your normal map values onto your model’s current normals. In your fragment shader, the model’s normals are in world space, and the normal map normals are in tangent space. Tangent space is a little hard to wrap your head around. Think of the brick cube with all its six faces pointing in different directions. Now think of the normal map with all the bricks the same color on all the six faces.

## The Starter App

➤ In Xcode, open the starter project for this chapter.

## Using Normal Maps

➤ In the Models ▸ Cottage group, open cottage1.mtl in a text editor.

``````map_tangentSpaceNormal cottage-normal
map_Kd cottage-color
``````
``````let normal: MTLTexture?
``````
``````normal = property(with: .tangentSpaceNormal)
``````
``````NormalTexture = 1
``````
``````encoder.setFragmentTexture(
submesh.textures.normal,
index: NormalTexture.index)
``````
``````texture2d<float> normalTexture [[texture(NormalTexture)]]
``````
``````float3 normal;
if (is_null_texture(normalTexture)) {
normal = in.worldNormal;
} else {
normal = normalTexture.sample(
textureSampler,
in.uv * params.tiling).rgb;
}
normal = normalize(normal);
return float4(normal, 1);
``````

``````return float4(normal, 1);
``````

### 1. Load Tangents and Bitangents

➤ Open VertexDescriptor.swift, and look at `MDLVertexDescriptor`’s `defaultLayout`. Here, you tell the vertex descriptor that there are normal values in the attribute named `MDLVertexAttributeNormal`.

``````let (mdlMeshes, mtkMeshes) = try! MTKMesh.newMeshes(
asset: asset,
device: Renderer.device)
``````
``````var mtkMeshes: [MTKMesh] = []
let mdlMeshes =
asset.childObjects(of: MDLMesh.self) as? [MDLMesh] ?? []
_ = mdlMeshes.map { mdlMesh in
withAttributeNamed: MDLVertexAttributeNormal,
creaseThreshold: 1.0)
mtkMeshes.append(
try! MTKMesh(
mesh: mdlMesh,
device: Renderer.device))
}
``````

``````mdlMesh.addNormals(
withAttributeNamed: MDLVertexAttributeNormal,
creaseThreshold: 1.0)
``````
``````mdlMesh.addTangentBasis(
forTextureCoordinateAttributeNamed:
MDLVertexAttributeTextureCoordinate,
tangentAttributeNamed: MDLVertexAttributeTangent,
bitangentAttributeNamed: MDLVertexAttributeBitangent)
``````
``````Tangent = 4,
Bitangent = 5
``````
``````TangentBuffer = 3,
BitangentBuffer = 4,
``````
``````vertexDescriptor.attributes[Tangent.index] =
MDLVertexAttribute(
name: MDLVertexAttributeTangent,
format: .float3,
offset: 0,
bufferIndex: TangentBuffer.index)
vertexDescriptor.layouts[TangentBuffer.index]
= MDLVertexBufferLayout(stride: MemoryLayout<float3>.stride)
vertexDescriptor.attributes[Bitangent.index] =
MDLVertexAttribute(
name: MDLVertexAttributeBitangent,
format: .float3,
offset: 0,
bufferIndex: BitangentBuffer.index)
vertexDescriptor.layouts[BitangentBuffer.index]
= MDLVertexBufferLayout(stride: MemoryLayout<float3>.stride)
``````

### 2. Send Tangent and Bitangent Values to the GPU

➤ Open Model.swift, and in `render(encoder:uniforms:params:)`, locate `for mesh in meshes`.

``````for (index, vertexBuffer) in mesh.vertexBuffers.enumerated() {
encoder.setVertexBuffer(
vertexBuffer,
offset: 0,
index: index)
}
``````

### 3. Convert Tangent and Bitangent Values to World Space

Just as you converted the model’s normals to world space, you need to convert the tangents and bitangents to world space in the vertex function.

``````float3 tangent [[attribute(Tangent)]];
float3 bitangent [[attribute(Bitangent)]];
``````
``````float3 worldTangent;
float3 worldBitangent;
``````
``````.worldTangent = uniforms.normalMatrix * in.tangent,
.worldBitangent = uniforms.normalMatrix * in.bitangent
``````

### 4. Calculate the New Normal

Now that you have everything in place, it’ll be a simple matter to calculate the new normal.

``````normal = normal * 2 - 1;
``````
``````normal = float3x3(
in.worldTangent,
in.worldBitangent,
in.worldNormal) * normal;
``````
``````float3 color = phongLighting(
normal,
in.worldPosition,
params,
lights,
baseColor
);
``````
``````float3 normalDirection = normalize(in.worldNormal);
``````

### Other Texture Map Types

Normal maps are not the only way of changing a model’s surface. There are other texture maps:

## Materials

Not all models have textures. For example, the train you rendered earlier in the book has different material groups that specify a color instead of using a texture.

``````typedef struct {
vector_float3 baseColor;
vector_float3 specularColor;
float roughness;
float metallic;
float ambientOcclusion;
float shininess;
} Material;
``````
``````let material: Material
``````
``````private extension Material {
init(material: MDLMaterial?) {
self.init()
if let baseColor = material?.property(with: .baseColor),
baseColor.type == .float3 {
self.baseColor = baseColor.float3Value
}
}
}
``````
``````if let specular = material?.property(with: .specular),
specular.type == .float3 {
self.specularColor = specular.float3Value
}
if let shininess = material?.property(with: .specularExponent),
shininess.type == .float {
self.shininess = shininess.floatValue
}
self.ambientOcclusion = 1
``````
``````material = Material(material: mdlSubmesh.material)
``````
``````MaterialBuffer = 14
``````
``````var material = submesh.material
encoder.setFragmentBytes(
&material,
length: MemoryLayout<Material>.stride,
index: MaterialBuffer.index)
``````
``````constant Material &_material [[buffer(MaterialBuffer)]],
``````
``````Material material = _material;
``````
``````float3 baseColor;
if (is_null_texture(baseColorTexture)) {
baseColor = in.color;
} else {
baseColor = baseColorTexture.sample(
textureSampler,
in.uv * params.tiling).rgb;
}
``````
``````if (!is_null_texture(baseColorTexture)) {
material.baseColor = baseColorTexture.sample(
textureSampler,
in.uv * params.tiling).rgb;
}
``````
``````float3 color = phongLighting(
normal,
in.worldPosition,
params,
lights,
material
);
``````
``````Material material
``````
``````Material material
``````
``````float3 baseColor = material.baseColor;
``````
``````float materialShininess = material.shininess;
float3 materialSpecularColor = material.specularColor;
``````

## Physically Based Rendering (PBR)

To achieve spectacular scenes, you need to have good textures, but shading plays an even more significant role. In recent years, the concept of PBR has replaced the simplistic Phong shading model. As its name suggests, PBR attempts physically realistic interaction of light with surfaces. Now that Augmented Reality has become part of our lives, it’s even more important to render your models to match their physical surroundings.

### PBR Workflow

First, change the fragment function to use the PBR calculations.

``````let roughness: MTLTexture?
``````
``````roughness = property(with: .roughness)
``````
``````if let roughness = material?.property(with: .roughness),
roughness.type == .float3 {
self.roughness = roughness.floatValue
}
``````
``````encoder.setFragmentTexture(
submesh.textures.roughness,
index: 2)
``````
``````camera.distance = 3.5
camera.target = .zero
``````

## Channel Packing

Later, you’ll be using the PBR fragment function for rendering. Even if you don’t understand the mathematics, understand the layout of the function and the concepts used.

``````roughness = roughnessTexture.sample(textureSampler, in.uv).r;
``````

## Challenge

In the resources folder for this chapter is a fabulous helmet model from Malopolska’s Virtual Museums collection at sketchfab.com. Your challenge is to render this model. There are five textures that you’ll load into the asset catalog. Don’t forget to change Interpretation from Color to Data, so the textures don’t load as sRGB.

## Where to Go From Here?

Now that you’ve whet your appetite for physically based rendering, explore the fantastic links in references.markdown, which you’ll find in the resources folder for this chapter. Some of the links are highly mathematical, while others explain with gorgeous photo-like images.

Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.