12. Environment

Written by Caroline Begbie

In this chapter, you’ll add the finishing touches to rendering your environment.

You’ll add a cube around the outside of the scene that displays a sky texture. That sky texture will then light the models within the scene, making them appear as if they belong.

Look at the following comparison of two renders.

This comparison demonstrates how you can use the same shader code, but change the sky image to create different lighting environments.

Getting started

Open the starter project for this chapter. For the most part, this project is similar to the engine you created in Chapter 9, “Scene Graph.” There are, however, a few notable changes:

• You can add a fragment function string name to the Model initializer, letting you test different rendering styles for different props.
• The racing car asset and the Textures asset catalog includes metallic and ambient occlusion maps.

Build and run the project, and you’ll see the car rendered using physically based shading, as described in Chapter 7, “Maps and Materials”. Chapter 8, “Character Animation”, and Chapter 9, “Scene Graph” used this same shading, but sneakily used a scaling factor in the fragment shader to lighten the shadows.

Aside from the darkness of the lighting, there are some glaring problems with the render:

• All metals, such as the metallic wheel hubs, aren’t looking shiny. Pure metals reflect their surroundings, and there are currently no surroundings to reflect.
• As you move around the car using the keyboard, notice where the light doesn’t directly hit the car, the color is black. This happens because the app doesn’t provide any ambient light. Later on in the chapter, you’ll use the skylight as global ambient light.

Note: If you’re using macOS, use the keyboard keys WASD to move, and use the QE or right and left arrow keys to rotate.

The skybox

Currently, the sky is a single color, which looks unrealistic. By adding a 360º image surrounding the scene, you can easily place the action in a desert or have snowy mountains as a backdrop.

To do this, you’ll create a skybox cube that surrounds the entire scene.

This skybox cube is the same as an ordinary model, but instead of viewing it from the outside, the camera is at the center of the cube looking out. You’ll texture the cube with a cube texture, which gives you a cheap way of creating a complete environment.

You may think the cube will be distorted at the corners, but as you’ll see, each fragment of the cube will render at an effectively infinite distance, and no distortion will occur. Cube maps are much easier to create than spherical ones and are hardware optimized.

Create a new Swift file for the skybox class named Skybox.swift. Remember to add the file to both the iOS and macOS targets. This class holds the geometry of the cube, and also the necessary pipeline and sky textures.

Replace the default code with:

import MetalKit

class Skybox {
  
  let mesh: MTKMesh
  var texture: MTLTexture?
  let pipelineState: MTLRenderPipelineState
  let depthStencilState: MTLDepthStencilState?
  
  init(textureName: String?) {
    
  }
}

You created the Skybox class and some properties:

• mesh: A cube that you’ll create using a Model I/O primitive.
• texture: A cube texture of the name given in the initializer.
• pipelineState: The skybox needs a simple vertex and fragment function, therefore it needs its own pipeline.
• depthStencilState: Each pixel of the skybox will be positioned at the very edge of normalized clip space. Renderer’s current depth stencil state only adds the fragment if the fragment is less than the current depth value. The skybox depth stencil should test less than or equal to the current depth value. You’ll see why shortly.

Your class currently doesn’t compile because you need to initialize those properties. Add the following code to init(textureName:):

let allocator = MTKMeshBufferAllocator(device: Renderer.device)
let cube = MDLMesh(boxWithExtent: [1,1,1], segments: [1, 1, 1],
                   inwardNormals: true, 
                   geometryType: .triangles,
                   allocator: allocator)
do {
  mesh = try MTKMesh(mesh: cube,
                     device: Renderer.device)
} catch {
  fatalError("failed to create skybox mesh")
}

Here, you create a cube mesh. Notice that you set the normals to face inwards. That’s because the whole scene will appear to be inside the cube.

Add a new static method to create the pipeline state:

private static func 
    buildPipelineState(vertexDescriptor: MDLVertexDescriptor) 
                              -> MTLRenderPipelineState {
  let descriptor = MTLRenderPipelineDescriptor()
  descriptor.colorAttachments[0].pixelFormat = 
       Renderer.colorPixelFormat
  descriptor.depthAttachmentPixelFormat = .depth32Float
  descriptor.vertexFunction = 
        Renderer.library?.makeFunction(name: "vertexSkybox")
  descriptor.fragmentFunction = 
        Renderer.library?.makeFunction(name: "fragmentSkybox")
  descriptor.vertexDescriptor = 
        MTKMetalVertexDescriptorFromModelIO(vertexDescriptor)
  do {
    return 
      try Renderer.device.makeRenderPipelineState(
          descriptor: descriptor)
  } catch {
    fatalError(error.localizedDescription)
  }
}

There’s nothing new here. You create a pipeline state with the cube’s Model I/O vertex descriptor, pointing to two shader functions that you’ll write shortly.

Add a new static method to create the depth stencil state:

private static func buildDepthStencilState() 
            -> MTLDepthStencilState? {
  let descriptor = MTLDepthStencilDescriptor()
  descriptor.depthCompareFunction = .lessEqual
  descriptor.isDepthWriteEnabled = true
  return Renderer.device.makeDepthStencilState(
      descriptor: descriptor)
}

This creates the depth stencil state with the less than or equal comparison method mentioned earlier.

Complete the initialization by adding this to the end of init(textureName:):

pipelineState = 
    Skybox.buildPipelineState(vertexDescriptor: cube.vertexDescriptor)
depthStencilState = Skybox.buildDepthStencilState()

Your project should now compile.

Rendering the skybox

Still in Skybox.swift, create a new method to perform the skybox rendering:

func render(renderEncoder: MTLRenderCommandEncoder, uniforms: Uniforms) {

}

Add these render encoder commands to the new method:

renderEncoder.pushDebugGroup("Skybox")
renderEncoder.setRenderPipelineState(pipelineState)
// renderEncoder.setDepthStencilState(depthStencilState)
renderEncoder.setVertexBuffer(mesh.vertexBuffers[0].buffer, 
                              offset: 0, index: 0)

Here, you set up the render encoder with all of the properties you initialized. Leave the depth stencil state line commented out for the moment.

After this code, set up uniforms:

var viewMatrix = uniforms.viewMatrix
viewMatrix.columns.3 = [0, 0, 0, 1]
var viewProjectionMatrix = uniforms.projectionMatrix 
                               * viewMatrix
renderEncoder.setVertexBytes(&viewProjectionMatrix, 
                      length: MemoryLayout<float4x4>.stride, 
                      index: 1)

As a reminder, when you render a scene, you multiply each model’s matrix with the view matrix and the projection matrix. When you move through the scene, it appears as if the camera is moving through the scene, but in fact, the whole scene is moving around the camera. You don’t want the skybox to move, so you zero out column 3 of viewMatrix to remove the camera’s translation.

However, you do still want the skybox to rotate with the rest of the scene, and also render with projection, so you multiply the view and projection matrices and send them to the GPU.

Add the following the previous code to do the draw:

let submesh = mesh.submeshes[0]
renderEncoder.drawIndexedPrimitives(type: .triangle,
  indexCount: submesh.indexCount,
  indexType: submesh.indexType,
  indexBuffer: submesh.indexBuffer.buffer,
  indexBufferOffset: 0)

The skybox shader functions

In the Metal Shaders group, add a new Metal file named Skybox.metal. Again, add this file to both the macOS and iOS targets.

Add the following code to the new file:

#import "Common.h"

struct VertexIn {
  float4 position [[ attribute(0) ]];
};

struct VertexOut {
  float4 position [[ position ]];
};

The structs are simple so far — you need a position in and a position out.

Add the shader functions:

vertex VertexOut vertexSkybox(const VertexIn in [[stage_in]],
                         constant float4x4 &vp [[buffer(1)]]) {
  VertexOut out;
  out.position = (vp * in.position).xyww;
  return out;
}

fragment half4 fragmentSkybox(VertexOut in [[stage_in]]) {
  return half4(1, 1, 0, 1);
}

Two very simple shaders — the vertex function moves the vertices to the projected position, and the fragment function returns yellow. This is a temporary color, which is startling enough that you’ll be able to see where the skybox renders.

Notice in the vertex function that you swizzled the xyzw position to xyww. To place the sky as far away as possible, it needs to be at the very edge of NDC.

During the change from clip space to NDC, the coordinates are all divided by w during the perspective divide stage.

This will now result in the z coordinate being 1, which will ensure that the skybox renders behind everything else within the scene.

The following diagram shows the skybox in camera space rotated by 45º. After projection and the perspective divide, the vertices will be flat against the far NDC plane.

Integrating the skybox into the scene

Open Scene.swift, and add a new property to Scene:

var skybox: Skybox?

Open SkyScene.swift. This is the scene that contains the car and the ground. Add a skybox at the top of setupScene():

skybox = Skybox(textureName: nil)

You haven’t written the code for the skybox texture yet, but soon you’ll set it up so that nil will give you a physically simulated sky, and a texture name will load a sky texture.

In Renderer.swift, in draw(in:), toward the end of the method, add the following after the // render skybox comment:

scene.skybox?.render(renderEncoder: renderEncoder, 
                     uniforms: scene.uniforms)

It may seem odd that you’re rendering the skybox last, after rendering the models, when it’s going to be the object that’s behind everything else. Remember early-Z testing from Chapter 3, “The Rendering Pipeline”: when objects are rendered, most of the skybox fragments will be behind them and will fail the depth test. Therefore, it’s more efficient to render the skybox last.

You’ve now integrated the skybox into the rendering process.

Build and run to see the new yellow sky.

As you can see from this image, not all of the sky is yellow. As you rotate the scene, the yellow sky flickers and shows the blue of the metal view’s clear color. This happens because the current depth stencil state is from Renderer, and it’s comparing new fragments to less than the current depth buffer. The skybox coordinates are right on the edge, so sometimes they’re equal to the edge of clip space.

In Skybox.swift, in render(renderEncoder:uniforms:), uncomment renderEncoder.setDepthStencilState(depthStencilState), and build and run the app again.

This time, the depth comparison is correct, and the sky is the solid yellow returned from the skybox fragment shader.

Procedural skies

Yellow skies might be appropriate on a different planet, but how about a procedural sky? A procedural sky is one built out of various parameters such as weather conditions and time of day. Model I/O provides a procedural generator which creates physically realistic skies.

Before exploring this API further, open and run skybox.playground in the projects folder for this chapter. This scene contains only a ground plane and a skybox. Use your mouse or trackpad to reorient the scene, and experiment with the sliders under the view to see how you can change the sky depending on:

• turbidity: Haze in the sky. 0.0 is a clear sky. 1.0 spreads the sun’s color.
• sun elevation: How high the sun is in the sky. 0.5 is on the horizon. 1.0 is overhead.
• upper atmosphere scattering: Atmospheric scattering influences the color of the sky from reddish through orange tones to the sky at midday.
• ground albedo: How clear the sky is. 0 is clear, while 10 can produce intense colors. It’s best to keep turbidity and upper atmosphere scattering low if you have high albedo.

As you move the sliders, the result is printed in the debug console so you can record these values for later use. See if you can create a sunrise:

This playground uses Model I/O to create an MDLSkyCubeTexture. From this, the playground creates an MTLTexture and applies this as a cube texture to the sky cube. You’ll do this in your starter project.

Cube textures

Cube textures are similar to the 2D textures that you’ve already been using. 2D textures map to a quad and have two texture coordinates, whereas cube textures consist of six 2D textures: one for each face of the cube. You sample the textures with a 3D vector.

The easiest way to load a cube texture into Metal is to use Model I/O’s MDLTexture initializer. When creating cube textures, you can arrange the images in various combinations:

Alternatively, you can create a cube texture in an asset catalog and load the six images there. Back in your starter project, open Textures.xcassets in the Textures group. This is pre-loaded with a sky texture complete with mipmaps.

The sky should always render on the base mipmap level 0, but you’ll see later how to use the other mipmaps.

Aside from there being six images to one texture, moving the images into the asset catalog and creating the mipmaps is the same process as described in Chapter 7, “Maps and Materials”.

Adding the procedural sky

You’ll use these sky textures shortly, but for now, you’ll add a procedural sky to the scene in the starter project. In Skybox.swift, add these properties to Skybox:

struct SkySettings {
  var turbidity: Float = 0.28
  var sunElevation: Float = 0.6
  var upperAtmosphereScattering: Float = 0.1
  var groundAlbedo: Float = 4
}

var skySettings = SkySettings() 

You can use the values from the appropriate sliders in the playground if you prefer.

Now add the following method:

func loadGeneratedSkyboxTexture(dimensions: int2) -> MTLTexture? {
  var texture: MTLTexture?
  let skyTexture = MDLSkyCubeTexture(name: "sky",
        channelEncoding: .uInt8,
        textureDimensions: dimensions,
        turbidity: skySettings.turbidity,
        sunElevation: skySettings.sunElevation,
        upperAtmosphereScattering: 
               skySettings.upperAtmosphereScattering,
        groundAlbedo: skySettings.groundAlbedo)
  do {
    let textureLoader = 
          MTKTextureLoader(device: Renderer.device)
    texture = try textureLoader.newTexture(texture: skyTexture, 
                                           options: nil)
  } catch {
    print(error.localizedDescription)
  }
  return texture
}

This uses the settings to create the sky texture using Model I/O. That’s all there is to creating a procedurally generated sky texture! Call this method at the end of init(textureName:):

if let textureName = textureName {
  
} else {
  texture = loadGeneratedSkyboxTexture(dimensions: [256, 256])
}

You’ll add the if part of this conditional shortly and load the named texture. The nil option provides a default sky.

To render the texture, you’ll change the skybox shader function and ensure that the texture gets to the GPU.

Still in Skybox.swift, in render(renderEncoder:uniforms:), add this before the draw call:

renderEncoder.setFragmentTexture(texture, 
                    index: Int(BufferIndexSkybox.rawValue))

The starter project already has the necessary enum indices set up in Common.h for the skybox textures.

In Skybox.metal, add this to the VertexOut struct:

float3 textureCoordinates;

Generally, when you load a model, you also load its texture coordinates. However, when sampling texels from a cube texture, instead of using a uv coordinate, you use a 3D vector.

For example, a vector from the center of any cube passes through the far top left corner at [-1, 1, 1].

Conveniently, even though the skybox’s far top-left vertex position is [-0.5, 0.5, 0.5], it still lies on the same vector, so you can use the skybox vertex position for the texture coordinates. You don’t even need to normalize this vector to read the cube texture.

Add this to vertexSkybox before the return:

out.textureCoordinates = in.position.xyz;

Change the fragment shader to:

fragment half4 
        fragmentSkybox(VertexOut in [[stage_in]],
                       texturecube<half> cubeTexture 
                           [[texture(BufferIndexSkybox)]]) {
  constexpr sampler default_sampler(filter::linear);
  half4 color = cubeTexture.sample(default_sampler, 
                                   in.textureCoordinates);
  return color;
}

Accessing a cube texture is similar to accessing a 2D texture. You mark the cube texture as texturecube in the shader function parameters and sample it using the textureCoordinates vector that you set up in the vertex function.

Build and run, and you now have a realistic sky, simulating physics:

Custom sky textures

As mentioned earlier, you can use your own 360º sky textures. The textures included in the starter project were downloaded from http://hdrihaven.com, a great place to find environment maps. The HDRI has been converted into six tone mapped sky cube textures before adding them to the asset catalog.

Note: If you want to create your own skybox textures or load HDRIs (high dynamic range images), you can find out how to do it in references.markdown included with this chapter’s files.

Loading a cube texture is almost the same as loading a 2D texture. Open Texturable.swift, and examine the method loadCubeTexture(imageName:). Just as with loadTexture(imageName:), you can load either a cube texture from the asset catalog or one 2D image consisting of the six faces vertically.

In Skybox.swift, add this to the end of the file so you can access the cube texture loading method:

extension Skybox: Texturable {}

At the end of init(textureName:), in the first half of the incomplete conditional, use the bound textureName to load the cube texture.

do {
  texture = try Skybox.loadCubeTexture(imageName: textureName)
} catch {
  fatalError(error.localizedDescription)
}

In SkyScene.swift, you want to load the sky texture, so change the skybox initialization to:

skybox = Skybox(textureName: "sky")

Build and run to see your new sky texture.

Notice that as you move about the scene, although the skybox rotates with the rest of the scene, it does not reposition.

You should be careful that the sky textures you use don’t have objects that appear to be close, as they will always appear to stay at the same distance from the camera. Sky textures should be for background only. This sky texture is not perfect as it has utility poles behind the car.

Reflection

Now that you have something to reflect, you can easily implement reflection of the sky onto the car. When rendering the car, all you have to do is take the camera view direction, reflect it about the surface normal, and sample the skycube along the reflected vector for the fragment color for the car.

The starter project has the capability of choosing which fragment shader to render a particular object, so in SkyScene.swift, change car’s initialization to:

let car = Model(name: "racing-car.obj", 
                fragmentFunctionName: "skyboxTest")

In Shaders.metal, create a new fragment shader at the end of the file:

fragment float4 skyboxTest(VertexOut in [[stage_in]],
  constant FragmentUniforms &fragmentUniforms
    [[buffer(BufferIndexFragmentUniforms)]],
  texturecube<float> skybox [[texture(BufferIndexSkybox)]]) {
  return float4(0, 1, 1, 1);
}

You’ll temporarily use this shader for rendering the car. The cyan return color is to ensure that you have the shader working.

Build and run to check that the car renders out fully in cyan and that your new shader is working.

To send the skybox texture to the car’s new fragment shader, in Skybox.swift, in Skybox, add this new method:

func update(renderEncoder: MTLRenderCommandEncoder) {  
  renderEncoder.setFragmentTexture(texture,
                      index: Int(BufferIndexSkybox.rawValue))
}

You’ll add other skybox textures to this method soon.

In Renderer.swift, in draw(in:), locate // render models, and add this just before the scene.renderables for loop:

scene.skybox?.update(renderEncoder: renderEncoder)

The sky texture is now available to the new fragment function.

You now calculate the camera’s reflection vector about the surface normal to get a vector for sampling the skybox texture. To get the camera’s view vector, you subtract the fragment world position from the camera position.

In Shaders.metal, replace the code in skyboxTest with:

float3 viewDirection = in.worldPosition.xyz - 
                              fragmentUniforms.cameraPosition;
float3 textureCoordinates = reflect(viewDirection, 
                                    in.worldNormal);

Here, you calculate the view vector and reflect it about the surface normal to get the vector for the cube texture coordinates.

Now, add this code:

constexpr sampler defaultSampler(filter::linear);
float4 color = skybox.sample(defaultSampler, 
                             textureCoordinates);
float4 copper = float4(0.86, 0.7, 0.48, 1);
color = color * copper;
return color;

Here, you sample the skybox texture for a color and multiply it by a copper color.

Build and run. The car now appears to be made of beautifully shiny copper. Walk about the scene and around the car to see the sky reflected on the car.

Note: This is not a true reflection; you’re only reflecting the sky texture. If you place any objects in the scene, they won’t be reflected. You can see this at the rear of the car where the road from the skybox texture instead of the ground node is reflected on the passenger side. However, this reflection is a fast and easy effect, and often sufficient.

Image-based lighting

At the beginning of the chapter, there were two problems with the original car render. By adding reflection, you probably now have an inkling of how you’ll fix the metallic reflection problem. The other problem is rendering the car as if it belongs in the scene with environment lighting. IBL or Image Based Lighting is one way of dealing with this problem.

Using the sky image you can extract lighting information. For example, the parts of the car that face the sun in the sky texture should shine more than the parts that face away. The parts that face away shouldn’t be entirely dark but should have ambient light filled in from the sky texture.

Epic Games developed a technique for Fortnite, which they adapted from Disney’s research, and this has become the standard technique for IBL in games today. If you want to be as physically correct as possible, there’s a link to their article on how to achieve this included with the references.markdown for this chapter.

You’ll be doing an approximation of their technique, making use of Model I/O for the diffuse.

Diffuse reflection

Light comes from all around us. Sunlight bounces around and colors reflect. When rendering an object, you should take into account the color of the light coming from every direction.

This is somewhat of an impossible task, but you can use convolution to compute a cube map called an irradiance map from which you can extract lighting information. You won’t need to know the mathematics behind this: Model I/O comes to the rescue again!

Included in the starter project is a fragment shader already set up in IBL.metal. Look at fragment_IBL in that file. The shader reads all of the possible textures for a Model and sets values for:

• base color
• normal
• roughness
• metallic (this should be 0 or 1)
• ambient occlusion

All of these maps for the car will interact with the sky texture and provide a beautiful render. Currently, the fragment function returns just the base color.

To use the fragment function, in SkyScene.swift, change car’s initialization to:

let car = Model(name: "racing-car.obj", 
                fragmentFunctionName: "fragment_IBL")

Build and run, and you’ll get a basic flat color render:

The diffuse reflection for the car will come from a second texture derived from the sky texture. In Skybox.swift, add a new property to hold this diffuse texture:

var diffuseTexture: MTLTexture?

To create the diffuse irradiance texture, add this temporary method:

func loadIrradianceMap() {
  // 1
  let skyCube = 
       MDLTexture(cubeWithImagesNamed: ["cube-sky.png"])!
  // 2
  let irradiance = 
       MDLTexture.irradianceTextureCube(with: skyCube, 
                              name: nil, dimensions: [64, 64], 
                              roughness: 0.6)
  // 3                           
  let loader = MTKTextureLoader(device: Renderer.device)
  diffuseTexture = try! loader.newTexture(texture: irradiance, 
                                          options: nil)
}

Going through this code:

1. Model I/O currently doesn’t load cube textures from the asset catalog, so your project has an image named cube-sky.png with the six faces included in it.
2. Use Model I/O to create the irradiance texture. It doesn’t have to be a large texture, as the diffuse color is spread out.
3. Load the resultant MDLTexture to the diffuseTexture MTLTexture.

To call this method, add the following to the end of init(textureName:):

loadIrradianceMap()

Still in Skybox.swift, in update(renderEncoder:), add the following:

renderEncoder.setFragmentTexture(diffuseTexture,
  index: Int(BufferIndexSkyboxDiffuse.rawValue))

This will send the diffuse texture to the GPU.

In IBL.metal, add the two skybox textures to the parameter list for fragment_IBL:

texturecube<float> skybox [[texture(BufferIndexSkybox)]],
texturecube<float> skyboxDiffuse 
                   [[texture(BufferIndexSkyboxDiffuse)]]

At the end of fragment_IBL, replace the return with:

float4 diffuse = skyboxDiffuse.sample(textureSampler, normal);
return diffuse * float4(baseColor, 1);

The diffuse value doesn’t depend on the angle of view, so you sample the diffuse texture using the surface normal. You then multiply the result by the base color.

Build and run the app. Because of the irradiance convolution, the app may take a minute or so to start. As you walk about the car, you’ll notice it’s slightly brighter where it faces the sun.

Click the Capture GPU frame icon to enter the GPU Debugger, and look at the generated irradiance map.

You can choose the different faces below the texture.

Instead of generating the irradiance texture each time, you can save the irradiance map to a file and load it from there. Included in the resources folder for this chapter is a project named IrradianceGenerator. You can use this app to generate your irradiance maps.

In the starter project, there’s an irradiance map named irradiance.png to match the sky textures. It’s time to switch to using this irradiance map for the diffuse texture instead of generating it.

In Skybox.swift, in init(textureName:), locate where you load texture in the do...catch, and add this immediately after loading texture:

diffuseTexture = 
     try Skybox.loadCubeTexture(imageName: "irradiance.png")

Remove loadIrradianceMap() and the code where you called that method at the end of init(textureName).

Build and run to ensure you have the same result as you did before swapping to the prebuilt irradiance map.

Specular reflection

The irradiance map provides the diffuse and ambient reflection, but the specular reflection is a bit more difficult.

You may remember from Chapter 5, “Lighting Fundamentals,” that, whereas the diffuse reflection comes from all light directions, specular reflection depends upon the angle of view and the roughness of the material.

See the difference in the light’s reflection:

In Chapter 7, “Maps and Materials”, you had a foretaste of physically based rendering using the Cook-Torrance microfacet specular shading model. This model is defined as:

Where you provide the light direction (l), view direction (v) and the half vector (h) between l and v. As described in Chapter 7, “Maps and Materials”, the functions are:

• D: Geometric slope distribution
• F: Fresnel
• G: Geometric attenuation

Just as with the diffuse light, to get the accuracy of the incoming specular light, you need to take many samples, which is impractical in real-time rendering. Epic Games’s approach in their paper Real Shading in Unreal Engine 4 is to split up the shading model calculation. They prefilter the sky cube texture with the geometry distribution for various roughness values. For each roughness level, the texture gets smaller and blurrier, and you can store these prefiltered environment maps as different mipmap levels in the sky cube texture.

Note: In the resources for this chapter, there’s a project named Specular, which uses the code from Epic Games’s paper. This project takes in six images — one for each cube face — and will generate prefiltered environment maps for as many levels as you specify in the code. The results are placed in a subdirectory of Documents named specular, which you should create before running the project. You can then add the created .png files to the mipmap levels of the sky cube texture in your asset catalog.

The sky texture in the starter project already contains the prefiltered environment maps.

BRDF look-up table

During runtime, you supply a look-up table with the actual roughness of the model and the current viewing angle and receive back the scale and bias for the Fresnel and geometric attenuation contributions to the final color. You can represent this two-dimensional look-up table as a texture that behaves as a two-dimensional array. One axis is the roughness value of the object, and the other is the angle between the normal and the view direction. You input these two values as the UV coordinates and receive back a color. The red value contains the scale, and the green value contains the bias.

The more photorealistic you want your scene to be, the higher the level of mathematics you’ll need to know. In references.markdown, there are links with suggested reading that explain the Cook-Torrance microfacet specular shading model.

The starter project contains functions provided by Epic Games to create the BRDF look-up texture. You can find these in the Utilities group.

In Skybox.swift, add a property for the new texture:

var brdfLut: MTLTexture?

At the end of init(textureName:), call the method supplied in the starter project to build the texture:

brdfLut = Renderer.buildBRDF()

This method will use a compute shader to create a new texture. You’ll find out how to create and use compute shaders in Chapter 16, “Particle Systems”.

In update(renderEncoder:), add the following to send the texture to the GPU:

renderEncoder.setFragmentTexture(brdfLut, 
                  index: Int(BufferIndexBRDFLut.rawValue))

Build and run the app, and click the Capture GPU frame icon to verify the look-up texture created by the BRDF compute shader is available to the GPU.

Notice the texture format is RG16Float. As a float format, this pixel format has a greater accuracy than RGBA8Unorm.

All the necessary information is now on the GPU, so you need to receive the new BRDF look-up texture into the fragment shader and do the shader math.

In IBL.metal, add this as a parameter to fragment_IBL:

texture2d<float> brdfLut [[texture(BufferIndexBRDFLut)]]

At the end of fragment_IBL, before the return, add this:

// 1
float3 viewDirection = in.worldPosition.xyz -
                           fragmentUniforms.cameraPosition;
float3 textureCoordinates = reflect(viewDirection, normal);
// 2
constexpr sampler s(filter::linear, mip_filter::linear);
float3 prefilteredColor = 
        skybox.sample(s, textureCoordinates, 
                      level(roughness * 10)).rgb;
// 3
float nDotV = saturate(dot(normal, normalize(-viewDirection)));
float2 envBRDF = brdfLut.sample(s, float2(roughness, nDotV)).rg;

Going through the code:

1. Calculate the view direction and the view direction reflected about the surface normal. This is the same code as you used earlier for reflection.
2. Read the skybox texture along this reflected vector as you did earlier. Using the extra parameter level(n), you can specify the mip level to read. You sample the appropriate mipmap for the roughness of the fragment.
3. Calculate the angle between the view direction and the surface normal, and use this as one of the UV coordinates to read the BRDF look-up texture. The other coordinate is the roughness of the surface. You receive back the red and green values which you’ll use to calculate the second part of the Cook Torrence equation.

Fresnel reflectance

When light hits an object straight on, some of the light is reflected. The amount of reflection is called Fresnel zero, or F0, and you can calculate this from the material’s index of refraction, or IOR.

When you view an object, at the viewing angle of 90º, the surface becomes nearly 100% reflective. For example, when you look across the water, it’s reflective; but when you look straight down into the water, it’s non-reflective.

Most dielectric (non-metal) materials have an F0 of about 4%, so most rendering engines use this amount as standard. For metals, F0 is the base color.

Add this after the previous code:

float3 f0 = mix(0.04, baseColor.rgb, metallic);
float3 specularIBL = f0 * envBRDF.r + envBRDF.g;

Here, you choose F0 as 0.04 for non-metals and the base color for metals. metallic should be a binary value of 0 or 1, but it’s best practice to avoid conditional branching in shaders, so you use mix(). You then calculate the second part of the rendering equation using the values from the look-up table.

Replace the return line with the following:

float3 specular = prefilteredColor * specularIBL;
float4 color = diffuse * float4(baseColor, 1) 
                  + float4(specular, 1);
return color;

You’re now including the diffuse and base colors. Build and run.

Your car render is almost complete. Non-metals take the roughness value — the seats are matte, and the car paint is shiny.

Metals reflect but take on the base color — the base color of the wheel hubs and the steel bar behind the seats is gray.

Being able to tweak shaders gives you complete power over how your renders look. Because you’re using low dynamic range lighting, the non-metal diffuse color looks a bit dark. You can tweak the color very easily.

Add this to the end of fragment_IBL, right after setting diffuse from the diffuse texture:

diffuse = mix(pow(diffuse, 0.5), diffuse, metallic);

This raises the power of the diffuse value but only for non-metals. Build and run, and the car body is a lot brighter.

The finishing touch will be to add a fake shadow effect. At the rear of the car, the exhausts look as if they are self-lit:

They should be shadowed because they are recessed. This is where ambient occlusion maps come in handy.

Ambient occlusion maps

Ambient occlusion is a technique that approximates how much light should fall on a surface. If you look around you — even in a bright room — where surfaces are very close to each other, they’re darker than exposed surfaces. In Chapter 19, “Advanced Shadows”, you’ll learn how to generate global ambient occlusion using ray marching, but assigning pre-built local ambient occlusion maps to models is a fast and effective alternative.

Apps such as Substance Painter can examine the model for proximate surfaces and produce an ambient occlusion map. This is the AO map for the car, which is included in the starter project.

The white areas on the left, with a color value of 1.0, are UV mapped to the car paint. These are fully exposed areas. When you multiply the final render color by 1.0, it’ll be unaffected. However, you can identify the wheel at the bottom right of the AO map, where the spokes are recessed. Those areas have a color value of perhaps 0.8, which darkens the final render color.

The ambient occlusion map is all set up in the starter project and ready for you to use. In fragment_IBL, just before the final return, add this:

color *= ambientOcclusion;

Build and run. Now compare the exhaust pipes to the previous render.

All of the recessed areas are darker, which gives more natural lighting to the model.

Challenge

On the first page of this chapter is a comparison of the car rendered in two different lighting situations. Your challenge is to create the red lighting scene.

Provided in the resources directory are six cube face png images converted from an HDRI downloaded from HDRIHaven.com.

1. Create an irradiance map using the included IrradianceGenerator project, and import the generated map into the project.
2. Create specular mipmap levels using the included Specular project.
3. Create a new cube texture in the asset catalog.
4. Assign this new cube texture the appropriate generated mipmap images.

There’s no code to change; it’s all imagery! You’ll find the completed project in the challenge directory for this chapter.

Where to go from here?

You’ve dipped a toe into the water of the great sea of realistic rendering, and you’ll read about more advanced concepts of lighting and reflectivity in Chapter 20, “Advanced Lighting.” If you want to explore more about realistic rendering, references.markdown for this chapter contains links to interesting articles and videos.

This chapter did not touch on spherical harmonics, which is an alternative method to using an irradiance texture map for diffuse reflection. Mathematically, you can approximate that irradiance map with 27 floats. Hopefully, the links in references.markdown will get you interested in this amazing technique.

Before you try to achieve the ultimate realistic render, one question you should ask yourself is whether your game will benefit from realism. One way to stand out from the crowd is to create your own rendering style. Games such as Fortnite aren’t entirely realistic and have a style all of their own. Experiment with shaders to see what you can create.