20. Advanced Lighting

Written by Marius Horga

As you’ve progressed through this book, you’ve encountered various lighting and reflection models:

• In Chapter 5, “Lighting Fundamentals,” you started with the Phong reflection model which defines light as a sum of three distinct components: ambient light, diffuse light and specular light.
• In Chapter 7, “Maps and Materials,” you briefly looked at physically based rendering and the Fresnel effect.
• In Chapter 12, “Environment,” you implemented skybox-based reflection and image-based lighting, and you used a Bidirectional Reflectance Distribution Function (BRDF) look-up table.

In this chapter, you’ll learn about global illumination and the famous rendering equation that defines it.

While reflection is possible using the local illumination techniques you’ve seen so far, advanced effects — like refraction, subsurface scattering, total internal reflection, caustics and color bleeding — are only possible with global illumination. By the end of this chapter, you’ll be able to render beautiful content like this:

You’ll start by examining the rendering equation. From there, you’ll move on to reflection and refraction, and you’ll render water in two ways: ray marching and rasterization.

The rendering equation

Two academic papers — one by James Kajiya and the other by David Immel et al. — introduced the rendering equation in 1986. In its raw form, this equation might look intimidating:

Note: A friendlier form of it was written recently by Pixar’s Julian Fong in a tweet. Check references.markdown for more information.

The rendering equation is based on the law of conservation of energy, and in simple terms, it translates to an equilibrium equation where the sum of all source lights must equal the sum of all destination lights:

incoming light + emitted light = transmitted light + outgoing light

If you rearrange the terms of the equilibrium equation, you get the most basic form of the rendering equation:

outgoing light = emitted light + incoming light - transmitted light

The incoming light - transmitted light part of the equation is subject to recursion because of multiple light bounces at that point. That recursion process translates to an integral over a unit hemisphere that’s centered on the normal vector at the point and which contains all the possible values for the negative direction of the incoming light.

Although the rendering equation might be a bit intimidating, think of it like this: all the light leaving an object is what remains from all the lights coming into the object after some of them were transmitted through the object.

The transmitted light can be either absorbed by the surface of the object (material), changing its color; or scattered through the object, which leads to a range of interesting optical effects such as refraction, subsurface scattering, total internal reflection, caustics and so on.

Reflection

Reflection is one of the most common interactions between light and objects. Imagine looking into a mirror. Not only would you see your image being reflected, but you’d also see the reflection of any nearby objects.

Reflection, like any other optical phenomenon, has an equation that depends on three things: the incoming light vector, the incident angle and the normal vector for the surface.

The law of reflection states that the angle at which an incident light hits the surface of an object will be the same as the angle of the light that’s being reflected off the normal.

But enough with the theory for now. Time to have some fun coding!

Getting started

Open the starter playground named AdvancedLighting, and select the 1. reflection playground page. Run the playground, and you’ll see this:

The code in this playground might look familiar to you, as you’ve seen it in the two previous chapters. Next, you’ll add a checkerboard pattern to the plane and get it to reflect onto the sphere.

Drawing a checkerboard pattern

To draw a pattern on the plane, you first you need to have a way of identifying objects within the scene by comparing their proximity to the camera based on distance.

Inside the Resources folder for this playground page, open Shaders.metal, and create two constants to identify the two objects in the scene:

constant float PlaneObj = 0.0;
constant float SphereObj = 1.0;

Then, in distToScene, after this line:

float dts = distToSphere(r, s);

Add this:

float object = (dtp > dts) ? SphereObj : PlaneObj;

Here, you check whether the distance to the plane is greater than the distance to the sphere, and you hold the result in object.

Include this information in the function return. Replace return dist; with:

return float2(dist, object);

Run the playground to verify the image hasn’t changed.

The kernel function compute is where you’re raymarching the scene. In a for loop, you iterate over a considerable number of samples and update the ray color until you attain enough precision. It’s in this code block that you’ll draw the pattern on the plane.

Inside the for loop, locate:

float2 dist = distToScene(cam.ray);

distToScene returns the closest object in dist.y.

Immediately after that line, add this:

float closestObject = dist.y;

After hit = true;, add this code:

// 1
if (closestObject == PlaneObj) {
  // 2
  float2 pos = cam.ray.origin.xz;
  pos *= 0.1;
  // 3
  pos = floor(fmod(pos, 2.0));
  float check = mod(pos.x + pos.y, 2.0);
  // 4
  col *= check * 0.5 + 0.5;
}

Going through the code:

1. Build the checkerboard if the selected object is the plane.

2. Get the position of the camera ray in the horizontal XZ plane since you’re interested in intersecting the floor plane only.

3. Create squares. You first alternate between 0s and 1s on both X and Z axes by applying the modulo operator.

   At this point, you have a series of pairs containing either 0s or 1s or both. Next, add the two values together from each pair, and apply the modulo operator again.

   If the sum is 2, roll it back to 0; otherwise, it will be 1.

4. Apply color. Initially, it’s a solid white color. Multiply by 0.5 to tone it down, and add 0.5 back, so you can have both white and grey squares.

If you were to run the playground now, you’d notice a red flag for the missing mod function. However, before adding the missing function, take a moment to understand why you need to implement a separate modulo operation.

The fmod function, as implemented by the Metal Shading Language, performs a truncated division where the remainder will have the same sign as the numerator:

fmod = numerator - denominator * trunc(numerator / denominator)

A second approach, missing from MSL, is called floored division where the remainder has the same sign as the denominator:

mod = numerator - denominator * floor(numerator / denominator)

These two approaches could have entirely different results. When calculating pos, the values need to alternate between 0s and 1s, so taking the floor of the truncated division is enough. However, when you add the two coordinates to determine the check value on the next line, you need to take the floor of their sum.

Add the new floored division function above the kernel function:

float mod(float x, float y) {
  return x - y * floor(x / y);
}

Run the playground, and you’ll see your checkerboard pattern.

All you need to do now is reflect the checkerboard onto the sphere.

Add a new reflection function above compute:

Camera reflectRay(Camera cam, float3 n, float eps) {
  cam.ray.origin += n * eps;
  cam.ray.dir = reflect(cam.ray.dir, n);
  return cam;
}

The MSL standard library provides a reflect() function that takes the incoming ray direction and intersecting surface normal as arguments and returns the outgoing (reflected) ray direction. This convenience function returns the Camera object, not just its ray direction.

In compute, after the if (closestObject == PlaneObj) block, but inside the if (dist.x < eps) block, add this code:

float3 normal = getNormal(cam.ray);
cam = reflectRay(cam, normal, eps);

This gets the normal where the camera ray intersects an object, and reflects it at that point. You move the ray away from the surface, along the normal and not along the ray direction as you might have expected because that could be almost parallel to the surface. You only move away a small distance eps that’s precise enough to tell you when there’s not a hit anymore.

The bigger eps is, the fewer steps you need to hit the surface, so the faster your tracing is; but it’ll also be less accurate. You can play with various values for eps until you find a balance between precision and speed that satisfies your needs.

Run the playground, and you’ll see this:

You’re successfully reflecting the checkerboard onto the sphere, but the sky is not reflecting. This is because in the starter code you used the boolean hit, which stops and breaks out of the loop when the ray first hits any object.

That’s not true anymore, because now you need the ray to keep hitting objects for reflection.

Replace this code:

if (!hit) {
  col = mix(float3(.8, .8, .4), float3(.4, .4, 1.), 
            cam.ray.dir.y);
} else {
  float3 n = getNormal(cam.ray);
  float o = ao(cam.ray.origin, n);
  col = col * o;
}

With this:

col *= mix(float3(0.8, 0.8, 0.4), float3(0.4, 0.4, 1.0), 
           cam.ray.dir.y);

You’re now adding the sky color to the scene color globally, not just when a ray failed to hit an object in the scene. You can optionally remove the ao() function and the two lines in compute where hit appears since you’re not using them anymore.

Run the playground, and you’ll see the sky is now also reflected on the sphere and the floor.

You can spin the camera a little bit to make the reflection look more interesting. Add this parameter to the kernel function:

constant float &time [[buffer(0)]]

And replace this line:

float3 camPos = float3(15.0, 7.0, 0.0);

With this:

float3 camPos = float3(sin(time) * 15.0,
                       sin(time) * 5.0 + 7.0,
                       cos(time) * 15.0);

Run the playground, and you’ll see the same image but now nicely animated.

Refraction

Refraction is another common interaction between light and objects that you often see in nature. While it’s true that most objects in nature are opaque — thus absorbing most of the light they get — the few objects that are translucent, or transparent, allow for the light to propagate through them.

The law of refraction is a little more complicated than mere equality between the incoming and outgoing light vector angles.

Refraction is dictated by Snell’s law which states that the ratio of angles equals the reversed ratio of indices of refraction:

The index of refraction (IOR) is a constant that defines how fast light propagates through various media. IOR is defined as the speed of light in a vacuum divided by the phase velocity of light in that particular medium.

Note: There are published lists with IOR values for various media but the ones that interest us here are that of air (IOR = 1) and that of water (IOR = 1.33). https://en.wikipedia.org/wiki/List_of_refractive_indices

If you’re trying to find the angle for the refracted light vector through water, for example, all you need to know is the incoming light vector angle which you can use from the reflected light vector, and then you can divide that by the IOR for water (since IOR for air is 1 and does not affect the calculation):

sin(theta2) = sin(theta1) / 1.33

Time for more coding!

Open the 2. refraction playground page and run it. The code is the same as the previous section, so you’ll see the same animation. Inside the Resources folder for this playground page open Shaders.metal.

You first need to have a way of knowing when the ray is inside the sphere, as you only do refraction in that case. Add this code to the kernel compute function, before the for loop:

bool inside = false;

In the first part of this chapter, you identified objects, so you now know when the ray hits the sphere. This means that you can change the sign of the distance depending on whether the ray enters the sphere, or leaves it. As you know from previous chapters, a negative distance means you’re inside the object you are sending your ray towards.

Locate:

float2 dist = distToScene(cam.ray);

Add this line below it:

dist.x *= inside ? -1.0 : 1.0;

This adjusts the x value to reflect whether you are inside the sphere or not. Next, you need to adjust the normals. Delete this line:

float3 normal = getNormal(cam.ray);

Now locate this line:

if (dist.x < eps) {

And after that line, add the normal definition back:

float3 normal = getNormal(cam.ray) * (inside ? -1.0 : 1.0);

You now have a normal that points outward when outside the sphere and inward when you’re inside the sphere.

Move the following line so that it is inside the inner if block because you only want the plane to be reflective from now on:

cam = reflectRay(cam, normal, eps);

After the inner if block, add an else block where you make the sphere refractive:

// 1
else if (closestObject == SphereObj) {
  inside = !inside;
  // 2
  float ior = inside ? 1.0 / 1.33 : 1.33;
  cam = refractRay(cam, normal, eps, ior);
}

Going through the code:

1. Check whether you’re inside the sphere. On the first intersection, the ray is now inside the sphere, so turn inside to true and do the refraction. On the second intersection, the ray now leaves the sphere, so turn inside to false, and refraction no longer occurs.
2. Set the index of refraction (IOR) based on the ray direction. IOR for water is 1.33. The ray is first going air-to-water, then it’s going water-to-air in which case the IOR becomes 1 / 1.33.

Add this function above the kernel compute function:

Camera refractRay(Camera cam, float3 n, float eps, float ior) {
  cam.ray.origin -= n * eps * 2.0;
  cam.ray.dir = refract(cam.ray.dir, n, ior);
  return cam;
}

The MSL standard library also provides a refract() function, so you’re just building a convenience function around it. You subtract the distance this time because the ray is inside the sphere.

You also double the eps value, which is enough to move far enough inside to avoid another collision.

If it were still the old value, the ray might stop and consider it another collision with the object since eps was defined precisely for this purpose: precision. Doubling it will make the ray pass just over the point that was already a collision point before.

Run the playground, and you’ll see the sphere now being refractive.

Raytraced water

It’s relatively straightforward to create a cheap, fake water-like effect on the sphere.

Open the 3. water playground page and run it. You’ll see the same animation from the previous section. Inside the Resources folder for this playground page, open Shaders.metal. In the distToScene function, locate:

float object = (dtp > dts) ? SphereObj : PlaneObj;

And add this code afterward:

if (object == SphereObj) {
  // 1
  float3 pos = r.origin;
  pos += float3(sin(pos.y * 5.0), 
                sin(pos.z * 5.0), 
                sin(pos.x * 5.0)) * 0.05;
  // 2
  Ray ray = Ray{pos, r.dir};
  dts = distToSphere(ray, s);
}

Going through the code:

1. Get the ray’s current position, and apply ripples to the surface of the sphere by altering all three coordinates. Use 0.05 to attenuate the altering. A value of 0.001 is not large enough to make an impact, while 0.01 is too much of an impact.
2. Construct a new ray using the altered position as the new ray origin while preserving the old direction. Calculate the distance to the sphere using this new ray.

In the kernel compute function, replace this line:

cam.ray.origin += cam.ray.dir * dist.x;

With this:

cam.ray.origin += cam.ray.dir * dist.x * 0.5;

You added an attenuation factor of 0.5 to make the animation slower but more precise.

Run the playground, and you’ll see a water-like ball.

For more realistic water you need a flat, larger surface, with reflection, refraction and a Fresnel effect. You are going to do that next.

Rasterized water

From this point on, and until the end of the chapter, you’ll work on adapting an exemplary algorithm for creating realistic water developed by Michael Horsch in 2005 (for more information, see references.markdown). This realistic water algorithm is purely based on lighting and its optical properties, as opposed to having a water simulation based on physics.

Here’s the plan on how you’ll proceed:

1. Create a large horizontal quad that will be the surface of the water.
2. Render the scene to a reflection texture.
3. Use a clipping plane to limit what geometry you render.
4. Distort the reflection using a normal map to create ripples on the surface.
5. Render the scene to a refraction texture.
6. Apply the Fresnel effect so that the dominance of each texture will change depending on the viewing angle.
7. Add smoothness to the water depth visibility using a depth texture.

Ready? It’s going to be a wild ride but stick around until the end, because you won’t want to miss this.

Open the starter project named Water. Build and run, and you’ll see this:

Because the water construction is reasonably complex, the project has been stripped to the bare minimum.

• RendererDraws.swift: Contains methods for separate draw calls for the house, terrain and skybox. Each of these has separate shaders and requires its own pipeline state.
• RendererExtension.swift: Contains texture and model loading methods, as well as the camera control. Using your mouse or trackpad, you can drag the camera around a stationary point. You can raise and lower the camera along its Y-axis with the mouse scroll wheel or with a two-finger swipe gesture on the trackpad. On iOS, use a drag or pinch gestures instead.
• RenderPass.swift: Contains the necessary details for a render-to-texture pass. It includes the render pass descriptor, color attachment texture and depth attachment texture.

1. Create the water surface

First, you’ll create a plane for the water surface. In Renderer.swift, add these properties to Renderer:

lazy var water: MTKMesh = {
  do {
    let mesh = Primitive.plane(device: Renderer.device)
    let water = try MTKMesh(mesh: mesh, device: Renderer.device)
    return water
  } catch let error {
    fatalError(error.localizedDescription)
  }
}()

var waterTransform = Transform()
var waterPipelineState: MTLRenderPipelineState!

You create the water mesh from a plane primitive; a transform object, so you can position, rotate and scale the mesh; and a render pipeline state for the water plane.

In buildPipelineState(), at the end of the do block, add this code:

// water pipeline state
descriptor.vertexFunction = 
  library.makeFunction(name: "vertex_water")
descriptor.fragmentFunction = 
  library.makeFunction(name: "fragment_water")
descriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
descriptor.vertexDescriptor = 
  MTKMetalVertexDescriptorFromModelIO(water.vertexDescriptor)
try waterPipelineState = 
  device.makeRenderPipelineState(descriptor: descriptor)

With this code, you build the water pipeline state using a reconfigured render pass descriptor that uses its own shader functions and vertex descriptor.

In RendererDraws.swift, add this new method to render the water:

func renderWater(renderEncoder: MTLRenderCommandEncoder) {
  renderEncoder.pushDebugGroup("water")
  renderEncoder.setRenderPipelineState(waterPipelineState)
  renderEncoder.setVertexBuffer(water.vertexBuffers[0].buffer, 
                                offset: 0, index: 0)
  uniforms.modelMatrix = waterTransform.matrix
  renderEncoder.setVertexBytes(&uniforms,
                     length: MemoryLayout<Uniforms>.stride,
                     index: Int(BufferIndexUniforms.rawValue))
  for submesh in water.submeshes {
    renderEncoder.drawIndexedPrimitives(type: .triangle,
             indexCount: submesh.indexCount,
             indexType: submesh.indexType,
             indexBuffer: submesh.indexBuffer.buffer,
             indexBufferOffset: submesh.indexBuffer.offset)
  }
  renderEncoder.popDebugGroup()
}

This configures the render encoder by setting its render pipeline state, vertex buffer and uniforms. For each submesh, you issue a draw call as you’ve done in other chapters.

Back in Renderer.swift, in draw(in:), add this code before ending the render encoding:

renderWater(renderEncoder: renderEncoder)

Next, create the water shaders. In the Shaders group, create a new Metal file named Water.metal. Remember to add it to both the macOS and iOS targets.

Add this code to the newly created file:

#import "Common.h"

struct VertexIn {
  float4 position [[attribute(0)]];
  float2 uv [[attribute(2)]];
};

struct VertexOut {
  float4 position [[position]];
  float2 uv;
};

vertex VertexOut 
     vertex_water(const VertexIn vertex_in [[stage_in]],
                  constant Uniforms &uniforms
                             [[buffer(BufferIndexUniforms)]]) {
  VertexOut vertex_out;
  float4x4 mvp = uniforms.projectionMatrix * uniforms.viewMatrix
                         * uniforms.modelMatrix;
  vertex_out.position = mvp * vertex_in.position;
  vertex_out.uv = vertex_in.uv;
  return vertex_out;
}

fragment float4 
     fragment_water(VertexOut vertex_in [[stage_in]]) {
  return float4(0.0, 0.3, 0.5, 1.0);
}

This is a minimal configuration for rendering the water surface quad and giving it a bluish color.

Build and run, and you’ll see this image with your new water plane.

2. The reflection render pass

The water plane should reflect its surroundings. In Chapter 12, “Environment,” you reflected the skybox onto objects, but this time you’re also going to reflect the house and terrain on the water.

You’re going to render the scene to a texture from a point underneath the water pointing upwards. You’ll then take this texture and render it flipped on the water surface.

To do this, you’ll first create a new render pass.

In Renderer.swift, add this new property to Renderer:

let reflectionRenderPass: RenderPass

In init(metalView:), before calling super.init(), add this:

reflectionRenderPass = RenderPass(name: "reflection",
                                  size: metalView.drawableSize)

RenderPass will create a render pass descriptor with a texture and a depth texture.

You need reflectionRenderPass to update the size of these textures on resize of the view, so in mtkView(_:drawableSizeWillChange:), add:

reflectionRenderPass.updateTextures(size: size)

Now that you have set up the render pass, create an encoder, and render the whole scene to the reflection render pass texture.

In draw(in:), add this code below // Water render:

// 1
let reflectEncoder = 
  commandBuffer.makeRenderCommandEncoder(
         descriptor: reflectionRenderPass.descriptor)!
reflectEncoder.setDepthStencilState(depthStencilState)
// 2
reflectionCamera.transform = camera.transform
reflectionCamera.transform.position.y = -camera.transform.position.y
reflectionCamera.transform.rotation.x = -camera.transform.rotation.x
uniforms.viewMatrix = reflectionCamera.viewMatrix
// 3
renderHouse(renderEncoder: reflectEncoder)
renderTerrain(renderEncoder: reflectEncoder)
renderSkybox(renderEncoder: reflectEncoder)
reflectEncoder.endEncoding()

Going through the code:

1. Create a new render command encoder for reflection, and set its depth stencil state.
2. This is the unique part about rendering reflection. You’re using a separate camera specially for reflection, and you set its position to the main camera’s negated values because you want the camera to be below the surface of the water and capture what’s above the surface of the water.
3. Render all of the elements of the scene before ending the rendering encoding.

Note: You can build and run at this point, capture the GPU frame, and check that you have two RenderCommandEncoders. The first RenderCommandEncoder is the texture captured from the reflectionCamera position. It’s half-size because you don’t need the reflection texture to be completely accurate. Save on memory where you can.

If you build and run the project again, it’ll look as it did before — which is nice, but where’s that reflection?

In RendererDraws.swift, add this line to renderWater(renderEncoder:) before the for loop:

renderEncoder.setFragmentTexture(reflectionRenderPass.texture, 
                                 index: 0)

This sends the texture from the reflection render pass to the water shader.

Now, in Water.metal, add a new parameter to the fragment function:

texture2d<float> reflectionTexture [[texture(0)]]

Replace the return line with this code:

// 1
constexpr sampler s(filter::linear, address::repeat);
// 2
float width = float(reflectionTexture.get_width() * 2.0);
float height = float(reflectionTexture.get_height() * 2.0);
float x = vertex_in.position.x / width;
float y = vertex_in.position.y / height;
float2 reflectionCoords = float2(x, 1 - y);
// 3
float4 color = reflectionTexture.sample(s, reflectionCoords);
color = mix(color, float4(0.0, 0.3, 0.5, 1.0), 0.3);
return color;

Going through the code:

1. Create a new sampler with linear filtering and repeat addressing mode.
2. Determine the reflection coordinates which will use an inverted Y value because the reflected image is a mirror of the scene above the water surface. Notice you multiplied by 2.0. You did this because the texture is only half-size.
3. Sample the color from the reflection texture, and mix it a little with the previous bluish color the water plane had before.

Build and run, and you’ll see the house, terrain and sky reflected on the water surface.

Nice. But move the camera slightly up the Y-axis by scrolling the mouse wheel or by swiping with two fingers on the trackpad. Oh no! What happened to the reflection?

On the left is the result you see after moving the camera up its Y-axis; on the right is the reflection texture from the reflection render pass. If you want, you can see this texture in the GPU debugger.

As the main camera moves up the Y-axis, the reflection camera moves down the Y-axis to below the terrain surface which blocks the view to the sky. You could temporarily solve this by culling the terrain’s back faces when you render, but this will only introduce other rendering artifacts. A better way of dealing with this issue is to clip the geometry you don’t want to render.

3. Clipping planes

A clipping plane, as its name suggests, clips the scene using a plane. It’s hardware accelerated, meaning that if geometry is not within the clip range, the GPU immediately discards it and doesn’t put it through the rasterizer.

This technique is also a significant performance boost as half of the geometry will not need to get processed by the shaders anymore.

For the reflection texture, you only need to render half the scene, flip it, and add it to the final render.

Placing the clipping plane in the center of the view ensures that only half the scene geometry is rendered.

In Common.h, add a new member to Uniforms:

vector_float4 clipPlane;

In Skybox.metal, add a new member to the VertexOut struct:

float clip_distance [[clip_distance]] [1];

Notice the new MSL attribute, [[clip_distance]], which is one of the built-in attributes exclusively used by vertex shaders. The clip_distance built-in attribute is an array of distances, and the [1] argument represents its size — a 1 in this case because you declared this variable as one float only.

Note: You can read more about this in the Metal Shading Language specification, in Section 5.7.1 “Vertex-Fragment Signature Matching.”

You may have noticed that here, and in Shaders.metal, that FragmentIn is a duplicate of VertexOut. [[clip_distance]] is a vertex-only attribute, so you duplicate the struct so the fragment shader can use it.

Keep FragmentIn the same as VertexOut, by adding the following new member to the FragmentIn struct:

float clip_distance;

In vertex_skybox, add this line before the return statement:

vertex_out.clip_distance[0] = 
    dot(uniforms.modelMatrix * vertex_in.position,
        uniforms.clipPlane);

Any negative result in vertex_out.clip_distance[0] will result in the vertex being clipped.

Repeat the last three steps in Shaders.metal as well: modify the VertexOut struct, the FragmentIn struct and vertex_main accordingly.

Back in Renderer.swift, add this line in draw(in:) below // Water render:

var clipPlane = float4(0, 1, 0, 0.1)
uniforms.clipPlane = clipPlane

This sets a clipping plane before rendering the reflected scene. The clipping plane XYZ is a direction vector that denotes the clipping direction. The last component, 0.1 in this case, represents the distance from the clipping plane that you want to capture — something close to 0 will do.

Below // Main render, add this code:

clipPlane = float4(0, -1, 0, 6)
uniforms.clipPlane = clipPlane

Build and run the project, and you’ll see this:

Whoops! What happened? It seems the fourth component of the clipping plane, the value 6, still affects part of the scene that you want rendered. Increase the clipping distance from 6 to 100:

clipPlane = float4(0, -1, 0, 100)

Build and run, and your water reflection appears smooth and calm. As you move the camera up and down, the reflection is now consistent.

Still water, no matter how calming, isn’t realistic. Time to give that water some ripples.

4. Rippling normal maps

The project contains a tiling normal map for the water ripples.

You’ll tile this map across the water and move it, perturbing the water normals, which will make the water appear to ripple.

In Renderer.swift, and this new property to Renderer:

var waterTexture: MTLTexture?

Inside the declaration for the water variable, add this line before the return to load up the normal texture:

waterTexture = 
    try Renderer.loadTexture(imageName: "normal-water.png")

In RendererDraws.swift, add this code to renderWater(renderEncoder:) before the for loop:

renderEncoder.setFragmentTexture(waterTexture, index: 2)
renderEncoder.setFragmentBytes(&timer, 
                               length: MemoryLayout<Float>.size, 
                               index: 3)

Here, you send the normal map and a timer variable to the fragment shader. Renderer’s draw(in:) maintains this timer.

In Water.metal, add two new parameters for the texture, and timer to fragment_water:

texture2d<float> normalTexture [[texture(2)]],
constant float& timer [[buffer(3)]]

Add this code before the line where color is defined:

// 1
float2 uv = vertex_in.uv * 2.0;
// 2
float waveStrength = 0.1;
float2 rippleX = float2(uv.x + timer, uv.y);
float2 rippleY = float2(-uv.x, uv.y) + timer;
float2 ripple = 
    ((normalTexture.sample(s, rippleX).rg * 2.0 - 1.0) +
     (normalTexture.sample(s, rippleY).rg * 2.0 - 1.0)) 
      * waveStrength;
reflectionCoords += ripple;
// 3  
reflectionCoords = clamp(reflectionCoords, 0.001, 0.999);

Going through the code:

1. Get the texture coordinates and multiply them by a tiling value. For 2 you get huge, ample ripples while for something like 16 you get quite small ripples. Pick a value that suits your needs.
2. Calculate ripples by distorting the texture coordinates with the timer value. Only grab the R and G values from the sampled texture because they are the U and V coordinates that determine the horizontal plane where the ripples will be, so the B value is not important here. waveStrength is an attenuator value, that gives you weaker or stronger waves.
3. Clamp the reflection coordinates to eliminate anomalies around the margins of the screen.

Build and run, and you’ll see gorgeous ripples on the water surface.

5. The refraction render pass

Refraction is very similar to reflection, except that you only need to preserve the part of the scene where the Y coordinate is negative.

In Renderer.swift, add this new property to Renderer:

let refractionRenderPass: RenderPass

In init(metalView:), initialize refractionRenderPass as a new render pass, just as you did for the reflection render pass:

refractionRenderPass = RenderPass(name: "refraction",
                                  size: metalView.drawableSize)

Update the size of these textures on resize of the view. In mtkView(_:drawableSizeWillChange:), add:

refractionRenderPass.updateTextures(size: size)

In draw(in:) add this code before // Main render:

// 1
clipPlane = float4(0, -1, 0, 0.1)
uniforms.clipPlane = clipPlane
uniforms.viewMatrix = camera.viewMatrix
// 2
let refractEncoder = 
    commandBuffer.makeRenderCommandEncoder(
          descriptor: refractionRenderPass.descriptor)!
refractEncoder.setDepthStencilState(depthStencilState)
renderHouse(renderEncoder: refractEncoder)
renderTerrain(renderEncoder: refractEncoder)
renderSkybox(renderEncoder: refractEncoder)
refractEncoder.endEncoding()

Going through the code:

1. Set the clip plane back to -1 since the camera is now again up and pointing down towards the water.
2. Create the refraction render encoder and render all the elements of the scene again.

In RendererDraws.swift, add this line to renderWater(renderEncoder:) before the for loop:

renderEncoder.setFragmentTexture(refractionRenderPass.texture, 
                                 index: 1)

In Water.metal, add a new parameter to fragment_water:

texture2d<float> refractionTexture [[texture(1)]]

Add this line below the one where you define reflectionCoords:

float2 refractionCoords = float2(x, y);

Similarly, add this line below the one for reflection:

refractionCoords += ripple;

And once more for this line to prevent edge anomalies:

refractionCoords = clamp(refractionCoords, 0.001, 0.999);

Finally, replace this line:

float4 color = reflectionTexture.sample(s, reflectionCoords);

With this line:

float4 color = refractionTexture.sample(s, refractionCoords);

Build and run the project, and you’ll see that the reflection on the water surface is gone, and instead, you have refraction through the water.

There’s one more visual enhancement you can make to your water to make it more realistic: adding rocks and grime. Fortunately, the project already has a texture that can simulate this.

In Shaders.metal, in fragment_terrain, uncomment the section under //uncomment this for pebbles.

Build and run the project, and you’ll now see a darker texture underwater.

The holy grail of realistic water, however, is having a Fresnel effect that harmoniously combines reflection and refraction based on the viewing angle.

6. The Fresnel effect

The Fresnel effect is a concept you’ve met with in previous chapters. As you may remember, the viewing angle plays a significant role in the amount of reflection you can see. What’s new in this chapter is that the viewing angle also affects refraction but in inverse proportion:

• The steeper the viewing angle is, the weaker the reflection and the stronger the refraction.
• The shallower the viewing angle is, the stronger the reflection and the weaker the refraction.

The Fresnel effect in action:

In Common.h, add a new member to Uniforms:

vector_float3 cameraPosition;

In Renderer, add this line to draw(in:) before // Water render:

uniforms.cameraPosition = camera.transform.position

In Water.metal, add two new members to VertexOut:

float3 worldPosition;
float3 toCamera;

Also, add this code to vertex_water before the return statement:

vertex_out.worldPosition = 
    (uniforms.modelMatrix * vertex_in.position).xyz;
vertex_out.toCamera = uniforms.cameraPosition - vertex_out.worldPosition;

Replace this line at the end of the fragment shader:

float4 color = refractionTexture.sample(s, refractionCoords);

With this code:

float3 viewVector = normalize(vertex_in.toCamera);
float mixRatio = dot(viewVector, float3(0.0, 1.0, 0.0));
float4 color = 
    mix(reflectionTexture.sample(s, reflectionCoords),
        refractionTexture.sample(s, refractionCoords),
        mixRatio);

A ratio of 0.5 would mean that reflection and refraction are mixed equally.

Build and run the project.

Move the camera around and notice how reflection predominates for a small viewing angle while refraction predominates when the viewing angle is getting closer to 90 degrees (perpendicular to the water surface).

7. Add smoothness using a depth texture

The light propagation varies for different transparent media, but for water, the colors with longer wavelengths (closer to infrared) quickly fade away as the light ray goes deeper. The bluish colors (closer to ultraviolet) tend to be visible at greater depths because they have shorter wavelengths.

At very shallow depths, however, most light should still be visible. You’ll make the water look smoother as depth gets smaller. You can improve the way the water surface blends with the terrain by using a depth map.

In RendererDraws.swift, add this code to renderWater(renderEncoder:) before the for loop:

renderEncoder.setFragmentTexture(
                  refractionRenderPass.depthTexture,
                  index: 4)

As well as sending the refraction texture from the refraction render pass, you’re now sending the depth texture too.

In Renderer, add this code to buildPipelineState() before you create waterPipelineState:

guard let attachment = descriptor.colorAttachments[0] 
      else { return }
attachment.isBlendingEnabled = true
attachment.rgbBlendOperation = .add
attachment.sourceRGBBlendFactor = .sourceAlpha
attachment.destinationRGBBlendFactor = .oneMinusSourceAlpha

Here, you configure the blending options on the color attachment just as you did back in Chapter 10, “Fragment post-processing.”

In Water.metal add a new parameter to fragment_water:

depth2d<float> depthMap [[texture(4)]]

Add this code before the ripples code:

float proj33 = far / (far - near); 
float proj43 = proj33 * -near;     
float depth = depthMap.sample(s, refractionCoords);
float floorDistance = proj43 / (depth - proj33);
depth = vertex_in.position.z;
float waterDistance = proj43 / (depth - proj33);
depth = floorDistance - waterDistance;

Note: See references.markdown for an explanation of converting a non-linear depth buffer value to a linear depth value.

Finally, change the alpha channel so that blending goes into effect. Add this line before the return statement:

color.a = clamp(depth * 0.75, 0.0, 1.0);

Build and run the project, and you’ll now see a smoother blending of the water with the terrain.

Challenge

Your challenge for this chapter is to use the normal map from the ripples section, this time to add surface lighting.

At the end of this section, you’ll render an image like this:

All the work you need to do will be in fragment_water. Here are the steps:

1. Get the normal vector as you did for the ripples. In this case, however, you also need to get the B channel, but since that will be the “up” vector, you need to swap G and B so you’ll end up with a normal vector of the form RBG.
2. Normalize both the normal and the direction of light before calculating the reflected light using the reflect function as you did before.
3. Compute the specular light using the reflected light and the view vector which you used for the Fresnel effect. Apply a higher power until you are pleased with the intensity of the specular light.
4. Add the specular light to the final color.

The completed project is in the Challenge folder.

Where to go from here?

You’ve certainly made a splash with this chapter! If you want to explore more about water rendering, the references.markdown file for this chapter contains links to interesting articles and videos.

This concludes the series of chapters using raymarching. But don’t worry — rendering is far from over. In the next chapter, “Metal Performance Shaders,” you’ll get to learn about the newly added feature, Metal for accelerating ray tracing, as well as dip your toes into image processing.