10. Fragment Post-Processing

Written by Marius Horga

Before embarking on complex features like tessellation and instancing, it’s best to start with simple techniques to improve your render quality. After the fragments have been processed in the pipeline, a series of operations are run on the GPU. These operations are sometimes referred to as Per-sample Processing (https://www.khronos.org/opengl/wiki/Per-Sample_Processing), and include: Alpha testing; Depth testing; Stencil testing; Scissor testing; Blending; and Anti-aliasing.

As you go through this chapter, you’ll learn about most of these.

Getting started

In the projects directory for this chapter, open the starter playground. Then, run the playground and you’ll see this image:

When you move closer to the tree — using either the scroll wheel or the two-finger gesture on your trackpad — you’ll notice the leaves have an unpleasant look. In the playground’s Resources folder, take a look at treeColor.png. The area of the texture surrounding the leaf is transparent.

To make the leaves look more natural, you’ll render the white part of the texture as transparent. However, before changing anything, it’s important to understand the difference between transparent objects and those that are translucent.

Alpha testing

An object that is transparent allows light to entirely pass through it. A translucent object, on the other hand, will distort light when it passes through. Objects like water, glass, and plastic are all translucent. Objects can also be opaque. In fact, most objects in nature are opaque, meaning they don’t allow any light to pass through, like trees and rocks.

All digital colors are formed as a combination of the three primary colors: red, green and blue. Hence the color scheme RGB. There’s a fourth component that’s added to the color definition. It’s called alpha and it ranges from 0 (fully transparent) to 1 (fully opaque). A common practice in determining transparency is checking the alpha property and ignoring values below a certain threshold. This is called alpha testing.

Start by creating a toggle key so you can see the difference for when an effect is on or off. In the playground page’s Sources folder, in Renderer.swift, add this property to Renderer:

var transparencyEnabled = false

With this Boolean, you’ll be able to control transparency.

Because the trees will have additional modifications, you need to create different shaders, which means you also need several different pipeline states for each shader combination. Add the following at the top of Renderer:

var treePipelineState: MTLRenderPipelineState! 

In buildPipelineState, just before the end of the do block, add these lines:

// 1
let treeFragmentFunction = 
    library.makeFunction(name: "fragment_tree")
descriptor.fragmentFunction = treeFragmentFunction
// 2
treePipelineState = 
  try device.makeRenderPipelineState(descriptor: descriptor)

Going through this code:

1. You create a function using the name you plan to provide to the Metal library, and you assign it to the pipeline state descriptor; you’ll write the fragment_tree() shader shortly. The playground won’t run correctly until you’ve completed the shader function.
2. You then create the pipeline state for the tree using the provided descriptor.

Next, in draw(in:), right below this line:

// render tree

Add the following code:

renderEncoder.setRenderPipelineState(treePipelineState)
renderEncoder.setFragmentBytes(&transparencyEnabled, 
                  length: MemoryLayout<Bool>.size, index: 0)

This tells the render command encoder to use the pipeline state for the next draw call. You also send the current value of transparencyEnabled to the fragment shader.

In MetalView.swift, add these two methods to MetalView, so the view can capture key presses:

// 1
public override var acceptsFirstResponder: Bool {
  return true
}

public override func keyDown(with event: NSEvent) {
  enum KeyCode: UInt16 {
    case t = 0x11
  }
  
  guard
    let renderer = renderer,
    let keyCode = KeyCode(rawValue: event.keyCode)
  else {return}
  
  // 2
  switch keyCode {
  case .t:
    renderer.transparencyEnabled = !renderer.transparencyEnabled
  }
}

Going through the code:

1. You activate key presses by setting acceptsFirstResponder to true.
2. You check whether the key pressed is a T (0x11 is the hexadecimal code for T) and toggle transparencyEnabled on or off depending on its previous value.

Finally, in Shaders.metal, in the Resources group, add this:

// 1
fragment float4 fragment_tree(VertexOut vertex_in [[stage_in]],
             texture2d<float> texture [[texture(0)]],
             constant bool & transparencyEnabled [[buffer(0)]]) {
  // 2
  constexpr sampler s(filter::linear);
  float4 color = texture.sample(s, vertex_in.uv);
  // 3
  if (transparencyEnabled && color.a < 0.1) {
    discard_fragment();
  }
  // 4
  return color;
}

With this code:

1. The fragment shader gets the processed vertices from the vertex shader, the tree texture and the value of transparencyEnabled from the CPU.
2. It creates a texture sampler with linear filtering and uses it to read the color information from the input texture.
3. If both transparencyEnabled is active and the alpha value of the current pixel is below a 0.1 threshold, it discards this fragment.
4. Otherwise, it returns the color from the sampled texture.

Run the playground again. Click the rendered image so your key strokes will get captured in that window. Then, press the T key a few times to watch the alpha testing (transparency) toggle on and off:

That’s better! If you get closer to the tree, you’ll notice the white background around the leaves is gone.

However, the leaves all look the same color. That’s because this scene lacks a directional light. To fix that, you need to write a new vertex function and configure the tree pipeline state again.

In Shaders.metal, add this:

vertex VertexOut 
         vertex_light(const VertexIn vertex_in [[stage_in]],
                constant float4x4 &mvp_matrix [[buffer(1)]]) {
  VertexOut vertex_out;
  vertex_out.position = mvp_matrix * vertex_in.position;
  vertex_out.uv = vertex_in.uv;
  return vertex_out;
}

This is the same code as vertex_main(), but you’re going to add any new code to this one from now on.

In Renderer.swift, you need to also reference vertex_light in the render pipeline descriptor configuration. Right above this line, in buildPipelineState():

treePipelineState = 
    try device.makeRenderPipelineState(descriptor: descriptor)

Add this code:

let lightVertexFunction = 
     library.makeFunction(name: "vertex_light")
descriptor.vertexFunction = lightVertexFunction

This sets the pipeline to use your new vertex function.

Back in Shaders.metal, in vertex_light, add this code right before return vertex_out;:

// 1
vertex_out.normal = (mvp_matrix 
                       * float4(vertex_in.normal, 0)).xyz;
// 2
float3 light_direction = {1.0, 1.0, -1.0};
float4 light_color = float4(1.0);
// 3
float intensity = dot(normalize(vertex_out.normal),
                      normalize(light_direction));
// 4
vertex_out.color = saturate(light_color * intensity);

Going through this code:

1. You calculate the vertex normal.
2. Set the light direction and light color to custom values.
3. Calculate the light intensity as a dot product of the vertex normal and light direction, just as you did in Chapter 5, “Lighting Fundamentals.”
4. Calculate the vertex color and clamp the resulting value between 0 and 1.

To include the light value, you need to multiply it with the existing texture color, so add this code inside the fragment_tree() function, right above return color;:

color *= vertex_in.color * 2;

Note: You could multiply the color value with various values, including values below 1, and see which one pleases your eye. Here, 2 seems to bring an acceptable brightness level to the color.

Run the playground again. Don’t forget to press the T key to toggle the transparency.

As you can see, the leaves are now showing their face or back depending on how each of them is oriented with respect to the light direction.

Depth testing

Depth testing compares the depth value of the current fragment to one stored in the framebuffer. If a fragment is farther away than the current depth value, this fragment fails the depth test and is discarded since it’s occluded by another fragment. You’ll learn more about depth testing in Chapter 14, “Multipass and Deferred Rendering.”

Stencil testing

Stencil testing compares the value stored in a stencil attachment to a masked reference value. If a fragment makes it through the mask it’s kept; otherwise it’s discarded.

Scissor testing

If you only want to render part of the screen, you can tell the GPU to only render within a particular rectangle. This is much more efficient than rendering the whole screen. The scissor test checks whether a fragment is inside a defined 2D area called the scissor rectangle. If the fragment falls outside of this rectangle, it’s discarded.

In Renderer.swift, add this line in draw(in:), right after the // render tree comment:

renderEncoder.setScissorRect(MTLScissorRect(x: 500, y: 500, 
                                   width: 600, height: 400))

Run the playground again.

Ouch! That’s not right. Only the tree was affected.

Note: Keep in mind that any objects rendered before you set the scissor rectangle are not affected by it.

Move that last line to right after the // render terrain comment, before you do the terrain draw.

Run the playground again.

Much better.

You’ll need to see the entire scene for the rest of the chapter, so comment out that scissor test line.

Alpha blending

Alpha blending is different from alpha testing in that the latter only works with total transparency. In that case, all you have to do is discard fragments. For translucent or partially transparent objects, discarding fragments is not the solution anymore because you want the fragment color to contribute to a certain extent to the existing framebuffer color. You don’t want to just replace it.

The formula for alpha blending is:

Going over this formula:

• Cs: Source color. The current color you just added to the scene.
• Cd: Destination color. The color that already exists in the framebuffer.
• Cb: Final blended color.
• ⍺1 and ⍺2: The alpha (opacity) factors for the source and destination color, respectively.

The final blended color is the result of adding the products between the two colors and their opacity factors. The source color is the fragment color you put in front, and the destination color is the color already existing in the framebuffer.

Often the two factors are the inverse of each other transforming this equation into linear color interpolation:

All right, time to install a glass window in front of the tree.

In Renderer.swift, create new texture and pipeline state properties at the top of the class:

var windowTexture: MTLTexture?
var windowPipelineState: MTLRenderPipelineState! 

Next, add this code right before init(metalView:):

lazy var window: MTKMesh = {
  do {
    let primitive = self.loadModel(name: "plane")!
    let model = try MTKMesh(mesh: primitive,
                            device: device)
    windowTexture = loadTexture(imageName: "windowColor")
    return model
  } catch {
    fatalError()
  }
}()
var windowTransform = Transform()

This code is the same as that for the terrain and tree, however, to quickly recap: you define a new window model using the plane mesh and the windowColor texture, and then you create a new transform matrix for this model.

At the end of init(metalView:), add this code:

windowTransform.scale = [2, 2, 2]
windowTransform.position = [0, 3, 4]
windowTransform.rotation = [-Float.pi / 2, 0, 0]

This code, again, is similar to terrain and tree transforms: First, set the scale to a size factor of 2; then, position the window closer to your eye (the tree is at 6); finally, rotate the window by 90 degrees on the x-axis, so it stands vertically.

Note: The circumference of a circle is 2π, so that means 90º is π / 2 (a quarter of a circle).

Inside buildPipelineState(), at the end of the do block, add these lines:

let windowFragmentFunction = 
  library.makeFunction(name: "fragment_window")
descriptor.fragmentFunction = windowFragmentFunction
descriptor.vertexFunction = lightVertexFunction
windowPipelineState = 
  try device.makeRenderPipelineState(descriptor: descriptor)

Here, you configure the pipeline state descriptor with the appropriate functions and then build the pipeline state using this descriptor. You’ll write the fragment_window() shader in a moment.

Toward the end of draw(in:), add this code right before renderEncoder.endEncoding():

// render window
renderEncoder.setRenderPipelineState(windowPipelineState)
modelViewProjectionMatrix = camera.projectionMatrix * 
  camera.viewMatrix * windowTransform.matrix
renderEncoder.setVertexBytes(&modelViewProjectionMatrix, 
                     length: MemoryLayout<float4x4>.stride, 
                     index: 1)
renderEncoder.setVertexBuffer(window.vertexBuffers[0].buffer, 
                              offset: 0, index: 0)
renderEncoder.setFragmentTexture(windowTexture, index: 0)
draw(renderEncoder: renderEncoder, model: window)

This code draws the window and should be familiar to you by now.

In Shaders.metal, add the new fragment function:

fragment float4 
            fragment_window(VertexOut vertex_in [[stage_in]],
                     texture2d<float> texture [[texture(0)]]) {
  constexpr sampler s(filter::linear);
  float4 color = texture.sample(s, vertex_in.uv);
  return color;
}

This function is similar to the earlier one: you create a default sampler, sample the texture to get the pixel color, and finally return the new color.

Run the playground again.

That’s a dull, opaque window!

You’ll fix that with blending. There are two ways to work with blending: the programmable way and the fixed-function way. In this chapter, you’ll learn about fixed-function blending.

In Renderer.swift, at the end of buildPipelineState(), right before:

windowPipelineState = 
    try device.makeRenderPipelineState(descriptor: descriptor)

Add this code:

// 1
guard let attachment = descriptor.colorAttachments[0] else { return }
// 2
attachment.isBlendingEnabled = true
// 3
attachment.rgbBlendOperation = .add
// 4
attachment.sourceRGBBlendFactor = .sourceAlpha
// 5
attachment.destinationRGBBlendFactor = .oneMinusSourceAlpha

With this code, you:

1. Grab the first color attachment from the render pipeline descriptor. A color attachment is a color render target that specifies the color configuration and color operations associated with a render pipeline. The render target holds the drawable texture where the rendering output goes.
2. Enable blending on the attachment.
3. Specify the blending type of operation used for color. Blend operations determine how a source fragment is combined with a destination value in a color attachment to determine the pixel value to be written.
4. Specify the blend factor used by the source color. A blend factor is how much the color will contribute to the final blended color. If not specified, this value is always 1 (.one) by default.
5. Specify the blend factor used by the destination color. If not specified, this value is always 0 (.zero) by default.

Note: There are quite a few blend factors available to use besides sourceAlpha and oneMinusSourceAlpha. For a complete list of options, consult Apple’s official page for Blend Factors: https://developer.apple.com/documentation/metal/mtlblendfactor.

You’re almost ready to see blending in action. However, as before, you’ll set up a toggle key for blending.

In Renderer.swift, add this property to the top of Renderer:

var blendingEnabled = false

In draw(in:), add this line above the draw call for rendering the window:

renderEncoder.setFragmentBytes(&blendingEnabled, 
           length: MemoryLayout<Bool>.size, index: 0)

In MetalView.swift, add a new case to the KeyCode enum in keyDown(with:) for the B key:

case b = 0xB

Then, check for this case in the switch statement, and toggle the status of blendingEnabled, just like you did for transparency:

case .b:
  renderer.blendingEnabled = !renderer.blendingEnabled

In Shaders.metal change fragment_window’s definition to:

fragment float4 
            fragment_window(VertexOut vertex_in [[stage_in]],
                constant bool &blendingEnabled [[buffer(0)]],
                     texture2d<float> texture [[texture(0)]])

Now, add this code to fragment_window, right before return color;:

if (blendingEnabled) {
  color.a = 0.5;
}

If blending is enabled, you set the alpha component of the current pixel to 0.5 (semi-transparent). Run the playground again. Click in the rendered image to make sure key strokes are captured there. Then, press the B key a few times to watch the blending toggle on and off:

How about some more fun by adding a second window?

You’ll reuse the window’s mesh and texture, so in Renderer.swift, add this code right before init(metalView:):

var window2Transform = Transform()

At the end of init(metalView:), set the size, position and rotation for the second window by adding this code:

window2Transform.scale = [3, 2, 2]
window2Transform.position = [0, 3, 5]
window2Transform.rotation = [-Float.pi / 2, 0, 0]

In draw(in:), while keeping the same pipeline state, add this code right before renderEncoder.endEncoding() to send information about the second window to the GPU:

modelViewProjectionMatrix = camera.projectionMatrix *
  camera.viewMatrix * window2Transform.matrix
renderEncoder.setVertexBytes(&modelViewProjectionMatrix,
                       length: MemoryLayout<float4x4>.stride, 
                       index: 1)
draw(renderEncoder: renderEncoder, model: window)

This block of code is almost identical to the one you used for the first window, except for the transform matrix, which is different for the second window.

Run the playground again and toggle blending on.

Whoa! What happened? It’s like looking through one single window instead of two windows. You know for sure the windows are not overlapped because one is positioned at Z = 4 and the other at Z = 5. So what happened?

Note: When blending is enabled, you should always be careful to render objects in a strict order, from back to front. You’re currently rendering the first window before rendering the second one, however, the first window is in front of the second one.

In this case, the tree pipeline state doesn’t have blending enabled, so it’s only important that you render the two windows in order. You have two options:

• Switch the two window rendering blocks inside draw(in:).
• Or move the second window in front of the first.

Option one is a simple cut and paste. Option two isn’t too difficult either, so, in Renderer.swift, in init(metalView:), change the second window’s transform as follows:

window2Transform.scale = [2, 2, 2]
window2Transform.position = [0, 2.75, 3]
window2Transform.rotation = [-Float.pi / 2, 0, 0]

Run the playground again and toggle blending on.

That’s more like it!

You may notice the overlapped portion of the second window is a bit darker than the rest of the window. This has a logical explanation.

Because you’re using an alpha value of 0.5 for each window, when blending is enabled, every window linearly interpolates the rendered result halfway closer to the solid green color in the windowColor texture. It’s the same idea as looking through multiple panes of green glass; put enough of them in front of each other, and all you’d see is pure green!

As previously mentioned, the other method of blending is programmable, where the developer does everything in the fragment shader. You can use the pipeline attachment’s [[color(0)]] attribute in the fragment shader to obtain the current color from the framebuffer and then combine it with the current fragment color.

Antialiasing

Often, rendered models show slightly jagged edges that are visible if you zoom in a few times. This is called aliasing and is caused by the rasterizer when generating the fragments. If you look at the edge of a triangle, or any straight line — especially one with a slope — you’ll notice the line doesn’t always go precisely through the center of a pixel; some pixels will be colored above the line and some below it.

The solution to fixing aliasing is called antialiasing, as you might have guessed, and it consists of techniques to render smoother edges. By default, the pipeline uses one sample point (subpixel) for each pixel that is close to the line to determine if they meet. It is, however, possible to use 4 or more points for increased accuracy of intersection determination. This is called Multisample Antialiasing (MSAA), and it is more expensive to compute.

Next, you’re going to configure the fixed-function MSAA on the pipeline and enable antialiasing on both the tree and the windows.

In Renderer.swift, add these properties to Renderer:

var antialiasingEnabled = false
var treePipelineStateAA: MTLRenderPipelineState!
var windowPipelineStateAA: MTLRenderPipelineState!

You’ll create two different pipeline states for the tree (same for the windows) so that you can toggle between aliased and antialiased.

In buildPipelineState(), right below this line:

treePipelineState = 
  try device.makeRenderPipelineState(descriptor: descriptor)

Add this code:

descriptor.sampleCount = 4
treePipelineStateAA = 
  try device.makeRenderPipelineState(descriptor: descriptor)
descriptor.sampleCount = 1

This second pipeline state is the same as treePipelineState with the number of samples changed. You also set the sample count back to 1 so the window pipeline state is not affected.

Now, it’s time to do the same for the windows.

In buildPipelineState(), right below this line:

windowPipelineState = 
  try device.makeRenderPipelineState(descriptor: descriptor)

Add this code:

descriptor.sampleCount = 4
windowPipelineStateAA = 
  try device.makeRenderPipelineState(descriptor: descriptor)
descriptor.sampleCount = 1

In draw(in:), change this code where you set the tree’s pipeline state:

renderEncoder.setRenderPipelineState(treePipelineState)

To:

view.sampleCount = antialiasingEnabled ? 4 : 1
var aaPipelineState = antialiasingEnabled ? 
                        treePipelineStateAA! : treePipelineState
renderEncoder.setRenderPipelineState(aaPipelineState!)

When you press the appropriate toggle key, the pipeline switches between the two different pipelines and also sets the view’s sample count to match.

Now do the same for the windows. Change the code where you set the window’s pipeline state from:

renderEncoder.setRenderPipelineState(windowPipelineState)

To:

aaPipelineState = antialiasingEnabled ? 
                    windowPipelineStateAA : windowPipelineState
renderEncoder.setRenderPipelineState(aaPipelineState!)

To add the antialiasing toggle key, in MetalView.swift, add another new case to the KeyCode enum in keyDown(with:); this time for the A key:

case a = 0

Add the check for it in the switch statement, toggling the status of antialiasingEnabled accordingly.

case .a:
  renderer.antialiasingEnabled = !renderer.antialiasingEnabled

Run the playground again and zoom right into the tree. Now tap the rendered image, so key strokes are captured in there. Press the A key a few times to watch the antialiasing being toggled on and off. If you have a retina screen, this effect may be quite hard to see. Look for diagonals on the leaves with transparency off.

Fog

If you still haven’t had enough fun in this chapter, why don’t you add some fog to the scene to make it even more interesting?

Fog is quite useful in rendering for a couple of reasons. First, it serves as a far delimiter for rendered content. The renderer can ignore objects that get lost in the fog since they’re not visible anymore. Second, fog helps with avoiding the popping-up effect for objects that just appeared in the scene from a distance, making their appearance into the scene more gradual.

As before, you’ll add a toggle key for fog.

In Renderer.swift, add this property at the top of the class:

var fogEnabled = false

You’ll need to add the fog to fragment shaders if you want all of the objects to be in the fog.

In draw(in:), before the first draw call, add this:

renderEncoder.setFragmentBytes(&fogEnabled, 
                               length: MemoryLayout<Bool>.size, 
							   index: 1)

This sends the fogEnabled boolean variable to the GPU using index: 1. Because no other resource is using this index, it’ll stay in the argument table at that index for all the subsequent draw calls.

In MetalView.swift, in keyDown(with:), add one last KeyCode case:

case f = 0x3

In the switch statement, check for a press of the F key to toggle the status of fogEnabled.

case .f:
  renderer.fogEnabled = !renderer.fogEnabled

Finally, in Shaders.metal, add the following argument to all three fragment shader definitions:

constant bool &fogEnabled [[buffer(1)]]

Then, create a new function in Shaders.metal, before any of the fragment functions:

float4 fog(float4 position, float4 color) {
  // 1
  float distance = position.z / position.w;
  // 2
  float density = 0.2;
  float fog = 1.0 - clamp(exp(-density * distance), 0.0, 1.0);
  // 3
  float4 fogColor = float4(1.0);
  color = mix(color, fogColor, fog);
  return color;
}

With this code, you:

1. Calculate the depth of the fragment position.

2. Define a distribution function that the fog will use next. It’s the inverse of the clamped (between 0 and 1) product between the fog density and the depth calculated in the previous step.

3. Mix the current color with the fog color (which you deliberately set to white) using the distribution function defined in the previous step.

Now, add the following code at the end of all three fragment shaders before return color;:

if (fogEnabled) {
  color = fog(vertex_in.position, color);
}

This code calls the fog function. Run the playground again, and toggle the fog with the F key:

As you can see, the entire scene is now in the fog. The closer you get to the tree, the less dense the fog. The same happens to the ground.

You see more of it as you go ahead. Go closer to the tree, and you’ll see the tree more clearly:

Note: Notice the cream sky is not affected by fog. This is because the cream color is coming from the MTKView instead of being rendered. Later, you’ll be creating a textured sky.

Challenge

I hope you had a blast playing with various fragment processing techniques. I know I did!

If you’re in for a few challenges:

1. Set the fog density to a lower, then higher value to see how that looks different.
2. Use different texture colors for the two windows to see how different colors blend.
3. Change the alpha value for one or both of the colors to see how this change influences the blended color.

Where to go from here?

In this chapter, you only looked into fixed-function blending and antialiasing. Per-fragment or per-sample programmable blending is possible by using the [[color]] attribute, which identifies the color attachment, as an argument to the fragment function.

Similarly, programmable antialiasing is possible via programmable sample positions which allow you to set custom sample positions for different render passes, unlike fixed-function antialiasing where the same sample positions apply to all render passes. For further reading, you can review the Programmable Sample Positions page at https://developer.apple.com/documentation/metal/graphics_rendering/render_targets/using_programmable_sample_positions.