25. Managing Resources

Written by Marius Horga & Caroline Begbie

So far, you’ve created an engine where you can load complex models with textures and materials, animate or update them per frame and render them. Your scenes will start to get more and more complicated as you develop your game, and you’ll want to find more performant ways of doing things and organizing your game resources.

Instead of processing each submesh and laboriously moving each of the submesh’s textures to the GPU, you’ll take advantage of the centralization of your textures in the Texture Controller. By the end of the chapter, you’ll be able to move all your textures to the GPU at once with just one render encoder command.

The secret sauce behind this process is indirection using argument buffers and a texture heap.

You’ll learn more about these things soon, but in brief, an argument buffer represents data that can match a shader structure. You can send the argument buffer to a shader function with one command, instead of sending each of the structure components individually.

A heap is exactly what it sounds like. You gather up your resources, such as textures and buffers, into an area of memory called a heap. You can then send this heap to the GPU with one command.

The Starter Project

With the basic idea under your belt, you’re ready to get started.

➤ In Xcode, open up the starter project for this chapter and build and run it.

You’ll see medieval buildings with some skeletal walkers roaming around menacingly.

The project consolidates many of the features that you’ve learned so far:

• Shadows
• IBL Lighting with sky box
• Animation
• Alpha testing
• Textured models
• Models with materials but no textures

There are a few added nifty features:

1. Shadows are now soft shadows with PCF filtering.

2. In the Textures folder, in TextureController.swift, TextureController has an extra level of indirection. The old textures dictionary is now named textureIndex and it holds indices into an array of textures.

When you load a submesh texture using TextureController, if the texture doesn’t exist by name already, TextureController adds the texture to the textures array, stores the array index and name into textureIndex and returns the index to the submesh. If the texture already exists by name, then the submesh simply holds the existing array index to the texture.

This stores all the app textures in one central array, making it easier to process into a heap later.

3. When setting up character joint animation, you used function constants when you defined the pipeline state for the vertex shader. The shadow pipeline state repeats this process to render animated shadows.

4. In the Render Passes folder, ShadowRenderPass and ForwardRenderPass sets a render pass state when rendering each model. The model then sets the correct mesh pipeline state depending on this render pass state, whether it is shadow or main.

Argument Buffers

When rendering a submesh, you currently send up to six textures and a material individually to the GPU for the fragment shader: Base color, normal, roughness, metalness, ambient occlusion and opacity textures. During the frame render loop, each of the textures requires a renderEncoder.setFragmentTexture(texture:at:) command.

All the textures have a purple exclamation mark, indicating that there is an inefficiency problem with the texture. In this case, the bindings are actually redundant because the textures have previously been bound in this render pass.

Using argument buffers, you can set pointers, defined as MTLResourceIDs, to these six textures in one buffer, and set this buffer on the render command encoder with just one command. This argument buffer doesn’t only have to point to textures, it can point to any other data necessary to render the frame.

When you come to draw call time, instead of setting the textures on the render command encoder, you set the single argument buffer. You then perform renderEncoder.useResource(_:usage:) for each texture so that you can access all six textures on the GPU as reusable indirect resources.

Once you set up an argument buffer, you can refer to it in a shader, using one structure that matches the buffer data as a parameter to the shader function.

During the render loop, setting textures and buffers on the render command encoder incurs some internal verification. This verifying process will now take place during initialization, when the texture and buffer resource ID pointers are initially set into the argument buffer. Anything you can move outside of the render loop is a gain.

Creating the Shader Structure

➤ In the Shaders folder, open IBL.metal.

The fragment_IBL function has six parameters for material textures and one for the Material. You’re going to combine all of these into one structure, and use the structure as the parameter.

➤ Open Material.h. This file contains the Material structure and texture index numbers. In previous chapters, these were in Common.h. You’ll need to access Material in Swift, so the bridging header file Common.h imports Material.h.

When you set up the argument buffer in Swift, you’ll store MTLResourceIDs. The fragment shader will expect texture2d<float>s.

You’ll now create structures that will work for both Swift and Metal Shading Language.

➤ Still in Material.h, before #endif, create a new structure:

#if __METAL_VERSION__
// MARK: - Metal Shading Language

#include <metal_stdlib>
using namespace metal;

struct ShaderMaterial {
  array<texture2d<float>, MaterialTextureCount> textures;
  Material material;
};

#endif // Metal version

There are six textures, and MaterialTextureCount is defined in TextureIndices as OpacityTexture (5) + 1.

➤ For the Swift side, add this code before #endif // Metal version:

#else
// MARK: - Swift side
#include <Metal/Metal.h>

struct ShaderMaterial {
  MTLResourceID textures[MaterialTextureCount];
  Material material;
};

Here, you make the equivalent declaration for Swift. Metal.h holds the necessary definition of MTLResourceID.

Each argument buffer structure element has an implicit ID. For example, textures[BaseColor] has an implicit ID of 0 here. If you want to use an out-of-order ID index, instead of defining the array, you can list all textures with explicit IDs with an attribute, for example: MTLResourceID baseColorTexture [[id(BaseColor)]].

Soon, you’ll create an argument buffer that matches these IDs. You’ll pass in material as a constant value. If you were to create an MTLBuffer containing material, you can define it in ShaderMaterial as: constant Material &material;.

➤ Open IBL.metal, and change the signature for fragment_IBL to:

fragment float4 fragment_IBL(
  VertexOut in [[stage_in]],
  constant Params &params [[buffer(ParamsBuffer)]],
  constant Light *lights [[buffer(LightBuffer)]],
  constant ShaderMaterial &shaderMaterial [[buffer(MaterialBuffer)]],
  depth2d<float> shadowTexture [[texture(ShadowTexture)]],
  texturecube<float> skybox [[texture(SkyboxTexture)]],
  texturecube<float> skyboxDiffuse [[texture(SkyboxDiffuseTexture)]],
  texture2d<float> brdfLut [[texture(BRDFLutTexture)]])

Instead of receiving the textures directly, fragment_IBL will receive just one structure containing all the textures and material needed for the model.

➤ Replace:

Material material = _material;

➤ With:

Material material = shaderMaterial.material;
auto textures = shaderMaterial.textures;
texture2d<float> baseColorTexture = textures[BaseColor];
texture2d<float> normalTexture = textures[NormalTexture];
texture2d<float> metallicTexture = textures[MetallicTexture];
texture2d<float> roughnessTexture = textures[RoughnessTexture];
texture2d<float> aoTexture = textures[AOTexture];
texture2d<float> opacityTexture = textures[OpacityTexture];

You remove all the compile errors with these assignments.

For the moment you’ve finished setting up the GPU shader.

Creating the Argument Buffer

Now you’ll set up the argument buffer containing all the texture resource IDs in Swift.

➤ In the Geometry folder, open Submesh.swift, and add the new argument buffer property to Submesh:

var materialBuffer: MTLBuffer!

materialBuffer will contain pointers to the textures and the material.

➤ Create a new method in Submesh:

mutating func initializeMaterials() {
  // 1
  let materialBufferSize = MemoryLayout<ShaderMaterial>.stride
  materialBuffer = Renderer.device.makeBuffer(
    length: materialBufferSize)
  materialBuffer.label = "Material Buffer"
  // 2
  let textureIDs = allTextures.map { texture in
    texture?.gpuResourceID ?? MTLResourceID()
  }
  // 3
  let pointer = materialBuffer.contents()
    .assumingMemoryBound(to: ShaderMaterial.self)
  // 4
  pointer.pointee.material = material
  pointer.pointee.textures.0 = textureIDs[0]
  pointer.pointee.textures.1 = textureIDs[1]
  pointer.pointee.textures.2 = textureIDs[2]
  pointer.pointee.textures.3 = textureIDs[3]
  pointer.pointee.textures.4 = textureIDs[4]
  pointer.pointee.textures.5 = textureIDs[5]
}

Going through the code:

1. Create an MTLBuffer to hold the materials. Swift recognizes the ShaderMaterial structure that you set up in Material.h.
2. Create an array of MTLResourceIDs that match the submesh textures.
3. Create a pointer into the material buffer mapped to the ShaderMaterial structure.
4. Load up the data into the buffer. Unfortunately, arrays in Metal Shading Language are not dynamic. They have a fixed number of elements. Swift treats fixed arrays as tuples.

➤ Add the following code to the end of init(mdlSubmesh:mtkSubmesh:):

initializeMaterials()

In your app, there’s one other method where you can change textures.

➤ Open Model.swift and add this at the end of setTexture(name:type:):

meshes[0].submeshes[0].initializeMaterials()

You call setTexture(name:type:) when you set the texture on the ground plane primitive.

You’ve now set up your argument buffer. Instead of setting the textures and material for the fragment shader during the render loop, you’ll be able to set the single argument buffer.

Updating the Draw Call

➤ In the Renderer folder, open Rendering.swift. This holds the extension on Model, where you render each model.

➤ In render(encoder:uniforms:params:renderState:), in the for submesh in mesh.submeshes loop, locate setMaterials(encoder:submesh:).

setMaterials(encoder:submesh:) encodes all the textures and materials to the fragment function. As you’re now using one buffer for all these textures and materials, this method is no longer necessary.

➤ Replace:

setMaterials(encoder: encoder, submesh: submesh)

➤ With:

encoder.setFragmentBuffer(
  submesh.materialBuffer,
  offset: 0,
  index: MaterialBuffer.index)

Instead of encoding all of the textures, you simply send the single argument buffer to the GPU.

Don’t build and run now. But if you did, you might get a lot of GPU errors appearing in the debug console:

Even if the render appears correct, if you capture the GPU workload, you might still get errors. When you have GPU memory errors, weird things can happen on the display. Debugging these errors can be frustrating as your display may lock up because you have accessed memory that you’re not supposed to.

In this case, running on iPad Air 3, the textures are either missing or pink. On the buildings, the colors come from material.baseColor.

You’ve set up a level of indirection with the argument buffer pointing to the textures, but you still have to tell the GPU to load these textures. When dealing with indirection and buffer data, it’s often easy to omit this vital step, so if you have errors at any time, check in the GPU debugger that the resource is available in the indirect resource list, but also check that you are using the resource in the render command encoder command list.

➤ Still inside the conditional if renderState != .shadowPass, but after the previous code, add this:

submesh.allTextures.forEach { texture in
  if let texture {
    encoder.useResource(texture, usage: .read, stages: .fragment)
  }
}

Here, you tell the GPU that you’re going to read from these textures, and they should be resident on the GPU available to fragment shaders.

➤ Delete the method setMaterials(encoder:submesh:) since you no longer need it.

➤ Build and run the app, and the scene will render as before.

➤ Capture the GPU workload.

➤ In the Debug navigator, open Command Buffer and Forward Render Pass and select ground.

➤ Select Bound Resources using the navigator icon at the top left of the pane and examine the resources.

For the ground plane, the Indirect section lists the color texture grass. MTLRenderCommandEncoder.useResource(_:usage:) explicitly makes the texture accessible to the GPU as an indirect resource.

➤ Under Fragment, double-click Material Buffer to examine it.

Note: It’s important to label buffers so that you can recognize what they are in the GPU capture. Because you labelled Material Buffer when you created it, the buffer isn’t called something like Buffer 0x840ab180.

Here, you can examine the textures and material properties in shaderMaterial. The ground doesn’t have a normal texture.

The arrow next to grass indicates the indirection. If you click the arrow, you’ll see the ground plane’s color texture with mip maps.

You’ve now set up your app to use argument buffers for textures and the material instead of sending them individually. This may not feel like a win yet, and you’ve increased overhead by adding a new buffer. But you’ve reduced overhead on the render command encoder. Instead of having to validate the textures each frame, the textures are validated when they are first placed into the argument buffer, while you’re still initializing your app data. In addition to this, you’re grouping your materials together into the one structure, and only using one argument table entry in the fragment function. If you have many parameters that you can group together, this will save resources.

Resource Heaps

You’ve grouped textures into an argument buffer for each submesh, but you can also combine all your app’s textures into a resource heap.

A resource heap is simply an area of memory where you bundle resources. These can be textures or data buffers. To make your textures available on the GPU, instead of having to perform renderEncoder.useResource(_:usage:) for every single texture, you can perform renderEncoder.useHeap(_:) once per frame instead. That’s one step further in the quest for reducing render commands.

Note: The following code is not optimized. You will create the MTLTexture twice. Once on loading, and then you’ll copy it to the heap as a new texture. This is the easiest way to include the process in your app, but you can work on loading techniques where you pre-load static mesh and textures straight into heaps.

➤ In the Textures folder, open TextureController.swift.

TextureController stores all your app’s textures in one central array: textures. From this array, you’ll gather all the textures into a heap and move the whole heap at one time to the GPU.

➤ In TextureController, create a new property:

static var heap: MTLHeap?

➤ Create a new type method to build the heap:

static func buildHeap() -> MTLHeap? {
  let heapDescriptor = MTLHeapDescriptor()

  // add code here

  guard let heap =
    Renderer.device.makeHeap(descriptor: heapDescriptor)
    else { return nil }
  return heap
}

MTLDevice.makeHeap(descriptor:) is a time-consuming operation, so make sure that you execute it at loading time, rather than when your app is in full swing. Once you’ve created the heap, it’s fast to add Metal buffers and textures to it.

You build a heap from a heap descriptor. This descriptor will need to know the size of all the textures combined. Unfortunately MTLTexture doesn’t hold that information, but you can retrieve the size of a texture from a texture descriptor.

In the Utility folder, in Extensions.swift, there’s an extension on MTLTexture that will provide a descriptor from the texture.

➤ In TextureController.swift, in buildHeap(), replace // add code here with:

let descriptors = textures.map { texture in
  texture.descriptor
}

Here, you create an array of texture descriptors to match the array of textures. Now you can add up the size of all these descriptors.

➤ Following on from the previous code, add this:

let sizeAndAligns = descriptors.map { descriptor in
  Renderer.device.heapTextureSizeAndAlign(descriptor: descriptor)
}
heapDescriptor.size = sizeAndAligns.reduce(0) { total, sizeAndAlign in
  let size = sizeAndAlign.size
  let align = sizeAndAlign.align
  return total + size - (size & (align - 1)) + align
}
if heapDescriptor.size == 0 {
  return nil
}

You calculate the size of the heap using size and correct alignment within the heap. As long as align is a power of two, (size & (align - 1)) will give you the remainder when size is divided by alignment. For example, if you have a size of 129 bytes, and you want to align it to memory blocks of 128 bytes, this is the result of size - (size & (align - 1)) + align:

129 - (129 & (128 - 1)) + 128 = 256

This result shows that if you want to align blocks to 128, you’ll need a 256 byte block to fit 129 bytes.

You have an empty heap, but you need to populate it with textures. Each texture must match the heap’s CPU cache mode and also the heap’s storage mode.

➤ At the end of buildHeap(), but before return heap, add this:

let heapTextures = descriptors.map { descriptor -> MTLTexture in
  descriptor.storageMode = heapDescriptor.storageMode
  descriptor.cpuCacheMode = heapDescriptor.cpuCacheMode
  guard let texture = heap.makeTexture(descriptor: descriptor) else {
    fatalError("Failed to create heap textures")
  }
  return texture
}

You iterate through the descriptors array and create a texture for each descriptor. You store this new texture in heapTextures.

heapTextures now contains a bunch of empty texture resources. To copy the submesh texture information to the heap texture resources, you’ll need a blit command encoder.

The Blit Command Encoder

To blit means to copy from one part of memory to another and is typically an extremely fast operation. You create a blit command encoder using a command buffer, just as you did the render and compute command encoders. You then use this encoder when you want to copy a resource such as a texture or Metal buffer.

➤ Add this after the previous code, before return heap:

guard
  let commandBuffer = Renderer.commandQueue.makeCommandBuffer(),
  let blitEncoder = commandBuffer.makeBlitCommandEncoder()
else { return nil }
  zip(textures, heapTextures)
  .forEach { texture, heapTexture in
    heapTexture.label = texture.label
    // blit here
  }

You create the blit command encoder using a command buffer. You then set up a forEach loop that will process all the textures and match them with the heap textures.

➤ Replace // blit here with:

var region =
  MTLRegionMake2D(0, 0, texture.width, texture.height)
for level in 0..<texture.mipmapLevelCount {
  for slice in 0..<texture.arrayLength {
    blitEncoder.copy(
      from: texture,
      sourceSlice: slice,
      sourceLevel: level,
      sourceOrigin: region.origin,
      sourceSize: region.size,
      to: heapTexture,
      destinationSlice: slice,
      destinationLevel: level,
      destinationOrigin: region.origin)
  }
  region.size.width /= 2
  region.size.height /= 2
}

When copying textures, you specify a region. Initially the region will be the entire texture’s width and height. You’ll then blit mip levels where the region will get progressively smaller.

You copy each texture to a heap texture. Within each texture, you copy each level and slice. Levels contain the texture mipmaps, which is why you halve the region each loop. A slice is either the index into a texture array, or, for a cube texture, one of six cube faces.

Even though there are a lot of parameters to the blit encoder copy method, they are simply for deciding which area of the texture is to be copied. You can copy part of a texture by setting the origin and source size of the region. You can also copy part of a texture to a different region in the destination texture.

➤ Before return heap, add the following:

blitEncoder.endEncoding()
commandBuffer.commit()
Self.textures = heapTextures

This ends the encoding, commits the command buffer and replaces the original textures with the heap textures. TextureController.buildHeap() will now create a heap from all the textures gathered during scene loading.

➤ In the Geometry folder, open Submesh.swift.

In initializeMaterials(), you create an argument buffer from the submesh textures when you load the submesh. Unfortunately, as you’ve now copied all the old textures to new heap textures, your submesh argument buffers point to the wrong textures now.

➤ At the end of init(mdlSubmesh:mtkSubmesh:), remove:

initializeMaterials()

Open Model.swift, and in setTexture(name:type:), remove:

meshes[0].submeshes[0].initializeMaterials()

You’ll need to create the argument buffers after you’ve created the heap.

➤ In the Renderer folder, open Renderer.swift, and add a new method to Renderer:

func initialize(_ scene: GameScene) {
  TextureController.heap = TextureController.buildHeap()
  for model in scene.models {
    model.meshes = model.meshes.map { mesh in
      var mesh = mesh
      mesh.submeshes = mesh.submeshes.map { submesh in
        var submesh = submesh
        submesh.initializeMaterials()
        return submesh
      }
      return mesh
    }
  }
}

initialize(_:) will ensure that the heap gets built before the main render loop. You then process all the submeshes and initialize the materials with the correct textures.

➤ In the Game folder, open GameController.swift, and in init(metalView:options:) after scene = GameScene(), add this:

renderer.initialize(scene)

➤ Build and run the app to ensure that everything still works.

You’ve now placed all your textures in a heap, and are using those individual textures, but aren’t yet taking full advantage of the heap. Before rendering any models, you can send the textures to the GPU at the start of a render pass to be all ready and waiting for processing.

➤ In the Render Passes folder, open ForwardRenderPass.swift.

➤ In draw(commandBuffer:scene:uniforms:params:), add this after creating renderEncoder:

if let heap = TextureController.heap {
  renderEncoder.useHeap(heap, stages: .fragment)
}

➤ Open Rendering.swift, and in render(encoder:uniforms:params:renderState:), remove:

submesh.allTextures.forEach { texture in
  if let texture = texture {
    encoder.useResource(texture, usage: .read, stages: .fragment)
  }
}

Instead of having a useResource command for every texture, you perform one useHeap every render pass. This could be a huge saving on the number of commands in a render command encoder, and so a reduction of the number of commands that a GPU has to process each frame.

➤ Build and run the app, and your render will be exactly the same as it was last time you ran the app.

➤ Capture the GPU workload.

➤ Open Command Buffer > Forward Render Pass and select the useHeap command that you set at the start of the render pass.

In the bound resources, all the scene textures are listed under indirect resources, and are available for use in any shader during this render pass.

If you check out each of the models’ draw calls, you’ll also see that you reduced the number of encoded render commands along with most of the redundant bindings.

Residency Sets

So far, you’ve moved your texture binding process from each submesh to each draw call. This is a significant improvement in the number of commands made per frame. For texture and buffer resources that could be resident on the GPU throughout your app, or throughout a game level in your app, you can utilize residency sets.

Residency sets allow you to load up a group of resources at one time, for example, at the start of a level, and then unload them at the end of the level.

You’ve created a heap which will avoid memory fragmentation, but you could bypass this step and add the textures to a residency set instead. Since you have the heap created already, you’ll add the heap to the residency set.

➤ In the Renderer folder, open Renderer.swift, and add a new property to Renderer:

let residencySet: MTLResidencySet

➤ In init(metalView:options:), before super.init, initialize the residency set:

let setDescriptor = MTLResidencySetDescriptor()
setDescriptor.label = "Residency Set"
setDescriptor.initialCapacity = 1
residencySet = try! device.makeResidencySet(
  descriptor: setDescriptor)

You set the initial capacity of the set to one, because you know that you are only adding one resource, the heap, to the set. If you add resources directly to the set, you should pre-calculate how many there are. Apple documentation says you can leave this as zero, and Metal will give the residency set a “standard starting capacity”.

➤ In initialize(scene:), after building the heap, but before the for loop, add this code:

residencySet.addAllocation(TextureController.heap!)
residencySet.commit()

You add the heap to the residency set and indicate that you have finished adding and removing resources.

You now have the choice of attaching the residency set to a command buffer on each frame, or attaching it to the command queue once. Your meshes are static and used throughout your app, so you can attach the set to the command queue.

➤ After the previous code add this code:

Renderer.commandQueue.addResidencySet(residencySet)

Now, whenever you commit your command buffer during a frame, Metal will automatically attach the residency set to the command buffer, making the resources available to GPU shader functions.

➤ In the Render Passes folder, open ForwardRenderPass.swift, and, in draw(commandBuffer:scene:uniforms:params:), remove:

if let heap = TextureController.heap {
  renderEncoder.useHeap(heap, stages: .fragment)
}

As your heap is now in a residency set, you don’t need this extra command.

Build and run your app, and check that it still works.

The residency set is now available when your app commits the command buffer.

You’ve now separated out binding your textures from your rendering code, with a level of indirection via the argument buffer. But have you seen any performance improvement? In this example, on a recent device, probably not. But the more complicated your render passes get, the better the improvement, as there will be better memory management and fewer render commands.

With the residency set, you have more control over when you can load and unload your set of textures.

Key Points

• An argument buffer is a collection of pointers to resources that you can pass to shaders.
• A resource heap is a collection of textures or Metal buffers. A heap can be static, as in this chapter’s example, but you can also reuse space on the heap where you use different textures at different times.
• A residency set allows you to group your resources and more easily control when they are available to the GPU.