22.
Integrating with SpriteKit & SceneKit
Written by Caroline Begbie & Marius Horga
Now that you have mastery over rendering, you can put your knowledge to use in other APIs.
SceneKit, SpriteKit and Core Image all have some integration with Metal shaders and Metal rendering.
There may be times when you don’t want to write a full-blown 3D Metal app, but you want to take advantage of SceneKit. Or perhaps all you want is a 2D layer in your Metal app — for that, you can use SpriteKit. And even though you have less control over your final output with SceneKit and SpriteKit, you can still incorporate shaders to give your games a unique look.
In this chapter, you’ll have a look at a collection of APIs that integrate with Metal. You’ll first create a toon outline and cel shader for the jet in Apple’s SceneKit template.
After that, you’ll open a scene similar to the game scene from Chapter 9, “Scene Graph.” and add to it a 2D overlay that feeds information to the player.
By creating this overlay, you’ll learn how to render a SpriteKit scene during each Metal frame rendering.
Finally, you’ll create a Core Image Metal kernel to do the blending of the SpriteKit overlay and the 3D rendered scene.
SceneKit starter project
Before creating the toon shader, you’ll first learn how to create your own custom shaders in SceneKit, and then find out how to pass data to the shaders.
Create a new project using the macOS Game template (or you can choose the iOS Game template if you prefer). Use the Product Name of Toon-SceneKit and make sure that SceneKit is specified in the Game Technology dropdown.
Note: Alternatively, you can choose to open the starter project for this chapter instead.
Build and run, and you’ll see Apple’s default game template jet animating on the screen. You can turn the jet by dragging.
This chapter assumes you have some familiarity with SceneKit, but if not, you can get the general idea of how SceneKit works by reading through GameViewController.swift, which has extensive comments.
Just like the game engine you’ve worked on throughout this book, each element is a node. You have geometry nodes such as the jet, light nodes and a camera node. These nodes are placed into a scene, and you attach the scene to a view. To animate the nodes, you run actions on them.
Open GameViewController.swift, and in viewDidLoad(), replace the rotation animation:
// animate the 3d object
ship.runAction(
SCNAction.repeatForever(SCNAction.rotateBy(x: 0, y: 2, z: 0,
duration: 1)))
With:
ship.eulerAngles = SCNVector3(1, 0.7, 0.9)
This gives you a better angle to see your toon shading.
Next, configure a different background color. Change:
scnView.backgroundColor = NSColor.black
To:
scene.background.contents = nil
scnView.backgroundColor = NSColor(calibratedWhite: 0.9,
alpha: 1.0)
This removes the procedural sky that ship.scn created, and replaces it with a light gray background.
SceneKit shaders
To run a Metal shader on a node, you create an SCNProgram object and attach it to the node.
Once you do this, you are fully responsible for transforming vertices, lighting and other shading and, even if you only want to change the fragment color, you have to create both a vertex shader and a fragment shader. However, to make it easier, SCNProgram will create uniform values, such as the model matrix or camera position, that you can access in your shaders.
Note: If you’re only performing a simple shader operation, such as changing the fragment color, look at
SCNShadablewhere you can provide short strings of shader code instead of taking control of transformations and lighting. The downside is that these compile at run-time, so your app will load slower, and you won’t have the benefit of seeing compiler errors if you make a syntax error.
In viewDidLoad(), just after configuring ship.eulerAngles, add this:
let program = SCNProgram()
program.vertexFunctionName = "shipVertex"
program.fragmentFunctionName = "shipFragment"
This creates a program and tells it what the shader names will be.
Shaders go with materials so that you can render each of your model’s materials differently. The jet only has one material, so add this to set the program on the jet’s first material:
if let material = ship.childNodes[0].geometry?.firstMaterial {
material.program = program
}
For your first SceneKit shader, you’ll create the most basic shader possible and render the jet in red. Create a new Metal file named Shaders.metal.
Add this to the bottom of your new file:
#include <SceneKit/scn_metal>
This lets you access the SceneKit buffers and uniforms.
Vertex attributes are available to the vertex shader using a struct. Add this:
struct VertexIn {
float4 position [[attribute(SCNVertexSemanticPosition)]];
};
This creates a struct to access vertex position, but you can also access normals, uvs, tangents and skeletal joint and weight information too.
Note: You can see a full list of all the vertex attributes, as well as all the other SceneKit data available to you, in the Apple documentation: https://developer.apple.com/documentation/scenekit/scnprogram.
Your vertex shader has to multiply the vertex by the current model-view-projection matrix. You can access this and other transform data by declaring a struct and binding it to [[buffer(1)]].
Add this new struct:
struct Uniforms {
float4x4 modelViewProjectionTransform;
};
Now add the struct containing the data that you’ll output from the vertex shader and input to the fragment shader:
struct VertexOut {
float4 position [[position]];
};
Add the vertex function:
vertex VertexOut shipVertex(VertexIn in [[stage_in]],
constant Uniforms& uniforms [[buffer(1)]]) {
VertexOut out;
out.position =
uniforms.modelViewProjectionTransform * in.position;
return out;
}
The hard-coded property name modelViewProjectionTransform is where the model-view-projection matrix is held. As you’ll see, there are various other properties available too.
Once you’ve worked out how to access the SceneKit data, all of the Metal Shading Language code should be familiar to you.
Add the fragment function:
fragment half4 shipFragment(VertexOut in [[stage_in]]) {
return half4(1, 0, 0, 1);
}
Build and run, and you get a plain gray window.
Match SceneKit names
When you’re using Metal with SceneKit, you’ll find that parameter names must match what SceneKit is expecting. Here, in the vertex function, SceneKit expects the Uniforms buffer to be named scn_node. When you give a parameter the name scn_node, you can name the struct Uniforms any name, and give the buffer index any index, and SceneKit will recognize the buffer as being the uniform values.
Replace uniforms with scn_node in two places, build and run, and you’ll see the jet rendered in red.
You just learned the basics of how to integrate SceneKit with Metal shaders, but there are a couple of tricks to send textures and constants to the shaders.
Send a texture to the fragment shader
In Shaders.metal, add these attributes to VertexIn:
float3 normal [[attribute(SCNVertexSemanticNormal)]];
float2 uv [[attribute(SCNVertexSemanticTexcoord0)]];
These properties will allow you to access normals and uv coordinates.
Add the uv to VertexOut:
float2 uv;
At the end of shipVertex, before return, to send the uv coordinates to the fragment shader, add:
out.uv = in.uv;
Add the texture as a parameter to the fragment function:
texture2d<float> baseColorTexture [[texture(0)]]
Replace the contents of the fragment function with:
constexpr sampler s(filter::linear);
float4 baseColor = baseColorTexture.sample(s, in.uv);
return half4(baseColor);
Here, you sample the texture based on the uv coordinates and return the sampled color. If you run the app now, you’ll see a white jet. You have to set up the texture in the jet’s materials.
In GameViewController.swift, in viewDidLoad(), locate:
material.program = program
Add this just afterward, but still inside the conditional block:
if let url =
Bundle.main.url(forResource: "art.scnassets/texture",
withExtension: "png") {
if let texture = NSImage(contentsOf: url) {
material.setValue(SCNMaterialProperty(contents: texture),
forKey: "baseColorTexture")
}
}
This locates the texture within the scene assets folder and loads it into the material. Note that baseColorTexture must be the exact name that you used as the texture parameter in the Metal fragment function.
Note: If your texture is outside of the scene assets folder, you can load it with
NSImage(named: "texture").
Build and run to see your textured jet:
Sending constant data to shaders
You’ve rendered a textured, unlit model, but if you want to do lighting, then you have to set it up yourself, with the usual light positions, model normals and lighting calculations.
Although SceneKit keeps uniform values for the transformation matrices, you’ll have to send lighting constants to the shaders by setting a value on the material.
In GameViewController.swift, in the conditional block where you set up the material, add this:
let lightPosition = lightNode.position
material.setValue(lightPosition, forKey: "lightPosition")
lightNode.position is of type SCNVector3. SceneKit will convert this to a Metal float3. If you need to send other data types, you can convert them to a Data type and pass that.
Previously in the book, you have done lighting in world space, but as long as you do calculations on light positions and normals in the same space, it doesn’t matter which space it is. With the matrices provided by SceneKit, it’s easier to do the lighting calculation in camera space. You’ll need both the light position in camera space, as well as the vertex position in camera space without the projection applied. You’ll do these calculations in the vertex function.
In Shaders.metal, add these fields to Uniforms:
float4x4 normalTransform;
float4x4 modelViewTransform;
SceneKit provides you with the inverse transpose of the model-view matrix to place the normals into camera space, and the model / view matrix to place the position into camera space.
Add these parameters to shipVertex:
constant SCNSceneBuffer& scn_frame [[buffer(0)]],
constant float3& lightPosition [[buffer(2)]]
Whereas Uniforms contains per-node data, SCNSceneBuffer contains frame data, and you’ll be able to access the camera transformation matrix in scn_frame.viewTransform.
Add these three variables to VertexOut:
float3 normal;
float4 viewLightPosition;
float4 viewPosition;
These will hold the normal, the light position and the vertex position in camera space to pass along to the fragment function.
Add this to shipVertex before the return:
out.normal =
(scn_node.normalTransform * float4(in.normal, 0)).xyz;
out.viewLightPosition =
scn_frame.viewTransform * float4(lightPosition, 1);
out.viewPosition = scn_node.modelViewTransform * in.position;
In shipFragment, add this before the return line:
float3 lightDirection =
(normalize(in.viewLightPosition - in.viewPosition)).xyz;
float diffuseIntensity =
saturate(dot(normalize(in.normal), lightDirection));
baseColor *= diffuseIntensity;
Build and run, and your jet is now shaded.
Obviously, you didn’t go through all of this work to only reproduce what the SceneKit renderer does, but now you can send constants, textures, and do anything in the shader that you care to, like what you’re about to see.
Toon shading
A full toon shader consists of an edge detection algorithm and a cel shader where you reduce the color palette. Generally, shading is a gradient from light to dark, but the shading in this rocket demonstrates cel shading where abrupt steps in the gradient occur:
Edge detection is difficult. One way of creating a solid outline is to render the model twice, one larger than the other. Render the larger one in black (or the outline color) behind the smaller model. That will give a solid outline around the model, but it won’t take into account curves within the outline.
In Chapter 11, “Tessellation and Terrains”, you used the Sobel filter on a height map to find out the difference in slope. If you render the entire screen to a texture, you could run a Sobel filter which convolutes the image and locates edges, but during a fragment shader, so far, you’ve only been able to access the current rendered fragment, and you have no idea whether that fragment is an edge.
The fwidth function
In fragment shaders, you have access to the current fragment, but you also have access to the change in slope of the fragment from neighboring fragments. Fragments pass through the rasterizer in a 2 x 2 arrangement, and the partial derivatives of each fragment can be derived from the other fragment in the group of four.
dfdx() and dfdy() return the horizontal and vertical changes in slope, and fwidth() gives you the absolute derivative of the combined dfdx() and dfdy().
To quickly see how this can be of use in edge detection, replace the entire contents of shipFragment with this:
float3 v = normalize(float3(0, 0, 10)); // camera position
float3 n = normalize(in.normal);
return fwidth(dot(v, n));
Here, you take the dot product of a constant camera position and the fragment’s normal and run it through fwidth().
Build and run to see the effect.
You’re seeing the slope of the dot product. White is a steep slope, where the fragment normal is at almost 90º to the camera, and black is front on to the camera with no slope at all. Reverse the color, and widen the line.
Replace the return with:
float edge = step(fwidth(dot(v, n)) * 10.0, 0.4);
return edge;
Here, you take the slope of the dot product, multiply it by 10 to increase it, and, if the slope is greater than a threshold of 0.4, return a black line. Otherwise, return white.
The step function returns 0.0 when the first argument value is greater than the second and 1.0 if the first argument value is less than the second.
Build and run to see the effect:
This is not a smooth edge line, but it gives an artistic touch to a toon.
Cel shading
When you render non-photorealistic toons, you generally use a minimal color range. Instead of having a smooth gradient for shading, you use a stepped flat color gradient.
In shipFragment, replace the return statement with the following:
if (edge < 1.0) {
return edge;
}
float3 l =
(normalize(in.viewLightPosition - in.viewPosition)).xyz;
float diffuseIntensity = saturate(dot(n, l));
float i = diffuseIntensity * 10.0;
i = floor(i) - fmod(floor(i), 2);
i *= 0.1;
half4 color = half4(0, 1, 1, 1);
return color * i;
If you’re not on an edge, calculate the lighting. Generally, you’d use diffuseIntensity for the shading, but by flooring the value and subtracting the remainder, you can manipulate the gradient value to be stepped.
Build and run, and you’ll see the gradient shading on the sides of the jet.
You can spread out the color variation by replacing the return with:
return color * pow(i, 4) * 4;
The last thing to add is a specular highlight. Before return, add this:
float specular = pow(max(0.0, dot(reflect(-l, n), v)), 5.0);
if (specular > 0.5) {
return 1.0;
}
Build and run, and you’ll see your final cel shaded jet.
Toon shading is just one of many non-photorealistic techniques. The field is called NPAR: Non-Photorealistic Animation and Rendering. You can find further reading on drawing toon lines and other non-photorealism in references.markdown accompanying this chapter.
Note: You can also use shaders with SpriteKit nodes. However, you write these in GLSL, as SpriteKit doesn’t support shaders written in the Metal Shading Language.
SpriteKit rendering in Metal
Note: As of the time of writing,
CIContext.renderhangs the app when using Xcode 11 or macOS Catalina 10.15. However, you can still do the rest of this chapter using Xcode 10 on macOS Mojave 10.14.
Apple’s WWDC 2017 video, “Going beyond 2D with SpriteKit” shows an example of a 3D rendered video game machine showing a playable 2D SpriteKit scene on the machine’s screen.
Another example of why you might want to use a SpriteKit scene rendered in Metal is a HUD — that’s a head-up display. A HUD is an overlay that shows scores or other 2D information you might want to show the player. This term originated from war-time military aviation, where the flight data was projected onto the airplane’s windshield. This meant that the pilot didn’t have to look away from the action to read the instruments.
SpriteKit is an excellent solution for a HUD because you can put it together quickly with very little code, and render the SpriteKit scene to a texture that you overlay on your 3D Metal scene.
Open the starter project CarGame. This a similar project to the one at the end of Chapter 9, “Scene Graph”. Relevant changes are:
-
Rendererholds the current command buffer as a class property. -
Scenemaintains the complete node tree in a flattened array calledallNodes. -
RenderPassallows you to hold separate render passes.
The main scene is GameScene.swift, which is where you set up the trees and oilcans and perform the game logic, such as collision testing. Build and run to remind yourself what the scene does. Use the keyboard controls W and S for forward and back and QR for rotate (or left and right arrow).
The aim is to collect all of the oilcans while avoiding trees.
You’ll create a simple HUD that counts the number of oilcans you collect while driving the car.
Create the HUD in SpriteKit
Create a new SpriteKit scene using the SpriteKit Scene template, and name it Hud.sks.
Choose View ▸ Show Library (or press Cmd Shift L to show the library) and drag two Labels onto the Scene. These are SKLabelNodes, and you’ll position them in code, so it doesn’t matter where they are in the scene. Open the Attributes inspector, and enter these values for the two labels:
- Name: label
- Label Text: Oil cans remaining:
- Name: count
- Label Text: 100
Choose any font and color you’d like for the two labels. The sample project uses Avenir Heavy 32.0 and the color yellow.
Click on the background of the scene to select the scene node. In the Custom Class inspector, name the custom class Hud. This is an important step that connects the .sks file with the Hud class you’re about to create.
Create a new Swift file named Hud. This will hold the class that controls the SpriteKit scene.
Add the code to initialize the new class:
import SpriteKit
class Hud: SKScene {
private var labelCount: SKLabelNode?
private var label: SKLabelNode?
override func sceneDidLoad() {
label = childNode(withName: "//label") as? SKLabelNode
labelCount = childNode(withName: "//count") as? SKLabelNode
}
}
This connects the two label SKNodes with their class properties.
Override the SpriteKit update method:
override func update(_ currentTime: TimeInterval) {
print("updating HUD")
}
When you update the scene, you’ll print out a message to the debug console. Later, you’ll use this method to update the HUD.
Override size to position the two nodes when the scene size changes:
override var size: CGSize {
didSet {
guard let label = label,
let labelCount = labelCount else { return }
label.horizontalAlignmentMode = .left
labelCount.horizontalAlignmentMode = .left
let topLeft =
CGPoint(x: -size.width * 0.5, y: size.height * 0.5)
let margin: CGFloat = 10
label.position.x = topLeft.x + margin
label.position.y = topLeft.y - label.frame.height - margin
labelCount.position = label.position
labelCount.position.x += label.frame.width + margin
}
}
This positions the two label nodes at the top left.
You now have a SpriteKit scene and a controlling class. You’ll create a subclass of Node so that you can add it to your game scene.
Create a new Swift file named HudNode.swift. Add this initialization code to the file:
import SpriteKit
class HudNode: Node {
let skScene: Hud
init(name: String, size: CGSize) {
guard let skScene = SKScene(fileNamed: name) as? Hud
else {
fatalError("No scene found")
}
self.skScene = skScene
super.init()
sceneSizeWillChange(to: size)
self.name = name
}
func sceneSizeWillChange(to size: CGSize) {
skScene.isPaused = false
skScene.size = size
}
}
This node subclass holds a reference to the SpriteKit scene and unpauses the scene when it loads. You separate out any initialization so that you can call this method when the user resizes the scene or rotates a device. On a change of size, the SpriteKit scene lays itself out.
In GameScene.swift, add the new HUD node. Add this new property:
var hud: HudNode!
In setupScene(), add the node to the scene:
hud = HudNode(name: "Hud", size: sceneSize)
add(node: hud, render: false)
Your HUD will be a non-rendering node. To be able to render the SpriteKit scene, you’ll render it to a separate texture. You then have the choice of rendering the texture on to a quad, or, as you’ll do in this chapter, blend the SpriteKit scene texture with the view’s current drawable texture.
In sceneSizeWillChange(to:), add the following line:
hud.sceneSizeWillChange(to: size)
The HUD now updates the label positions when the screen size changes.
SKRenderer
Generally, when you create a SpriteKit app, you hook up the SKScene with an SKView. The SKView takes care of all the rendering and places the scene onto the view. SKRenderer takes the place of SKView, allowing you to control updating and rendering of your SpriteKit scene.
Instead of allowing the view’s current drawable to render straight to the scene, you’ll intercept the render to blend the SpriteKit scene texture with the drawable in a post-processing stage.
To refresh how this will work in your engine, at each frame, Renderer currently calls update on the current scene. The current scene then updates all nodes. Renderer then renders all the nodes that conform to Renderable.
You’ll insert a post-processing stage into Renderer after the render encoder completes, and post process all the nodes that conform to a new PostProcess protocol.
In HudNode.swift, add two new properties to HudNode:
let skRenderer: SKRenderer
let renderPass: RenderPass
The first is the SpriteKit renderer, and the second is a render pass that will contain the rendered SpriteKit scene texture. RenderPass is the same class as you used when rendering the various water passes in Chapter 20, “Advanced Lighting”.
In init(name:size:), initialize these before calling super.init():
skRenderer = SKRenderer(device: Renderer.device)
skRenderer.scene = skScene
renderPass = RenderPass(name: name, size: size)
renderPass contains the HUD texture, so to update the size of it every time the scene size changes, add this to sceneSizeWillChange(to:):
renderPass.updateTextures(size: size)
To render the SpriteKit scene, override update(deltatime:):
override func update(deltaTime: Float) {
skRenderer.update(atTime: CACurrentMediaTime())
guard let commandBuffer = Renderer.commandBuffer else {
return
}
let viewPort = CGRect(origin: .zero, size: skScene.size)
skRenderer.render(withViewport: viewPort,
commandBuffer: commandBuffer,
renderPassDescriptor: renderPass.descriptor)
}
During each frame, Renderer asks the scene to update its nodes before doing the rendering. By overriding update(deltaTime:), you can update and render the SpriteKit HUD and hold the resulting texture until the post-processing stage. First, you tell skRenderer to update the SpriteKit scene. This will call Hud’s update(_:) which currently prints out “updating HUD” to the debug console.
You then render the scene using Renderer’s current command buffer. skRenderer will write the SpriteKit scene to the texture held in renderPass.
Build and run the app. Aside from the debug console message which assures you that the SpriteKit scene is updating, you won’t see any difference. Click the Capture GPU Frame icon to see what’s happening behind the scenes.
The frame’s command buffer holds the SKRenderer Pass. The purple exclamation point warns you that this pass has an unused texture - i.e., you’re rendering the texture but not using it yet.
Select CommandBuffer to see what textures have rendered during the frame:
If you look closely, the black texture on the left shows your SpriteKit label nodes. All you have to do now is combine these textures in a post-processing stage.
Post-processing
Create a new Swift file named PostProcess.swift, and replace the code with:
import MetalKit
protocol PostProcess {
func postProcess(inputTexture: MTLTexture)
}
Each node that conforms to PostProcess will take the view’s drawable texture as a parameter.
In Scene.swift, add this computed property:
var postProcessNodes: [PostProcess] {
return allNodes.compactMap { $0 as? PostProcess }
}
This will extract the PostProcess nodes from the flattened node hierarchy.
In Renderer.swift, toward the end of draw(in:), before commandBuffer.present(drawable) process each node:
scene.postProcessNodes.forEach { node in
node.postProcess(inputTexture: drawable.texture)
}
In HudNode.swift, conform HudNode to PostProcess by adding this after the class definition:
extension HudNode: PostProcess {
func postProcess(inputTexture: MTLTexture) {
print("post processing")
}
}
Build and run to ensure that you see both “post-processing” and “updating HUD” in the debug console.
Core Image
So far in this chapter, you’ve used Metal with SpriteKit and SceneKit. There’s one other framework whose shaders you can replace with your own custom Metal shaders: Core Image.
It would be more efficient to render a quad onto the screen and do the blending during the quad’s fragment shader. However, Core Image, with its huge number of filters, gives you great flexibility in how your final scene looks. These Core Image filters use Metal Performance Shaders under the hood, so they are blazingly fast.
Note: The Core Image Filter reference lists all the available filters: https://developer.apple.com/library/mac/documentation/GraphicsImaging/Reference/CoreImageFilterReference/index.html. Each of the filters shows the parameters you can use with the filter.
First, you’ll see how easy it is to render your scene as a cartoon in real time.
In HudNode.swift, in postProcess(inputTexture:), replace print with:
guard let commandBuffer = Renderer.commandBuffer else { return }
let drawableImage = CIImage(mtlTexture: inputTexture)!
Add the Core Image filter you want to use.
let filter = CIFilter(name: "CIComicEffect")!
filter.setValue(drawableImage, forKey: kCIInputImageKey)
let outputImage = filter.outputImage!
CIComicEffect will detect edges and render a half-tone effect. There’s only one input parameter for CIComicEffect: the input image. Set the input image on the filter and extract the output image.
outputImage is of type CIImage, so you need to render it to a new MTLTexture before you can use it in your render.
So that you don’t have the overhead of creating a new MTLTexture every frame, create a new property in HudNode for this new texture:
var outputTexture: MTLTexture
In init(name:size:), before calling super.init(), use RenderPass’s type method to create the new texture:
outputTexture = RenderPass.buildTexture(size: size,
label: "output texture",
pixelFormat: renderPass.texture.pixelFormat,
usage: [.shaderWrite])
At the top of postProcess(inputTexture:), add this code:
if inputTexture.width != outputTexture.width ||
inputTexture.height != outputTexture.height {
let size = CGSize(width: inputTexture.width,
height: inputTexture.height)
outputTexture = RenderPass.buildTexture(size: size,
label: "output texture",
pixelFormat: renderPass.texture.pixelFormat,
usage: [.shaderWrite])
}
This will ensure that the output texture has the same size as the input (drawable) texture.
At the end of postProcess(inputTexture:), add this code:
let context = CIContext(mtlDevice: Renderer.device)
let colorSpace = CGColorSpaceCreateDeviceRGB()
context.render(outputImage, to: outputTexture,
commandBuffer: commandBuffer,
bounds: outputImage.extent,
colorSpace: colorSpace)
This will create a new CIContext and render outputImage to your new outputTexture. If you specify nil for the commandBuffer, the context will create its own command buffer. In this situation, you’re in the middle of a frame render, so ensure that you use the current command buffer in Renderer.
To see outputTexture on the screen, blit the texture to the view’s current drawable. Add this to the end of postProcess(inputTexture:):
let blitEncoder = commandBuffer.makeBlitCommandEncoder()!
let origin = MTLOrigin(x: 0, y: 0, z: 0)
let size = MTLSize(width: inputTexture.width,
height: inputTexture.height,
depth: 1)
blitEncoder.copy(from: outputTexture, sourceSlice: 0,
sourceLevel: 0,
sourceOrigin: origin, sourceSize: size,
to: inputTexture, destinationSlice: 0,
destinationLevel: 0, destinationOrigin: origin)
blitEncoder.endEncoding()
You’ve created a few blit encoders along the way, and there’s nothing new here. You blit outputTexture to inputTexture. Build and run, and you’ll get a runtime error:
frameBufferOnly texture not supported for compute
This means that you haven’t set the view’s current drawable texture to be writable. For efficiency, Metal sets up the drawable as read-only by default. However, if you’re really sure you want to write to the drawable, even though it will slow things down, then you can.
In Renderer.swift, in init(metalView:), add this:
metalView.framebufferOnly = false
Now build and run again to see your real-time comic book effect rendering.
You’ve now seen how easy it is to apply whole screen effects to your render using Core Image.
Returning to the aim of this section, which is to layer the SpriteKit HUD on top of the rendered scene, you could use a Core Image composite operation using this code:
let hudImage = CIImage(mtlTexture: renderPass.texture)!
let drawableImage = CIImage(mtlTexture: inputTexture)!
let filter = CIFilter(name: "CISourceOverCompositing")!
filter.setValue(drawableImage, forKey: kCIInputBackgroundImageKey)
filter.setValue(hudImage, forKey: kCIInputImageKey)
However, all of these operations take place in sRGB space, and your render expects linear space. If you try out the above code, the final image will appear washed out and pale.
This is an excellent excuse to try out a Core Image kernel written in Metal.
You’ll take in the two images into the kernel and return the correct pixel from each, at the same time converting the sRGB value to linear.
Core Image Metal kernels
There are a few pre-defined kernel types you can use with Core Image:
-
CIKernel: a general custom filter. -
CIColorKernel: process color information. -
CIWarpKernel: process geometry (pixel position) information. -
CIBlendKernel: blend two images.
Each of these expects a different data type in the function arguments. It’s the last of these that you’ll use: CIBlendKernel. This takes two inputs of type sample_t for each of the images.
Create a Metal library for Core Image
Create a new file using the Metal template named CIBlend.metal. Include the Core Image headers:
#include <CoreImage/CoreImage.h>
To access the Core Image types, wrap your functions inside the coreimage namespace:
extern "C" { namespace coreimage {
}}
Pre-multiplied alpha blending
In this example, the HUD texture has anti-aliasing, where the texture is partially transparent around the edge of the letters:
Checking the HUD texture in the GPU debugger, you can see that the anti-aliased texture has been pre-multiplied with the alpha value:
In this image, where the solid yellow value in BGRA format is (2, 254, 251, 255), when the anti-aliasing alpha becomes 174, then the multiplied BGR is (1, 173, 171).
When you blend with a pre-multiplied color, the blend formula is:
destination.rgb =
source.rgb + (destination.rgb * (1 - source.a))
Inside the coreimage namespace, add the new blending function:
float4 hudBlend(sample_t hudTexture, sample_t drawableTexture) {
float4 color =
(1 - hudTexture.a) * drawableTexture + hudTexture;
color = float4(srgb_to_linear(color.rgb), 1);
return color;
}
This function takes in the two images and performs pre-multiplied alpha blending. You also convert this pixel from sRGB to linear and return it.
Note: Core Image has its own Kernel Language reference at https://developer.apple.com/metal/CoreImageKernelLanguageReference11.pdf.
srgb_to_linear()is an example of one of the color functions included in the Core Image Kernel Language.
Compiling and linking a Core Image kernel
You can build the kernel to check that it has no syntactical errors, however, because it’s wrapped up in the Core Image namespace, it won’t be available at runtime from your default Metal library.
As a rule, when you compile and link your Metal shaders, internally, the Xcode compiler builds to a .air file. The linker then links this .air file to a .metallib file and includes it with the app resources. The default library with all the Metal shader functions is called default.metallib.
Note: To view your default library in Finder, build your project, and under the Products group Ctrl-click CarGame-macOS.app. Choose Show In Finder. Ctrl-click the app file in Finder and choose Show Package Contents. Under Contents ▸ Resources, you’ll find the Metal library with compiled shaders default.metallib.
Core Image kernels require extra flags when compiling and linking. If you create a project that doesn’t already use Metal, you can set these flags in the Target Build Settings, however, in this project, those Build Settings would conflict with your existing Metal shaders.
The answer is to compile and link the kernel through the command line in Terminal.
Open a terminal window and navigate to the directory that contains CIBlend.metal.
Run these two commands to compile and link the Core Image kernel file:
xcrun -sdk macosx metal -c -fcikernel CIBlend.metal -o CIBlend.air
xcrun -sdk macosx metallib -cikernel CIBlend.air -o CIBlend.metallib
Alternatively, for iOS:
xcrun -sdk iphoneos metal -c -fcikernel CIBlend.metal -o CIBlend.air
xcrun -sdk iphoneos metallib -cikernel CIBlend.air -o CIBlend.metallib
Note: You may have to set the path correctly for the Command Line Tools to work. Go to Xcode ▸ Preferences, select the Locations tab and make sure the Command Line Tools dropdown is set appropriately.
Two new files will appear in the directory: CIBlend.air and CIBlend.metallib.
Add CIBlend.metallib to your project, checking only the target, either macOS or iOS, that you compiled for. Whenever you change the Core Image kernel, you’ll have to recompile and relink this .metallib file.
Make sure that you move the compiled .metallib file to the correct CarGame-macOS or CarGame-iOS group. You can delete the .air file.
You can now use the compiled kernel in your app.
In HudNode.swift, create a new property for the kernel:
let kernel: CIBlendKernel
In init(name:size:), before calling super.init(), initialize kernel:
// 1
let url = Bundle.main.url(forResource: "CIBlend",
withExtension: "metallib")!
do {
// 2
let data = try Data(contentsOf: url)
// 3
kernel = try CIBlendKernel(functionName: "hudBlend",
fromMetalLibraryData: data)
} catch {
fatalError("Kernel not found")
}
With this code, you:
- Get the URL for the .metallib resource.
- Load the library data.
- Get the kernel from the library.
In postProcess(inputTexture:), remove the lines that initialize filter, and replace:
let outputImage = filter.outputImage!
With:
let hudImage = CIImage(mtlTexture: renderPass.texture)!
let extent = hudImage.extent
let arguments = [hudImage, drawableImage]
let outputImage = kernel.apply(extent: extent,
arguments: arguments)!
Here’s the breakdown:
-
extentis aCGRectthat provides the size of the image. -
argumentsmatch the two arguments that you set up inhudBlend:
float4 hudBlend(sample_t hudTexture, sample_t drawableTexture)
Build and run, and you’ll see your HUD overlaying your rendered scene:
Challenge
Currently, you’re printing out “updating HUD” in the debug console. This print comes from update(_:) in the SpriteKit scene class Hud. Your challenge is to update the HUD, keeping track of the oil cans that the player has collected. To achieve this, in GameScene, on colliding with an oil can, you’ll tell HudNode to update Hud using oilcanCount. As always, you’ll find a solution in the challenge directory for this project.
Where to go from here?
In this chapter, you created two simple examples that you can experiment with further. Fragment shaders, and to a lesser extent vertex shaders, are endlessly fascinating.
Examine contributions for each shader type at:
- Fragment shader: https://www.shadertoy.com
- Vertex shader: https://www.vertexshaderart.com
After looking things over, try to express the shader in the Metal Shading Language.
For Core Image filters, the go-to (free for now) book is Core Image for Swift by Simon Gladman available at https://itunes.apple.com/us/book/core-image-for-swift/id1073029980?mt=13. You will learn how to chain filters together and explore your creativity.
Finally, in references.markdown for this chapter there are some interesting links for Non-Photorealistic Rendering.