vi. Introduction

Written by Caroline Begbie & Marius Horga

Welcome to Metal by Tutorials!

Metal is a unified, low-level, low-overhead application programming interface (API) for the graphics processing unit, or GPU. It’s unified because it applies to both 3D graphics and data-parallel computation paradigms. Metal is a low-level API because it provides programmers near-direct access to the GPU. Finally, Metal is a low-overhead API because it reduces the runtime cost by multi-threading and pre-compiling of resources.

But beyond the technical definition, Metal is the most appropriate way to use the GPU’s parallel processing power to visualize data or solve numerical challenges. It’s also tailored to be used for machine learning, image/video processing or, as this book describes, graphics rendering.

About This Book

This book introduces you to low-level graphics programming in Metal — Apple’s framework for programming on the graphics processing unit (GPU). As you progress through this book, you’ll learn many of the fundamentals that go into making a game engine and gradually put together your own engine. Once your game engine is complete, you’ll be able to put together 3D scenes and program your own simple 3D games. Because you’ll have built your 3D game engine from scratch, you’ll be able to customize every aspect of what you see on your screen.

How Did Metal Come to Life?

Historically, you had two choices to take advantage of the power of the GPU: OpenGL and the Windows-only DirectX. In 2013, the GPU vendor AMD announced the Mantle project in an effort to revamp GPU APIs and come up with an alternative to Direct3D (which is part of DirectX) and OpenGL. AMD were the first to create a true low-overhead API for low-level access to the GPU. Mantle promised to be able to generate up to 9 times more draw calls (the number of objects drawn to the screen) than similar APIs and also introduced asynchronous command queues so that graphics and compute workloads could be run in parallel. Unfortunately, the project was terminated before it could become a mainstream API.

Metal was announced at the Worldwide Developers Conference (WWDC) on June 2, 2014 and was initially made available only on A7 or newer GPUs. Apple created a new language to program the GPU directly via shader functions. This is the Metal Shading Language (MSL) based on the C++11 specification. A year later at WWDC 2015, Apple announced two Metal sub-frameworks: MetalKit and Metal Performance Shaders (MPS). In 2018, MPS made a spectacular debut as a Ray Tracing accelerator.

The API continues to evolve to work with the exciting features of the new Apple GPUS designed in-house by Apple. Metal 2 adds support for Virtual Reality (VR), Augmented Reality (AR) and accelerated machine learning (ML), among many new features, including image blocks, tile shading and threadgroup sharing. MSL was also updated to version 2.0 in Fall 2017 and is now based on the C++14 specification.

Why Would You Use Metal?

Metal is a top-notch graphics API. That means Metal can empower graphics pipelines and, more specifically, game engines such as the following:

• Unity and Unreal Engine: The two leading cross-platform game engines today are ideal for game programmers who target a range of console, desktop and mobile devices. However, these engines haven’t always kept pace with new features in Metal. For example, Unity announced that tessellation on iOS was to be released in 2018, despite it being demonstrated live at WWDC 2016. If you to use cutting-edge Metal developments, you can’t always depend on third-party engines.

• Divinity - Original Sin 2: Larian Studios worked closely with Apple to bring their amazing AAA game to iPad, taking advantage of Metal and the Apple GPU hardware. It truly is a stunning visual experience.

• The Witness: This award-winning puzzle game has a custom engine that runs on top of Metal. By taking advantage of Metal, the iPad version is every bit as stunning as the desktop version and is highly recommended for puzzle game fans.

• Many Others: From notable game titles such as Hitman, BioShock, Deus Ex, Mafia, Starcraft, World of Warcraft, Fortnite, Unreal Tournament, Batman and even the beloved Minecraft.

But Metal isn’t limited to the world of gaming. There are many apps that benefit from GPU acceleration for image and video processing:

• Procreate: An app for sketching, painting and illustrating. Since converting to Metal, it runs four times faster than it did before.
• Pixelmator: A Metal-based app that provides image distortion tools. In fact, they were able to implement a new painting engine and dynamic paint blending technology powered by Metal 2.
• Affinity Photo: Available on the iPad. According to the developer Serif, “Using Metal allows users to work easily on large, super high-resolution photographs, or complex compositions with potentially thousands of layers.”

Metal, and in particular, the MPS sub-framework, is incredibly useful in the realm of machine and deep learning on convolutional neural networks. Apple presented a practical machine learning application at WWDC 2016 that demonstrated the power of CNNs in high-precision image recognition.

When Should You Use Metal?

GPUs belong to a special class of computation that Flynn’s taxonomy terms Single Instruction Multiple Data (SIMD). Simply, GPUs are processors that are optimized for throughput (how much data can be processed in one unit of time), while CPUs are optimized for latency (how much time it takes a single unit of data to be processed). Most programs execute serially: they receive input, process it, provide output and then the cycle repeats.

Those cycles sometimes perform computationally-intensive tasks, such as large matrix multiplication, which would take CPUs a lot of time process serially, even in a multithreaded manner on a handful of cores.

In contrast, GPUs have hundreds or even thousands of cores which are smaller and have less memory than CPU cores, but perform fast parallel mathematical calculations.

Choose Metal when:

• You want to render 3D models as efficiently as possible.
• You want your game to have its own unique style, perhaps with custom lighting and shading.
• You will be performing intensive data processes, such as calculating and changing the color of each pixel on the screen every frame, as you would when processing images and video.
• You have large numerical problems, such as scientific simulations, that you can partition into independent sub-problems to be processed in parallel.
• You need to process multiple large datasets in parallel, such as when you train models for deep learning.

Who This Book Is For?

This book is for intermediate Swift developers interested in learning 3D graphics or gaining a deeper understanding of how game engines work. If you don’t know Swift, you can still follow along, as all the code instructions are included in the book. You’ll gain general graphics knowledge, but it would be less confusing if you cover Swift basics first. We recommend the Swift Apprentice book, available in our catlagoue:

https://www.raywenderlich.com/books/swift-apprentice

A smattering of C++ knowledge would be useful too. The Metal Shader Language that you’ll use when writing GPU shader functions is based on C++. But, again, all the code you’ll need is included in the book.

How to Read This Book?

If you’re a beginner to iOS/macOS development or Metal, you should read this book from cover to cover.

If you’re an advanced developer, or already have experience with Metal, you can skip from chapter to chapter or use this book as a reference.