18. Creating 3D Models With Object Capture

Written by Chris Language

Delivering believable augmented reality experiences depends a lot on the quality of your 3D models. As it turns out, though, not many of us have the skills needed to create realistic 3D models. Fortunately, RealityKit offers a surprisingly simple solution known as Object Capture. Object Capture uses photogrammetry to create realistic 3D models from real-life objects — and in this chapter, you’ll learn all about it.

What is Photogrammetry?

Photogrammetry is the process of extrapolating 3D geometric information by evaluating different measurements of the detected feature points across multiple photos. Wow, that’s a lot to process! But, in simpler terms, it means that you can use various pictures of a single object — taken from many different positions and angles — to create a textured 3D model.

Preparing for Object Capture

Before you start snapping pics and converting real-life objects into their digital counter-parts, you’ll first need to build yourself a photo booth. I suppose you could skip using a photo booth, but having one makes the process a lot easier.

Building a Photo Booth

To build a photo booth on a budget, you need only a few things:

First, you’ll need a seamless background. If you have a 3D printer — or know someone who does — you can print yourself a Photographic Sweep Stand from Thingiverse - Photographic Sweep Stand.

For extra support, clamp the stands to a piece of cardboard.

Next, you’ll need a way to capture the object in 360° degrees; you’ll also need a way to stabilize the camera — in this case, your phone. One option is to get yourself an inexpensive lazy susan and a small tripod for your phone.

For best results, place the lazy susan on top of the Photographics Sweep Stand.

You’ll also need some kind of lighting — and of course, something to photograph.

Mr. Pig is ready for his close-up.

Now that you have everything you need, it’s time to put it together.

Let’s take a closer look at how you can set up your new DYI photo booth:

• Light Source. Place the light source in front of the object, slightly above the camera. If the light is too bright, you can use a white piece of paper to diffuse it. The goal is to eliminate all of the shadows cast by the object.

• Camera. Place the camera on the tripod in front of the object. Remember, a tripod helps stabilize the camera while you’re taking photos.

• Photographic Sweep Stand. Place your sweep stand on a table to create an infinite background behind your object.

• Lazy Susan. Place the lazy susan on top of the sweep stand, and place the object on top of that. For the best results, find the center of the lazy susan so that the object will remain in one location as you rotate the lazy susan.

• Cup of Coffee. Coffee is the most crucial part of your little DYI project, so always keep a cup of coffee at hand.

Perfect, you’re ready to capture your first object.

Taking Photos

Although RealityKit’s Photogrammetry framework makes it possible to convert real-world objects into 3D models, taking pictures of those objects requires patience — and can sometimes get tricky. Here are a few things you can do to help:

• Make sure you capture all angles of the object.
• Make sure that there’s at least a 70% overlap from one photo to the next.
• Use a clutter-free background, preferably dark.
• Your object requires some texture for the photogrammetry process to track its features.
• Light your object from the front to eliminate shadows as far as possible.
• Place your object on a white or clear, non-reflective surface.

The image capturing process isn’t always easy — and, despite all of your hard work, things can still go wrong — as was the case with this series of images:

Although this collection of images does capture Mr. Pig from multiple 45° angles, there wasn’t enough overlap from one photo to another, causing the photogrammetry process to fail. For better results, you need to have at least a 70% overlap from one image to the next.

Creating a Photogrammetry App

Let’s look at a practical example. An excellent place to start is with Apple’s very own photogrammetry example: Creating a Photogrammetry Command-Line App

Download the project and extract it. Then, in Xcode, open HelloPhotogrammetry.xcodeproj.

HelloPhotogrammetry is a command-line application. With command-line apps, you have the option to compile and use the program in Terminal, or you can just run it from Xcode, which is what you will do here.

Before running the app, let’s explore the key components that make photogrammetry possible. With the project open, select main.swift.

Checking Supported Hardware

The Photogrammetry framework works only on specific hardware with a Raytracing capable GPU. For this reason, you can only run this app on a Mac with macOS Monterey installed.

The first check tests for object reconstruction:

private func supportsObjectReconstruction() -> Bool {
  for device in MTLCopyAllDevices() where
    !device.isLowPower && 
	device.areBarycentricCoordsSupported &&
    device.recommendedMaxWorkingSetSize >= UInt64(4e9) {
    return true
  }
  return false
}

Here, supportsObjectReconstruction() verifies that at least one Metal device meets the minimum requirements for object reconstruction. It must be a high-power device with at least 4GB of RAM, with barycentric coordinates to the fragment shader.

The second check tests for a Raytracing capable GPU:

private func supportsRayTracing() -> Bool {
  for device in MTLCopyAllDevices() where 
    device.supportsRaytracing {
    return true
  }
return false
}

Here, supportsRayTracing() iterates through all the available Metal devices until one is found that supports Raytracing. Excellent, it looks like you’ve got the hardware, so let’s move on.

PhotogrammetrySession Configuration

The magic component that does all the work is the PhotogrammetrySession. Before you can start a request, you first need to configure the session:

typealias Configuration = PhotogrammetrySession.Configuration

This code defines an alias named Configuration for the session configuration. The command-line parameters are then parsed and processed to set the following:

• SampleOrdering. Defines the order of the images. Are they out of order or sequenced?
• FeatureSensitivity. Defines the precision of landmark detection. Should it be normal or high?

PhotogrammetrySession Request

With the configuration in place, you can now create the photogrammetry request:

typealias Request = PhotogrammetrySession.Request

This code defines an alias named Request for the session request. Additional command-line parameters provide the request settings:

• Detail. Defines the level of detail for the created geometry of the created object, ranging from preview, reduced, medium, full to raw.
• ModelFile. Defines the path for the created object as a .USDZ file.

The last step in the process is to create the session:

var maybeSession: PhotogrammetrySession? = nil
do {
  maybeSession = try PhotogrammetrySession(
    input: inputFolderUrl,
    configuration: configuration)
      logger.log("Successfully created session.")
} catch { ... }

This code provides the input path to the source images along with the session configuration. Finally, the session is created and kicked off.

Monitoring Session Output

While the session is running, you can monitor the session.outputs for any information:

for try await output in session.outputs {
  switch output { ... }
}

You can expect the following output:

• processingComplete. Indicates that the processing is complete.

• requestError. Indicates that some error occurred.

• requestComplete. Indicates that the request is complete.

• requestProgress. Provides feedback as each fraction completes.

• inputComplete. Indicates the input data has been processed and that the processing will continue.

• invalidSample. Indicates that the provided sample is invalid.

• skippedSample. Indicates that a provided sample has been skipped.

• automaticDownsampling. Indicates that a provided sample was automatically downsampled.

• processCancelled. Indicates that the process has been canceled.

Testing the Photogrammetry App

All right, enough about the code; let’s take this app for a spin!

If you’ve got a collection of photos ready, copy them into a folder on your desktop. Alternatively, you can use one of the examples that Apple provides. For instance, this slice of Fruit Cake.

Download the example from Apple, and extract the files into a folder on your desktop. Then, take a moment to explore the photos.

Oh my, wouldn’t that be lovely with a cup of coffee?

With main.swift open, add the following code to HelloPhotogrammetry.main():

HelloPhotogrammetry.main([
  "/Users/<YourName>/Desktop/FruitCakeSlice",
  "/Users/<YourName>/Desktop/FruitCakeSlice.usdz",
  "--detail", "preview",
  "--feature-sensitivity", "normal",
  "--sample-ordering", "sequential"
])

This code provides a few command line parameters directly to the app from within Xcode:

• Path to Images. This is where the input images are located. Be sure to set the path correctly for your machine.
• Path to USDZ. This is where the output USDZ will be generated. Be sure to set the path correctly for your machine.
• Detail. This sets the required detail to preview. This is great for quick tests. Feel free to change this value to a higher detail once you’re happy with the results.
• Feature Sensitivity. This sets the sensitivity to normal, but you can intensify it by changing it to high.
• Sample Ordering. This indicates that the images are provided sequentially; if this is not the case, you can change it to unordered.

To start the process — which might take a while — build and run the project.

If all went well, you’ll see the following message in the output:

[HelloPhotogrammetry] Processing is complete!

You’ll also see a FruitCakeSlice.usdz file on your desktop. Double-click the file to preview it with Xcode.

Amazing — and now I’m hungry!

Introducing PhotoCatch

Not a fan of command-line apps? No problem! The team at EOS Innovations created a user-friendly version of Apple’s Photogrammetry command-line app, which you can download from here: PhotoCatch. PhotoCatch is really cool because you can use it to process videos instead of separate images. Let’s try it out.

Launch PhotoCatch, and choose Select a Video.

Locate and select the Rock.m4v file under the provided materials, then choose Open.

There’s a total of 900 images in the video file; you’re only going to process every third frame, so change the amount from 4 to 3. Then, select and set the first frame.

Scroll all the way to the right, and set the last frame. Then, select Done at the top right.

Review the conversion settings. Set the File Format to USDZ, and set Model Quality to Preview. Leave the Sample Ordering as Unordered, and the Feature Sensitivity as Normal.

Select Create Model at the bottom to start the conversion process.

Sit back and sip your coffee while you wait for the photogrammetry process to complete. (See, I told you that coffee was an essential part of this process!)

Once done, you’ll see a cute little rock with a bee carved into it. Open the Crop section on the left and drag the Bottom slider slightly to the right to clip the un-even bottom of the rock.

Fantastic, it’s looking real pretty now. Select the Save option and export it as Rock.usdz on your Desktop.

Finally, double-click Rock.usdz to preview your model in Xcode.

Wasn’t that fun? What a great result, you rock! :]

Key Points

Congratulations, you just became a master artist capable of creating hyper-realistic 3D models in just a few basic steps.

Let’s take a quick look at some of the key take-aways from this chapter:

• Photogrammetry. A process where multiple photos are analyzed to reproduce a digital version of real-life objects.
• Building a DYI photo booth. It’s super easy to build your own little photo booth that will simplify the photo capture process.
• Taking Photos. There are a few gotchas to avoid when taking photos of objects. The most important part is good lighting and properly overlapping images.
• Photogrammetry command line. You’ve reviewed Apple’s example app, and now you know all the basic requirements for when you want to create your photogrammetry app.
• PhotoCatch. You also tried out a super cool variant that takes user-friendly to the next level. To top things off, you can now take a little video clip and quickly process that through PhotoCatch. Amazing!

Now, finish off that cup of coffee so that you can quickly capture it into digital format. Enjoy!