Getting Started with AI-Augmented Coding in iOS

Welcome to the first demo in lesson three. In this demo, you’ll learn about some of the new capabilities for AI agents in Xcode 27. My favorite capability is how the AI agents are able to see, so to speak, and interact with the running apps on the simulator. So let’s give this a shot.

I’ll ask the agent what does it see on the iPhone simulator. The agent took a screenshot of the whole simulator and will identify what are the elements on the screen, and will start to describe them. Let me close this a little. So it identified the title of the game, the game modes available, and how the scoring works. From here, I can show or hide the artifacts from the conversation. Let’s keep this hidden.

So the next step I’m going to do is tell it to actually start playing the game. Start a classic game, guess the country correctly in two rounds, then choose the wrong one in the third round. Choose it correctly on the next attempt, and end the round. Let’s see how this goes.

As you can see, the agent executed our instructions, succeeded twice, intentionally put a wrong value, and then put the third guess correctly. This capability is new in Xcode 27, but it’s not exclusively available in Xcode only. You can use any agent, any AI tool to run this. In the next lesson, you’ll see how to set this up through Cursor. See you then.