Using AI for Testing in iOS

Welcome to the second lesson of Testing with AI module. In this lesson, we will focus on the UI tests that the plan mentioned that they are meant to verify the app while it’s running in the simulator.

The difference between UI tests and unit tests is you do not test code, you test the application as a whole. In Xcode, it basically creates a new test runner that takes control over the application itself. It’s not about testing functions, it’s asserting on elements on the screen.

We will not implement regular UI tests in this lesson. Instead, we will instruct the AI agent to take control over the app while running on the simulator and execute the test instruction itself and verify if everything is working as it should be. Let’s get started.

I have the project open, and this is what was completed in the previous lesson. There is an already existing skill called device-interaction. This skill gives instructions to the AI model on how to control the application, how to calculate touch areas, and some additional info. Feel free to look at the skill yourself, or even start it or invoke it, and ask the AI model to answer questions on your behalf, explaining how the skill itself looks like.

Let me use it. Run UI tests in testplan.md directly in the simulator within the AI session and report the findings. Those tests were meant to be implemented in TokensDashboardUITests.swift. Fix issues as you find them and confirm it’s fixed. Don’t implement actual UI tests.

I don’t want the AI agent to get lost or explore too much on what the test plan was supposed to do, as this file is already mentioned in the plan, so this gives it a direct way to figure out what I’m referring to. I want to include a couple of other important points to keep this cheap.

Use xcrun simcontrol commands to control the simulator settings for dynamic type or any accessibility settings. This is important because it’s very possible that the agent will start opening the simulator app, fig and take multiple screenshots to change the settings of the simulator while it’s possible to do it directly in the command line. Note that this works for a simulator, not for an actual device.

And I also want to run one test, not all the tests that are mentioned in the test plan because they might be a lot. And running this relies on screenshots, which can get expensive, so I want you to control yourself how many tests you want to run instead of giving a single prompt that ends up being too expensive. Let’s get started.

/device-interaction run UI tests in "test-plan.md" directly in the simulator within the AI session and report findings. Those Tests were meant to be implemented in TokensDashboardUITests.swift. Fix issues as you find them then confirm its fixed.
Don't Implement actual UI tests for now.
Use `xcrun simctl` commands to control the simulator settings for dynamic type or any accessibility settings.
Run only 1 test not all the cases and explain the findings as the test is running for Demo purposes

As you can see, the agent is taking screenshots of the application while it’s running, interacting with the scrolling to find the card that is meant to be opened, and to also ensure that visually everything is as expected. It opened the detail screen that we included or that we implemented. It’s also, as you can see, showing the list of screenshots that it took within the session directly. Please note that those screenshots can increase the costs of this session significantly.

The AI model did indeed find a bug, included its fix, and now it’s rerunning the application. There we go. We’re all done. And I have a summary of what actually happened. It’s explaining what the test that ran based on the definition or based on the information of the test within the plan, how was the setup, explaining simulator ID or the simulator name, its ID, the actual settings of the dynamic type. Here’s the work through, the findings, what was triaged, and the final result.

Giving an AI agent control over the simulator is very similar to… Well, maybe very similar is a strong word, but it can simulate human interactions where information on the screen is understood through the usage of screenshots and the actual interactions happened. The screenshots themselves are the expensive part.

You can improve this by using accessibility identifiers, and if the identifier is not found, then the AI model or the AI agent to take over and then take a screenshot afterwards. So go with the cheaper approach. If it didn’t work, instead of having flaky UI tests, an AI can actually take control afterwards as a fallback.

The potential from this is huge, but also it needs to be price-checked when you scale this to run instead of one test or a single fix to run multiple. Using this on a daily basis is not cost-efficient. You can use cheaper AI models, or you can have an open budget, but that doesn’t go well in the long term.

I hope you enjoyed this lesson, and best of luck.