Beginning Automated Testing With Xcode Part 1/2

Note from Ray: This is the tenth and final iOS 6 tutorial in the iOS 6 Feast! This tutorial comes from our new book iOS 6 By Tutorials. Charlie Fulton wrote this chapter – a friend of mine and one of the newest members of the Tutorial Team. Enjoy! This is a blog post by […] By Charlie Fulton.

Login to leave a rating/review
Save for later

Note from Ray: This is the tenth and final iOS 6 tutorial in the iOS 6 Feast! This tutorial comes from our new book iOS 6 By Tutorials. Charlie Fulton wrote this chapter – a friend of mine and one of the newest members of the Tutorial Team. Enjoy! This is a blog post by […]

Learn how to add automated testing into your iOS 6 apps!

Note from Ray: This is the tenth and final iOS 6 tutorial in the iOS 6 Feast! This tutorial comes from our new book iOS 6 By Tutorials. Charlie Fulton wrote this chapter – a friend of mine and one of the newest members of the Tutorial Team. Enjoy!

This is a blog post by iOS Tutorial Team member Charlie Fulton, a full time iOS developer who enjoys hunting, fishing, and hanging out with his family.

All developers have to test their software, and many of the smart ones create test suites for this purpose. A test suite is a collection of test cases, also known as unit tests, targeting small “units” of code, usually a specific method or class.

Unit tests allow you to employ test-driven development practices and ensure that your code has fewer bugs. And by creating a unit test for each bug found, you can help make sure the bug never comes back! By unit testing and debugging proactively, you significantly narrow the chances that your app will behave unexpectedly out of the development environment.

But as you know, we’re often rushed to get finished – or we get lazy – so automating your unit tests to run automatically each time you change your code is even better. This is especially important when you’re working as part of a team.

So if you could put in place a system to automatically build, test, and then submit your app to beta testers, would you be interested? If so, then read on!

This bonus chapter and the next will walk you through setting up an automated building and testing system for your iOS apps. You will take an example project, add some unit tests to it, add a remote GitHub ( repository, and then set up a continuous integration (CI) server with Jenkins. Jenkins will periodically check your GitHub repository for changes and automatically build, test, and submit your app to TestFlight!

If you’re new to automated testing, you might be wondering what some (or all) of the above terms are, including GitHub, Jenkins, and TestFlight. We’ll go into each of these in detail in this tutorial, but for now here are instant micro-introductions to these tools:

  • GitHub – Free public Git repositories, collaborator management, issue tracking, wikis, downloads, code review, graphs, and much more!
  • Jenkins – A continuous integration server and scheduler that can watch for changes to Git repositories, kick off builds and tests automatically, and notify you of results.
  • TestFlight – A service to automate the sending of iOS apps over the air to testers, and perform crash log analysis.

That ought to tide you over for a short while, giving me time to introduce the geektastic example project that will be your vehicle for this educational odyssey – but be ready to learn more about all of this in the coming pages!

Introducing GuildBrowser

In this tutorial, you will be creating an automated build and test system for an app called GuildBrowser. GuildBrowser is a simple app that lets you browse the members of a guild from the popular game World of Warcraft. You can enter a guild and a realm name to see all of the members listed in a UICollectionViewController, as you can see below:

The app is driven by the freely-available WOW API from Blizzard. You will be using the AFNetworking framework to make RESTful calls to the WOW API, retrieving JSON data that contains guild and character information.

If you never have time to play games, the next best thing is making apps that help with games!

Note: For more details about the WOW API, check out Blizzard’s GitHub project (, the official WOW API docs (, and the WOW forum devoted to the community platform API (

Gitting started

If you’re not using some form of version control for your code these days, then you’re definitely not doing it right! Arguably the most popular system right now is Git. Linus Torvalds (yes, he who created Linux) designed and developed Git as a better system than its predecessor, SVN, to manage the code for the Linux kernel. Not only is git a great source control system, but it’s integrated directly into Xcode, which makes it very convenient to use in iOS development.

Git is a “distributed” version control system, meaning that every Git working directory is its own repository, and not dependent upon a central system.

However, when you want to share your local repository with other developers, you can define a connection in your “remotes” setting. Remote repositories are versions of your project that are usually set up on a central remote server. These can even be a shared via a Dropbox folder (although this isn’t particularly recommended).

A number of services have emerged in the last few years to make it easier to set up remote repositories for Git, and other distributed version control systems. Github has become one of the best and most popular of these services, and for good reason. It offers free public Git repositories, collaborator management, issue tracking, wikis, downloads, code review, graphs and much more… sorry to sound like an advertisement, but Github is pretty awesome!

Note: For a more detailed description of Git and how you can work with it, check out the tutorial on How To Use Git Source Control with Xcode in iOS 6, coming soon to

Your first action in this tutorial will be setting up a remote repository (repo for short) on GitHub. It will be your remote “origin” repo. It will allow you to work locally as normal, and then when you are ready to sync with teammates or send a new build to testers, you’ll push all of your local commits to this repo.

Note: This tutorial assumes you already have a GitHub account with SSH key access setup. If you do not, please consult this guide (, which shows you how to get set up by generating and using an SSH key pair. You can also use an https URL with your GitHub username/password, but the guide assumes SSH key access setup.

First, make sure you have the command line tools installed, choose the Xcode/Preferences/Downloads tab, and make sure your screen looks similar to this (the important thing is to have Installed next to Command Line Tools):

This tutorial comes with a starter project, which you can download here. Copy the project to a location of your choice on your hard drive, and open the GuildBrowser project in Xcode.

The GuildBrowser app was created with the Single View Application template using ARC, Unit testing, Storyboards, iPad-only, and a local Git repository. Compile and run, and you should see the app showing some characters from the default guild.

Open a browser, go to and login to your account, or create an account if don’t have one already. Then, click on the Create a New Repo button on the top-right corner:

Enter whatever you like for the Repository name and Description fields – the name does not actually have to correspond with the name of your Xcode project. Then click the Create repository button.

Note: Do NOT check the Initialize this repository with a README box. You want to create an empty repo and then push your Xcode project into it.

Get a copy of the repository SSH URL by clicking the copy to clipboard button or typing ⌘-C (remember to tap the SSH button first to switch to the SSH URL).

Now open Organizer (Cmd-shift-2) and then select the Repositories tab.

You should see the GuildBrowser project listed at the bottom. Click on the Remotes folder and then click the Add Remote button.

Enter origin for the Remote Name, and paste in your personal GitHub URL that you copied earlier into Location:

Click Create, and Organizer should look like the following:

Don’t be alarmed that there is no password entered for the origin location. You’ll be using Git’s SSH access, so no password is needed.

You now have a remote repo set up! You can use these steps for all of your Xcode projects. This is great, but you still have not pushed anything from your local Git repo to the remote Git repo. Let’s do that now with Xcode.

Go back to your project and first make sure you have committed all of your local changes. Xcode will not allow you to push to a remote repo until you have done that. Here are the steps:

  1. Go to File\Source Control\Commit (⌥⌘-C) and on the following screen, enter a commit message like “initial commit” and click Commit.
  2. Now you need to push the local master branch to the remote origin/master branch. Go to File\Source Control\Push and click Push.
  3. Since this is the first time you’re pushing to this repo, you should see (Create) beside the Remote name, origin/master. Tap the Push button, and Xcode will handle the rest.

Once the push is complete, go back to your browser, refresh the page, and the changes you committed should now show up in your GitHub repo:

You should also now see your remote repo and its master branch listed in Organizer\Repositories\Guild Browser project\Remotes:

Congratulations, you know how to connect your local Git repo to a remote repo on GitHub and push changes to it!

Now that you have a GitHub repository set up, you could invite others to clone your repository, accept pull requests, or take a look at your code. When someone else pushes their commits to the repo on GitHub origin/master repo you go to File\Source Control\Pull (⌥⌘-X), in the popup Remote should be set to origin/master, select Choose and those changes will be added to your project.

To see what has changed, go to Organizer\Repositories\GuildBrowser, expand the arrow and click on the View Changes button.

Continuous integration

Continuous integration (CI) is having your code automatically built and tested (and even deployed!) by a server that is continuously watching your revision control system for changes.

CI becomes tremendously important when you have a lot of developers on a project sending in their changes multiple times per day. If any conflicts or errors arise, everyone is notified immediately. Eventually, someone breaks the build!

Meet Jenkins

Jenkins ( is the open source CI server you’re going to use in this tutorial to automatically build, test, and deploy code that is pushed to the GitHub remote repo.

You’ll spend the rest of this section getting Jenkins up and running.

Installing Jenkins

There are a few different ways you can install Jenkins on a Mac:

  1. You can use the installer from the Jenkins website, which will set up Jenkins to run as a daemon. This is nice because it is configured to use the Max OS X startup system, but it will result in you, the user, having to go through a few more steps in order to get Xcode building to work properly.
  2. You can run the WAR file (a Java archive filetype) from the Terminal, which uses the built-in winstone servlet container to run Jenkins.
  3. You can deploy the WAR file to an application container you have already set up, such as Tomcat, JBoss, etc.

In this tutorial, you are going to run Jenkins from the Terminal, because it’s the quickest and easiest way to do it. Download the Jenkins WAR file from Open a Terminal window, and then run this command:

nohup java -jar ~/Downloads/jenkins.war --httpPort=8081 --ajp13Port=8010 > /tmp/jenkins.log 2>&1 &

Note: The above assumes that you have the Jenkins WAR file in your Downloads folder. If you downloaded the WAR file to a different location, adjust the path in the command accordingly.

Note: You might find it useful to create an alias for starting up Jenkins. From a Terminal, or text editor showing hidden files, open up /Users/(username)/.bash_profile and enter this line:

alias jenkins="nohup java -jar ~/jenkins.war --httpPort=8081 --ajp13Port=8010 > /tmp/jenkins.log 2>&1 &"

Save .bash_profile, open a new Terminal, and now type in jenkins to start it up.

This starts the Jenkins server on port 8081. The nohup command is a UNIX command that allows a process to keep running after you have logged out or the shell has exited – it stands for “no hangup”.

The above command also sets up a log file to /tmp/jenkins.log. I like to configure the ports to avoid any conflicts with servers I have running. To access Jenkins, open the following URL in a browser:

Configure Jenkins plugins

You need to add a few things to Jenkins before you can start setting up jobs. Open the Jenkins dashboard (via http://localhost:8081) and click on the Manage Jenkins\Manage Plugins\Available tab. There are a lot of plugins available, so filter the choices down by typing Git into the filter search box in the top right.

Here’s what you want:

Check the box next to the Git Plugin and then click the Install without restart button. You should see the following screen when it’s done installing:

Now enter Chuck Norris in the filter search box, and look for the Chuck Norris quote plugin. While this one is optional for the tutorial, I highly recommend you check it out! If you don’t install it, he will know! Beware the wrath of Chuck. :]

Note: My friends call me Charlie, but my real name is Charles. My Dad’s middle name was Norris. So I had legitimate claims to naming one of my sons Chuck Norris Fulton… but unfortunately she wouldn’t let me. :]

Setting up Jenkins email notification

It would be nice to have an SMTP server so that Jenkins can send you build notification errors. But if you’re not running an SMTP server, you can use your Gmail account with Jenkins (or any other SMTP server you prefer).

Go to Manage Jenkins\Configure System and in the Email Notification section, enter the following settings (use the Advanced Settings button to access some of the settings):

  • SMTP Server :
  • Sender Email address :
  • Use SMTP Authentication
  • User Name:
  • Password:
  • Use SSL
  • SMTP Port:465

Now test out email notifications by checking the Test configuration by sending test-email, entering an email address, and clicking the Test configuration button. Once you’ve confirmed that the test is successful, remember to tap the Save button to save your configuration changes.

Now you can receive emails from Jenkins about build failures.

Creating a Jenkins job

You are now ready to create a Jenkins job! Open the Jenkins dashboard, click on New Job, enter GuildBrowser for the job name, choose the Build a free-style software project radio button, and finally click the OK button. You should now see the job configuration screen.

In the Source Code Management section, click the Git radio button, and enter your GitHub remote repo (origin/master) URL into the Repository URL text field. Your screen should look something like this:

Go to the Post-build Actions section, select Email Notification from the Add post-build action button, enter the email address(es) for the recipient(s), and finally check Send email for every unstable build. You should see the following:

If you installed the Chuck Norris plugin, you can also add a post-build action to Activate Chuck Norris. That sounds kinda scary though! Warning, don’t break the build!!

Note: The programs that Chuck Norris writes don’t have version numbers because he only writes them once. If a user reports a bug or has a feature request, they don’t live to see the sun set. :]

Don’t forget to click the Save button.

Let’s make sure everything is set up correctly with GitHub by clicking the Build Now button on the left sidebar. When the job starts, you will see an update in the Build History box:

If everything works correctly, your build #1 should be blue, indicating success:

To see what exactly happened during the build, hover over the build and look at the Console Output:

Your console output should look similar to this, showing that the job successfully pulled down the code from the GitHub repo:

So far, you’ve connected to GitHub with your job and set up email notification. Let’s press on and add the interesting bits: testing, building, and uploading your project to TestFlight.

Note: A Jenkins pro tip! To quickly get to the job configure screen, hover over the job name on the breadcrumb trail at the top of the screen and select Configure from the menu. This is especially helpful when you’re looking at console output

Automating xcodebuild from Jenkins

Let’s update your Jenkins job to build the project. Go to the Jenkins Dashboard\GuildBrowser job\Configure. In the Build section, click on the Add build step dropdown select Execute shell.

Enter the following code into the Command window:

export DEVELOPER_DIR=/Applications/

xcrun xcodebuild clean build

Your command window should look like this:

Now click Save.
Let’s test the job again by clicking on Build Now. When the job starts executing, pay attention! You might see the following screen, asking you if you want to allow Jenkins access to run codesign so that it can sign your code. Select Always Allow.

If you missed the prompt (because you were chasing your two year-old who was running off with your iPhone, for example), you will see this message in your failed job’s console output:

Command /usr/bin/codesign failed with exit code 1
The following build commands failed:
CodeSign build/Release-iphoneos/
(1 failure)
Build step ‘Execute shell’ marked build as failure

If this happens, simply try running the job again, but don’t miss the prompt (or let your two year-old run away with your iPhone) this time. :]

While writing this feast post, I seriously just had to use find my phone to locate my iPhone, the little rascal had stuffed it into the trunk of his toy car!!

Assuming you didn’t miss the prompt, then in your output you will see a message similar to this:

Unable to validate your application. – (null)

This means the build was successful! You can safely ignore the “unable to validate” message – it’s a known bug that we were able to verify at WWDC this year.

Note: There is a Jenkins plugin for building iOS apps named Xcodeplugin. It works great, but I think it’s best to understand exactly what is going on when building your app. You can achieve the same results as the plugin with some simple build scripts of your own.

What’s more, becoming dependent on the plugin carries some risks. If Apple changes how things work and that breaks the plugin, and it’s no longer updated, you will still be able to adapt if you know how things work.

Unit testing

Unit testing ensures that the output of your code remains unchanged as you are developing it. After establishing expectations of what a class will do, you write unit tests to verify these expectations remain true. Unit testing also allows you to make changes incrementally, seeing results step-by-step, which makes it easier to develop new features and make big changes.

Some other benefits of including unit tests in your projects are:

  • Reliability – With a good set of tests, you will most likely have fewer bugs. This is especially true with code that was developed with tests from the beginning, verifying every change and new bit of code along the way.
  • Code confidence – It’s a lot less nerve-wracking to go in and refactor a massive amount of code with unit tests in place, knowing that your code should still be producing the same results and passing tests.
  • Stability – When bugs are found, new tests can be added ensuring the same bugs don’t harass you again.

Xcode has built-in support for unit tests via the SenTestingKit framework. When writing unit tests, there are two terms to be aware of:

  • Test case – The smallest component of a unit test is the test case. It is what verifies the expectations of what your unit of code should produce. For example, a test case might focus on testing a single method, or a small group of methods in your class. In Xcode, all test cases must start with the name test. You will see an example of this below.
  • Test Suite – A group of test cases. This is usually a group of test cases against a particular class. In Xcode, there is a template for creating a test suite named Objective-C test case class. It produces a class that is a subclass of SenTestCase.

There are two general approaches to setting up unit tests: you can either test from the bottom up (testing each method/class individually), or the top down (testing functionality of the app as a whole). Both approaches are important – it’s usually good to have a combination of both.

In this tutorial, you are going to test from the bottom up, focusing on a few of the model classes. In iOS 6 by Tutorials, there is another chapter where you’ll try your hand at testing from the top down, from the User Interface to the model code via UI Automation.

Application vs. logic test targets

Next let’s take a look at the two different kinds of unit tests available in Xcode.

Application unit tests

An application unit test basically means any code that needs to use something in UIKit, or really needs to run in the context of a host app. This is the type of unit test target you get by default when creating a new project and including unit tests.

All of the code in this target will be run within your app bundle. You can see the difference by looking in the build settings for each target and checking the settings for Test Host and Bundle Loader:

Logic unit tests

For now, this is the only type of unit test you can run on your units of code from the command line. This code is tested in isolation from the app.

These are unit tests that test your own code, basically anything that doesn’t need to run in the Simulator. It’s great for testing custom logic, network access code, or your data model classes.

Running tests from command line notes – no touching!

Unfortunately, as of Xcode 4.5 we are still not “officially” allowed to run application unit tests via the xcodebuild command. This is very frustrating, because you obviously can run application unit tests from within Xcode. For this reason, you will create a special logic test target in the next section.

Some creative developers have made patches to the testing scripts inside of the contents. For this tutorial, you are going to stick to running your logic tests from Jenkins, and application tests from Xcode. If you’re curious about the patches, take a peek at these files:


This is the script that runs by default when you run unit tests via ⌘-U:


Here is the error message you’ll get, and the reason we can’t run application unit tests in the Simulator from the command line (remember that this works fine from within Xcode):

RunTestsForApplication() {
Warning ${LINENO} “Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).”

Some clever hacks have gotten application tests to work, but the scripts have changed from Xcode 4.4 to 4.5, so do so at your own risk.

Adding a logic unit test target

You are now going to create a new test target that can be called from your build script.
In Xcode, Go to File\New\Target…\iOS\Other\Cocoa Touch Unit Testing Bundle and then click Next.

Enter GuildBrowserLogicTests for the target name, make sure that Use Automatic Reference Counting is checked, and then click Finish.

You will now have a new scheme with the same name. To test that everything is working, switch to the new scheme (using the scheme selector on the Xcode toolbar next to the Run and Stop buttons) and go to Product\Test (⌘-U).
You should see that the build succeeded, but the test failed.

As the error message indicates, this is because the unit tests have not been implemented yet. Let’s fix that!

First, delete the default test class by highlighting GuildBrowserLogicTest.h and GuildBrowserLogicTest.m, tapping Delete, and then selecting Move to Trash.

So that you don’t have to remember to switch schemes, let’s modify the main app scheme, GuildBrowser, to include your new target when running Product\Test.

Switch to the GuildBrowser scheme and then edit the scheme via Product\Edit Scheme (or you can simply Alt-click on the Run button). In the scheme editor, highlight the Test configuration, click on the + button to add a new target, select GuildBrowserLogicTests in the window and then click Add.

Your screen should look similar to this:

You are now ready to add unit test classes to this test target. You will then be able to call this target from your jenkins job.

You will do exaclty that in the next installment of this feast! Stay Thirsty.. err Hungry!

Where To Go From Here?

In a couple days, we’ll post part 2 of this tutorial. In the meantime, if you have any questions or comments, please join the forum discussion below!

This is a blog post by iOS Tutorial Team member Charlie Fulton, a full time iOS developer who enjoys hunting, fishing, and hanging out with his family.

Charlie Fulton


Charlie Fulton


Over 300 content creators. Join our team.