Chapters
Chapters

Chapters

Hide chapters

Android Test-Driven Development by Tutorials

Second Edition · Android 11 · Kotlin 1.5 · Android Studio 4.2.1

Before You Begin

Section 0: 5 chapters
Show chapters Hide chapters

Section I: Introduction to Test-Driven Development

Section 1: 3 chapters
Show chapters Hide chapters

Section II: Testing on a New Project

Section 2: 8 chapters
Show chapters Hide chapters

Section III: TDD on Legacy Projects

Section 3: 8 chapters
Show chapters Hide chapters

Appendix

Section 4: 1 chapter
Show chapters Hide chapters

17. Continuous Integration & Other Related Tools
Written by Lance Gleason

Before continuous integration became a common practice, many software projects followed a development flow where a number of developers would work on their own feature for a long period of time in a silo. Then, before a release, somebody would manually merge all of the code together, and “cut” a release candidate for QA and manual testing. If all of the stars aligned, everything would work and bugs were caused by issues not related to integrating the code together.

What actually happened on a lot of projects was much uglier. Often there would be issues merging the code together. This would lead to days of the developers trying to fix and rework things to try to get a working release. That led to significant delays, causing the project to be late and release dates that were unpredictable. Instead of being a minor part of your development process, integration became a big, unpredictable multi-day, week, or event.

Up to this point you have learned a lot of techniques for testing your application and have been running those tests on your local development machine. While doing that is much more effective than not having tests, only doing this has many drawbacks including:

  • Situations where the tests run successfully on one developers machine but not another.
  • Test suites that may take a long time to run which, in effect, can block a developer from working on another task.
  • No common build status that all developers have visibility into.

Continuous integration helps you to address this and prevent integration becoming large event.

Continuous integration fundamentals

Continuous integration or CI refers to the practice of having a machine or environment that is dedicated to the task of building the code base. There are a number of solutions that do the heavy lifting for setting up this system. You will learn more about that later on in this chapter. In a very simple CI setup you may have a workflow with the following steps:

  1. When a user pushes up new changes to a Git repo, the CI server kicks off a new build.
  2. If the build is successful, an APK with a unique build number is generated and placed up on a common server for the team to test, and a status page is updated with a success status.
  3. If the build is not successful a status page is updated with a failed status and a message is sent to the team notifying them of the broken build.

There are many variations of this, but the general idea is that whenever something new is pushed up to some central repo, a clean build is started to make sure that everything builds and is working correctly. Many development teams with no automated tests will use this approach.

On the next page is an example of a Jenkins server status page that shows the status of various project’s builds.

The green check marks indicate the last build was successful, the ! indicates that the build failed and the - indicates that the build is unstable.

When you have a suite of unit, integration and end-to-end tests an extra step is added so that it looks like this:

  1. A build is kicked off on a commit to the repo.
  2. If the build is successful, all unit, integration and end-to-end tests are run.
  3. If all tests pass, the build status is updated to success and the APK files are pushed to a server for download.
  4. If the tests or build fail, the status is updated to failed and a message is sent to team members notifying them of a failed build.

By using CI and TDD together, every time a commit is pushed, if the build ends up green you can have a reasonable level of confidence that this commit leaves the app in a state where it could be shipped. Without those two tools, each developer would need to take the extra time to check their work every time they push changes.

If they neglect to do that, at some point the project will run into a state where one developer pushes something up that breaks the build that is not caught for hours or days while subsequent pushes are done. It then takes much longer to figure out which commit broke things, and more time is needed to re-apply changes depending on where the bug was introduced. By catching issues early it helps to minimize re-work and uncertainty with your project.

Branches and CI

Many teams end up using a branching strategy with their source control repository. One that is commonly used is called Git Flow https://nvie.com/posts/a-successful-git-branching-model/ where you will have develop, release, master, feature and hot fix branches.

If your entire test suite does not take a lot of time to run, a best practice is to have CI run whenever a new push is made to a branch.

But, if you have a large test suite that takes several minutes to run you may need to take a more sophisticated approach. To better understand this consider a test suite that has the following attributes:

  • Fifty percent of tests are unit tests that run quickly on the JVM. The entire suite of unit tests take one minute to execute.
  • Thirty five percent are integration tests that can run in either Robolectric or Espresso. The suite of integration tests take two minutes to execute in Robolectric and twenty minutes in Espresso.
  • Fifteen percent of the tests are end-to-end that mostly only run in Robolectric. This suite takes eight minutes to execute in Espresso.

Running tests in Espresso versus Robolectric give the most realistic test results, but they take twenty nine minutes to execute. In this scenario, running this test suite using Espresso on your local development machine before pushing up changes would not be a practical thing to do.

Most development teams facing this will:

  • Run tests related to a feature they are working on along with all unit and Robolectric tests locally.
  • Push it up to source control and let CI run the full test suite with Espresso.

With the CI test runs you have another issue. When using a branching strategy, a good practice is to have short running feature branches. Ideally that branch is only worked on for few hours, reviewed and merged them into the development branch. Waiting for a half an hour before merging feature branches would slow down integration. Especially on a larger team. There are multiple ways you could address that, but like most things in programming, there’s trade-offs for each one. Possible solutions include:

  • Only running Robolectric and unit tests via CI on feature branches.
  • Using a less frequent schedule, IE: once or twice a day, for running the full Espresso suite from your development or feature branches.

CI tools

CI solutions need to do a lot of things including:

  • Monitoring your source code repository for changes.
  • Managing and orchestrating build agents (computers or VMs that actually perform builds).
  • Offering some sort of dashboard or visualization of the build status.

When looking at CI tools there are two main categories you will run across:

  1. Self hosted solutions.
  2. Cloud based solutions.

Self hosted CI

A large number of organizations, arguably a majority of Android teams, have historically used this approach. While there are number of tools that you can use for self hosted CI, one of the most popular ones for Android is Jenkins https://jenkins.io/. Jenkins installations have two main components:

  1. A Build Server.
  2. One, or multiple build executors.

The build server is what handles things such as:

  • The project configuration.
  • Build pipeline through checkout, build, test, deploy, etc.
  • Coordinating build executors.

Build executors are the machines/VM’s where builds take place. Each executor that you use needs to have the proper tools installed to be able to compile and run your project. A build executor can only run one pipeline at a time. So let’s say that you have one developer working on feature branch A and another that is working on feature branch B. Developer A pushes up a commit that starts a CI pipeline. Before that build is completed a commit is pushed up to feature branch B. If your Jenkins system only has one build executor, it will not start that pipeline until the build executor is done working an the pipeline for branch A. If the test suite takes a while to run, or there are a lot of developers on the project this could cause a CI backlog. To address this you can add additional build executors which will allow multiple branch pipelines to run at the same time.

Advantages of these self hosted solutions include:

  • More control over your build environment.
  • Less expensive than using a cloud based solution, especially when running a lot of builds.
  • The ability to add your own physical devices to perform automated tests on.
  • Better security when working in a high security environment or behind a corporate firewall.

Disadvantages include:

  • Significantly more work involved in managing server instances.
  • Build executors need to be updated when updating versions of Android tools.
  • Headless servers can provide extra configuration challenges for Espresso based tests.

Cloud based CI

Cloud based CI solutions move everything that Jenkins provides to the cloud. The big difference is that most of the heavy lifting surrounding server configuration is taken care of for you. Some of the more popular services include:

  • CircleCI
  • Bitrise
  • Travis CI

Each service has its own nuances, but in general they have have different plans that go up in price based on the concurrency and total system build/run time that you need. All of these services also have free plans for open source software projects.

Advantages of these solutions include:

  • Significantly less effort is needed to configure and manage server instances.
  • Updates for build tools are often taken care of by the service.
  • Scaling for more concurrency can be done by changing your plan.
  • Free plans for open source projects.

Disadvantages include:

  • Generally, less control over your build agents.
  • Limited support for running Espresso suites on your own test devices.
  • No support for high security environments behind firewalls.
  • A system needing a lot of concurrency and build time can get expensive.

Device farms

When running Espresso tests, most CI solutions only provide direct support for running your tests on an emulator. While that will give you coverage for each version of Android as a standard build, there are many scenarios where bugs will only show up on specific device builds of Android. Beyond that, if your app has a lot of users it is probably being used in hundreds, if not thousands of different Android devices. Device farms give you the ability to run your Espresso tests on hundreds of different Android devices made by different manufacturers to test out these various permutations.

Depending on your needs, you can even opt to run all of your CI Espresso tests in a device farm. There are a number of device farm providers. Two popular ones are:

  1. Firebase Test Lab
  2. AWS Device Farm

As is the case with cloud based CI offerings, the more device time you use, the more the farms will charge you.

CI strategy guidelines

Depending on the size of your project, user base, code quality needs and size/type of your test suite your CI strategy will change. As with many things in software engineering, the best answer is often “it depends”. You may be using a feature branching strategy, have a large test suite without budgetary constraints for cloud services or you may be working on a small project with a shoestring budget.

Wait time

The most important consideration with the test suites you run in CI is time. Ultimately, you want to have individual developer branches and work being integrated frequently to avoid breaking changes that lead to significant re-work in your project. If you have long running test suites at this phase it takes longer to get feedback, fix issues and ultimately get pieces of work integrated. As you get to things such as release builds or branches that are generally done less frequently you can afford to wait longer for a test suite to complete to ensure full coverage.

Here are some guidelines based on branch commit frequency:

  • One or more commits per hour - test suites that run in less than five minutes.
  • One to two commits per day - test suites that run in less than thirty minutes.
  • Releases or less than once per day - all test suites.

Test service cost

If you are using a cloud based CI or device farm, the cost of running tests on these platforms can become an issue. Developer time is generally your largest project cost. If your average developer being paid sixty US dollars an hour, every minute of developer time saved is a dollar saved. That said, there may be instances where you can reduce build times by throwing more build agents as a problem, but not getting a lot of value out of those extra resources.

An example of this might be where you have a large, modularized application with a long running Espresso test suite. One approach that some teams have taken to reduce test execution time is to “shard” (divide up) the test runs so that they can be executed in parallel. Doing this could allow you to run a test suite that takes 30 minutes in less than five minutes by breaking the work up among multiple build agents as part a complex build pipeline. That will reduce your developer time, but if you are providing enough resources to allow all of these tests to run for feature branches it will also significantly increase you CI service cost. This will give you better coverage by executing more tests frequently, but is that extra frequency providing enough value to offset the cost?

Ultimately, you want to have all of your tests run in CI before cutting QA and Release versions of your app, but there may be times where the cost to run some of the more expensive test frequently may not be offset by the benefits.

Device coverage

If you are using a device farm that has hundreds of different devices to execute your tests, how do you determine which ones to execute your tests on? In an ideal world, the answer would be, all devices all of the time. Unfortunately, in reality that would be a very expensive proposition.

For example, lets take a look at the cost, at the time of this book writing, to run an Espresso test suite in Firebase Test Lab. This suite takes six minutes to run on a device, and you want to run it on one hundred different Android devices:

  • 100 devices * 6 minutes = 600 device minutes or 6 device hours
  • 1 Test lab physical device hour costs $5.00 USD
  • 6 device hours * $5.00 = $30.00 USD

If you average twenty work days in a month and ran these tests once a day on work days your device farm cost would be $600 USD. For a small project that cost may not justify the benefit. If your project could justify that, but you ran these this for all daily commits it would become cost prohibitive.

There are no magic bullets on how to address this but here are some ideas to figure out the best strategy:

  • Look at your applications analytics and only choose the devices that have the most users.
  • Run tests more frequently on devices that have surfaced more device specific bugs.
  • Run frequently executed test runs on a randomly selected device from a list.
  • Only run your suite on a large number of devices infrequently, i.e. before cutting a release candidate.

Key points

  • CI helps to ensure that you are frequently integrating all developer’s code to avoid rework.
  • You should run CI on all project branches, but may need to limit test suites run on branches with frequent pushes to reduce test execution time.
  • Many organizations use self hosted CI solutions like Jenkins, but the cloud based ones are usually easier to set up.
  • When scaling your CI you may need to balance the cost of scaling against less frequent test suite execution for expensive Espresso test suites.
  • Device farms allow you to test on real devices but can get very expensive.
  • There are no magic bullets when figuring out the best CI strategy.

Where to go from here?

16. Strategies for Handling Test Data
18. Testing Around Other Components
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2026 Kodeco Inc.