Continuous Integration & Other Related Tools
Written by Lance Gleason
Heads up... You're reading this book for free, with parts of this chapter shown beyond this point astext.
Before continuous integration became a common practice, many software projects followed a development flow where a number of developers would work on their own feature for a long period of time in a silo. Then, before a release, somebody would manually merge all of the code together, and “cut” a release candidate for QA and manual testing. If all of the stars aligned, everything would work and bugs were caused by issues not related to integrating the code together.
What actually happened on a lot of projects was much uglier. Often there would be issues merging the code together. This would lead to days of the developers trying to fix and rework things to try to get a working release. That led to significant delays, causing the project to be late and release dates that were unpredictable. Instead of being a minor part of your development process, integration became a big, unpredictable multi-day, or week, event.
Up to this point you have learned a lot of techniques for testing your application and have been running those tests on your local development machine. While doing that is much more effective than not having tests, only doing this has many drawbacks including:
- Situations where the tests run on one developers machine but not another.
- Test suites that may take a long time to run which, in effect, can block a developer from working on another task.
- No common build status that all developers have visibility into.
Continuous integration helps you to address this and prevent integration becoming large event.
Continuous integration fundamentals
Continuous integration or CI refers to the practice of having a machine or environment that is dedicated to the task of building the code base. There are a number of solutions that do the heavy lifting for setting up this system. You will learn more about that later on in this chapter. In a very simple CI setup you may have a workflow with the following steps:
- When a user pushes up a new some changes to a Git repo, the CI server kicks off a new build.
- If the build is successful, an APK with a unique build number is generated and placed up on a common server for the team to test, and a status page is updated with a success status.
- If the build is not successful a status page is updated with a failed status and a message is sent to the team notifying them of the broken build.
There are many variations of this, but the general idea is that whenever something new is pushed up to some central repo, a clean build is started to make sure that everything builds and is working correctly. Many development teams with no automated tests will use this approach.
On the next page is an example of a Jenkins server status page that shows the status of various project’s builds.
The green check marks indicate the last build was successful, the ! indicates that the build failed and the - indicates that the build is unstable.
When you have a suite of unit, integration and end-to-end tests an extra step is added so that it looks like this:
- A build is kicked off on a commit to the repo.
- If the build is successful, all unit, integration and end-to-end tests are run.
- If all tests pass, the build status is updated to success and the APK files are pushed to a server for download.
- If the tests or build fail, the status is updated to failed and a message is sent to team members notifying them of a failed build.
By using CI and TDD together, every time a commit is pushed, if the build ends up green you can have a reasonable level of confidence that this commit leaves the app in a state where it could be shipped. Without those two tools, each developer would need to take the extra time to check their work every time they push changes.
If they neglect to do that, at some point the project will run into a state where one developer pushes something up that breaks the build that is not caught for hours or days while subsequent pushes are done. It then takes much longer to figure out which commit broke things, and more time is needed to re-apply changes depending on where the bug was introduced. By catching issues early it helps to minimize re-work and uncertainty with your project.
Branches and CI
Many teams end up using a branching strategy with their source control repository. One that is commonly used is called Git Flow https://nvie.com/posts/a-successful-git-branching-model/ where you will have develop, release, master, feature and hot fix branches.
CI solutions need to do a lot of things including:
Self hosted CI
A large number of organizations, arguably a majority of Android teams, have historically used this approach. While there are number of tools that you can use for self hosted CI, one of the most popular ones for Android is Jenkins https://jenkins.io/. Jenkins installations have two main components:
Cloud based CI
Cloud based CI solutions move everything that Jenkins provides to the cloud. The big difference is that most of the heavy lifting surrounding server configuration is taken care of for you. Some of the more popular services include:
When running Espresso tests, most CI solutions only provide direct support for running your tests on an emulator. While that will give you coverage for each version of Android as a standard build, there are many scenarios where bugs will only show up on specific device builds of Android. Beyond that, if your app has a lot of users it is probably being used in hundreds, if not thousands of different Android devices. Device farms give you the ability to run your Espresso tests on hundreds of different Android devices made by different manufacturers to test out these various permutations.
CI strategy guidelines
Depending on the size of your project, user base, code quality needs and size/type of your test suite your CI strategy will change. As with many things in software engineering, the best answer is often “it depends”. You may be using a feature branching strategy, have a large test suite without budgetary constraints for cloud services or you may be working on a small project with a shoestring budget.
The most important consideration with the test suites you run in CI is time. Ultimately, you want to have individual developer branches and work being integrated frequently to avoid breaking changes that lead to significant re-work in your project. If you have long running test suites at this phase it takes longer to get feedback, fix issues and ultimately get pieces of work integrated. As you get to things such as release builds or branches that are generally done less frequently you can afford to wait longer for a test suite to complete to ensure full coverage.
Test service cost
If you are using a cloud based CI or device farm, the cost of running tests on these platforms can become an issue. Developer time is generally your largest project cost. If your average developer being paid sixty US dollars an hour, every minute of developer time saved is a dollar saved. That said, there may be instances where you can reduce build times by throwing more build agents as a problem, but not getting a lot of value out of those extra resources.
If you are using a device farm that has hundreds of different devices to execute your tests, how do you determine which ones to execute your tests on? In an ideal world, the answer would be, all devices all of the time. Unfortunately, in reality that would be a very expensive proposition.
- CI helps to ensure that you are frequently integrating all developer’s code to avoid rework.
- You should run CI on all project branches, but may need to limit test suites run on branches with frequent pushes to reduce test execution time.
- Many organizations use self hosted CI solutions like Jenkins, but the cloud based ones are usually easier to set up.
- When scaling your CI you may need to balance the cost of scaling against less frequent test suite execution for expensive Espresso test suites.
- Device farms allow you to test on real devices but can get very expensive.
- There are no magic bullets when figuring out the best CI strategy.
Where to go from here?
- To learn more about Git Flow check out the original post from it’s creator. https://nvie.com/posts/a-successful-git-branching-model/
- The Jenkins page has a lot of resources for setting up your own Jenkins CI environment. https://jenkins.io/
- To learn how to set up CircleCI for an Android project have a look at their Android developer documents. https://circleci.com/docs/2.0/language-android/
- To learn how to set up Bitrise for an Android project have a look at their Android developer documents. https://devcenter.bitrise.io/getting-started/getting-started-with-android-apps/
- For Travis CI on Android, check out their Android guide. https://docs.travis-ci.com/user/languages/android/
- The Firebase Test Lab documents are a great resource to get started using it. https://firebase.google.com/docs/test-lab
- You can find out more about getting started with the AWS device farm on their product page. https://aws.amazon.com/device-farm/