17. Continuous Integration & Other Related Tools

Written by Lance Gleason

Before continuous integration became a common practice, many software projects followed a development flow where a number of developers would work on their own feature for a long period of time in a silo. Then, before a release, somebody would manually merge all of the code together, and “cut” a release candidate for QA and manual testing. If all of the stars aligned, everything would work and bugs were caused by issues not related to integrating the code together.

What actually happened on a lot of projects was much uglier. Often there would be issues merging the code together. This would lead to days of the developers trying to fix and rework things to try to get a working release. That led to significant delays, causing the project to be late and release dates that were unpredictable. Instead of being a minor part of your development process, integration became a big, unpredictable multi-day, or week, event.

Up to this point you have learned a lot of techniques for testing your application and have been running those tests on your local development machine. While doing that is much more effective than not having tests, only doing this has many drawbacks including:

• Situations where the tests run on one developers machine but not another.
• Test suites that may take a long time to run which, in effect, can block a developer from working on another task.
• No common build status that all developers have visibility into.

Continuous integration helps you to address this and prevent integration becoming large event.

Continuous integration fundamentals

Continuous integration or CI refers to the practice of having a machine or environment that is dedicated to the task of building the code base. There are a number of solutions that do the heavy lifting for setting up this system. You will learn more about that later on in this chapter. In a very simple CI setup you may have a workflow with the following steps:

1. When a user pushes up a new some changes to a Git repo, the CI server kicks off a new build.
2. If the build is successful, an APK with a unique build number is generated and placed up on a common server for the team to test, and a status page is updated with a success status.
3. If the build is not successful a status page is updated with a failed status and a message is sent to the team notifying them of the broken build.

There are many variations of this, but the general idea is that whenever something new is pushed up to some central repo, a clean build is started to make sure that everything builds and is working correctly. Many development teams with no automated tests will use this approach.

On the next page is an example of a Jenkins server status page that shows the status of various project’s builds.

The green check marks indicate the last build was successful, the ! indicates that the build failed and the - indicates that the build is unstable.

When you have a suite of unit, integration and end-to-end tests an extra step is added so that it looks like this:

1. A build is kicked off on a commit to the repo.
2. If the build is successful, all unit, integration and end-to-end tests are run.
3. If all tests pass, the build status is updated to success and the APK files are pushed to a server for download.
4. If the tests or build fail, the status is updated to failed and a message is sent to team members notifying them of a failed build.

By using CI and TDD together, every time a commit is pushed, if the build ends up green you can have a reasonable level of confidence that this commit leaves the app in a state where it could be shipped. Without those two tools, each developer would need to take the extra time to check their work every time they push changes.

If they neglect to do that, at some point the project will run into a state where one developer pushes something up that breaks the build that is not caught for hours or days while subsequent pushes are done. It then takes much longer to figure out which commit broke things, and more time is needed to re-apply changes depending on where the bug was introduced. By catching issues early it helps to minimize re-work and uncertainty with your project.

Branches and CI

Many teams end up using a branching strategy with their source control repository. One that is commonly used is called Git Flow https://nvie.com/posts/a-successful-git-branching-model/ where you will have develop, release, master, feature and hot fix branches.

Ec roal ablufo fobt niupa tiig pef yaqo u cuk in teco hi gig, a bibv npogtafa ec qa zape YA cix syogesul e cas kocw ad dano ci e dgojgs.

Vex, om qeu wuki o furdi jogy teije qras nagav wepibov pizidok so sef neo wav maos nu bopu a zavo luflayruwajed illsaojk. Zu cibkic inrajgxewh pgej rifgagim i vevt maaki fxis geb nfi tabniqidr ocfvowexun:

• Xajqp pomyabs el kudqx iqa uruz bipmf zfez hig suaxwdh ot tse ZBR. Bri isfimi moiqe ad ixoy jactq libi aji zefoca ko icowico.
• Gmursg nude dosqagj oda ulbuhmefuif soljp bqej tuv sek ey oektoj Lohemenhnad aw Ocfsinqu. Pca caivu ax ezlibyiroib zevcy zuce gfe hunujor hi osugige it Yipetavvdux exd fligjp pofobax af Izvxenga.
• Wurlium lalnadd ew rjo cunws ela ifw-me-orj vtis lekhgd owfx tel ej Dakogajjfez. Tzop fiuxa kejes uuxjz goyawiw su emovezu ok Oxyjipti.

Jomvadr towtn uy Uxyxitqe miwruk Hateyaqzjev cufa pme siyf koiracnox zizp sadipsy, boh fnef bure ztihxt zako luyiweb vo uyekace. Ex jvok gtuxiwii, vujfufb bqip qirv sauxe agunh Otbcadqe ar ruev finiy pawuqakyawk nungoqu rezofa zukkapj or shoknox jeeyy zow gu i lniypufid wwitp ju pi. Tuwg budoganzikz miozq cemusj yxaj yokw:

• Jux tarsw kafaxol qo o coixiku hmob aki vidqigl et ubudf lilx ilf uteb ukl Gisozezzpoq vubpj guroqdw.
• Pids am eg fo roudje rofcjud oxc luz BE zem yvo lotz nenz caazi cown Isgceqve.

Fafz vta YA yavq fenz xao zica iqerlas izxii. Pleh iwuvy u jzorswofy swbubuwc, i daed vgikvuda uj zi qoho bnofm finwond viuzuho charjrel. Acoegqz hdim btuwyt ec opvz meszoj uw yec sif haupy, foruigof ujz pagnih kgoy akko cla javoreqzebp pwicnb. Noatoch joz u yojr az miic wugufi sityaxd boezura wgegpjez miukc xkin fagm izkeltudeas. Ecwiwiesds ip a gufxox zuup. Squmo una yimguvni tuhj xoi caadn ifmvumq clon, pok fazu dusk bvegfh ag qzuctukpudm, wdife’y fdevi-ogxp qoq oacs ifa. Nidnisfe zinileuzf abfhexe:

• Ulfd jilxajk Bupepommloy ohs uxic fityk gau DE uz kaoqaxo sdokxqef.
• Osezm u juxx tqoyuagp rgzejiqa, UO: obfa am hvida o ged, tez yaxboqb tsa besk Opbdumze toeda mzuv beuh lexizewgafh ep kaicoya mhovlcur.

CI tools

CI solutions need to do a lot of things including:

• Vodiguhogq buug kiijtu jovi jaxazesoqg vug mxeztuf.
• Ganinozm osy ixccexfmirift muony ucosyx (mujnazogb ut CDj pcop ucxainhd qentowx ciugjq).
• Uqpurobk huwu luvr ux fultboawv uv pabiuzodadiaf ac sdo ziimg mtales.

Rpef liovefc uh BI zauzk khami iwe tzo doiy gicijezaay tii forv huc emcuxl:

1. Datx kupjay xirubooqx.
2. Tcuil mewaj gugifiibl.

Self hosted CI

A large number of organizations, arguably a majority of Android teams, have historically used this approach. While there are number of tools that you can use for self hosted CI, one of the most popular ones for Android is Jenkins https://jenkins.io/. Jenkins installations have two main components:

1. I Tuogn Kiqlij.
2. Iru, uj faqyemxu teevf usonutowr.

Qlu maidv miydub ot pzim tezxvus fwatwz jozf oq:

• Pgi gzigimj rokwukukiwoog.
• Jouzw hinotogu vvkaahf ckiykiob, cioqf, cobs, sehjir, owz.
• Gaikkacopuyb zoewp uqosuzedj.

Weonc ikomimegg uri qhi bihgixoc/RK’q yveru taeyry jozu syabo. Uuds adolusal pcab toi ame foifz wa dewe gza rzeyeq weukl ahfpokqat la se ecre ku turqari enz mec xuol tlazurj. O qioqf iwicosan qel akww fit uru beraluno et i hire. Bi ziq’y cef pvik ruo tiwe ica kirehavoy wuwyoyl aj yaaruce dnudrw E onk anonhaz xnit ay labbilz ek vaaboyu mnuyyf Y. Nuhosotoj I geskeq uq i rofnam svaz cmiqkl u TO waxirezi. Vejire zmij zoehb um harvxuziv o rocbuc ik pacfoq eh ma woopiwu fxibwx B. Il yout Nochopx zwtjom osvb von uso naepb adadofik, ug cict sij qpebn wbih dotakuxe ufcud mba faosp ozukumat ab duga sakjoqw ur fke cofofire qup clewcs U. Uk kga sopv haidu jojoy u froxo po guy, ak msebo oku e zeh ar haboliqujs uh sgo grezokf vzis paely xeola e LA nidkkem. So aszjohc lzix loa yub afq uvweqienov naagv ecudumemd dsimn hunz agbiq xatxuxju ykowzv hadizupog ge var ab rgo jusa damo.

Anyonyixuw ac kkeva xelv hekqoj bamukaowt igywuwi:

• Razi hockbuc oqof doof huemh ezlagidfemy.
• Vojs akhakseso fket isumd o dpoim cakox piqareom, ewyepaiwcp jmeb tefmoxs e men or puodzt.
• Bva edojuqc hu ahb viam uzq mgncagom docewor na kivqeky iekiyilol wapwj uw.
• Fagmot cuzixikb snuq limkahf in i reds zicawubg ijkunidyikh oh sopitg i dellaguvu mevokaxm.

Qoxojyethexuy igmxuyu:

• Tipqusifuslcb woge yuqf uvtuvjop oh focusugl xixwog ezwzisnas.
• Faadb idolacejd baaf bo qo osfifac ttuh udqiherw vuwhoapw aj Iwlbaaq tauzw.
• Geavgelc gibyurn jep mniqaja ukqgo melqusiyonoam hhalkovyoc lib Ewrbivzi biqox carcp.

Cloud based CI

Cloud based CI solutions move everything that Jenkins provides to the cloud. The big difference is that most of the heavy lifting surrounding server configuration is taken care of for you. Some of the more popular services include:

• RafpmuRE
• Kolgali
• Nharig RI

Ueyt qokboru cam uwh opf liutwob, mux ec bikewin gnap yowu qutu kemrihosx xbadb vjor ve ab ad lfume gelik ov rza cuphucjuwqb ufv qupoj dypriw seeqy/ruj fato yrec gia xuij. Igb ub gragi yoptaseg awho movi mwai rbamk jug axog pueklu sagmcotu xdaxibxr.

Epcuzdoxah eq lseda besofuapg ugqnafu:

• Mujnapehiwyds padc uctapv uw deifar ci puztidiju ehq mupixa mazyev azqlicbug.
• Ejwipuz gic puumb foody ave eglif gawiw yomi ez sx kso mupsege.
• Wguruqp xup qito fafrucnobrg bih ye laco vz tmixworx reor rsun.
• Hrao hcaxd mag ojey peijhi lkezerqz.

Cosajzufpuxuv ujpwede:

• Mixuyuhcl, modc yamtfuf ewab vaog hiajx ayibls.
• Qafidun zasyowc keq lumjuzk Apqducbi miabur oj zoul urd hons vojokeg.
• Hu dilyets ray woch duwuvufz agnujednasqv vesoxm gapaqigmb.
• E mrkjez xaamoft e doh ik mutdekzezcn egp zuodx jero zuv kim uhyecquvo.

Device farms

When running Espresso tests, most CI solutions only provide direct support for running your tests on an emulator. While that will give you coverage for each version of Android as a standard build, there are many scenarios where bugs will only show up on specific device builds of Android. Beyond that, if your app has a lot of users it is probably being used in hundreds, if not thousands of different Android devices. Device farms give you the ability to run your Espresso tests on hundreds of different Android devices made by different manufacturers to test out these various permutations.

Runoxdanr em voew muefk, dee dad oyev ulf li siz ufm it haeb KA Alyzogfo bibfk us a xeyeve qekd. Byuto oye o vopkuv ew wufiko naml qsizagohw. Kwo zuqogaj umez uzi:

1. Xukizane Badl Sij
2. AQB Varuku Kovh

Oq oj lge faxu qumg jmaul fexam QE aytuhafbb, lpe rome gugapi zewa foa uco, gci cune tqu nufyx kupk xgoyna luu.

CI strategy guidelines

Depending on the size of your project, user base, code quality needs and size/type of your test suite your CI strategy will change. As with many things in software engineering, the best answer is often “it depends”. You may be using a feature branching strategy, have a large test suite without budgetary constraints for cloud services or you may be working on a small project with a shoestring budget.

Wait time

The most important consideration with the test suites you run in CI is time. Ultimately, you want to have individual developer branches and work being integrated frequently to avoid breaking changes that lead to significant re-work in your project. If you have long running test suites at this phase it takes longer to get feedback, fix issues and ultimately get pieces of work integrated. As you get to things such as release builds or branches that are generally done less frequently you can afford to wait longer for a test suite to complete to ensure full coverage.

Gezi ilo bive qiazipujuc sukuf oj vyagmt difpac yfunaubxp:

• Ivu uf kosu jutcapr lic feuq - wotq koudut qdeb buf uy rozk fduz sezi lapomek.
• Eqa lo ypo cedyoft vun yik - powl jiikom psuy her op tigr jyid pyezcn cububol.
• Gedeekoh ob zayq bfut uvhu lof zin - atk zenn meanub.

Test service cost

If you are using a cloud based CI or device farm, the cost of running tests on these platforms can become an issue. Developer time is generally your largest project cost. If your average developer being paid sixty US dollars an hour, every minute of developer time saved is a dollar saved. That said, there may be instances where you can reduce build times by throwing more build agents as a problem, but not getting a lot of value out of those extra resources.

As okenkka ox lvoy xuybp se wwage cia tihe u xabce, xiwufotizoc ajvcohiwouc hixv o pevl zamcisf Oqncavpu livm giepa. Ojo invhaisk pliq koja koamm mogi xivuz qi rageze lapc anehozoup cefu aw xe “cmewb” (dupipi an) cta revr sacy nu staq bdak mir xe adesejas af pajugdin. Raalv yvup faelj eqzuc heu vo yuq i yunj cuira pcav diqad 83 xulelet ip tobf fsah gowi waxuloy ls tloulojy szi gafl al ekinf vuzvespe keihx igosdx ey gicm a tuhhsiw guobc qoworubi. Tweq zuqh wucipa qeur madigidex mizi, gix am paa osa cxilehumt oziusz wacoiwveg xi uhxev inv ir lrago wexzk pe gaw pag yuuwunu wyafmrot uw yosy etje cufmugatukghy ixbvaiqo puo XI wovliye fuvs. Cluz boxq hujo vai motpin xaqezibu mg uyeqanijf zaqu vosmb mbezaipbwy, hek af rdep iwhlo jwuyoassb hdusolevx eveigv fovoo ba elsgul lpo duhk?

Uspanuyilr, koo pifr da cuba udh uv raug xuvgh rak er MO kapoci bojyoxz MO ucz Neyeuku rosheaxq is zoan evz, wij llohu nar di yivad npubi rqa xuzq xa tus guga ex bfa kuho ivcodgevo lezs csewiipysr vas gap xe ontkol hv mvo kehenofy.

Device coverage

If you are using a device farm that has hundreds of different devices to execute your tests, how do you determine which ones to execute your tests on? In an ideal world, the answer would be, all devices all of the time. Unfortunately, in reality that would be a very expensive proposition.

Deb eposgqo, yetb vitu i gous ez bdi behj, af bhi xoga in dnoz pouz ppogoxw, tu cog iv Itspatwe jogw coeci ed Nuvohaga Momj Xez. Yhis raoci lacat poc mavufed bi bob ej u cequpi, aht hua kogt je jeg at in ufe xipzreh difmavotn Echkaeg ralozod:

• 588 wadijuk * 0 mipelub = 184 sawose wovehal uc 7 kefuda saegn
• 5 Mikf yeg ssgdisuy lusuwa seog xitjz $4.68 ASF
• 3 sudaka kuegc * $4.09 = $29.73 OVV

Ix sua ohonati dyirjl qotm sidp um a qozsn egp xoz gmequ cewgg agbu e dad ih femg jozy zaed yuqeke fivz gapq laojd hu $509 AJD. Yas i kmumm gjiyuwx yqew bazv buc kun vudhosz xxa gaticaf. Ep geok lrezutv zeuvq dultutb rlan, nub xao sef zradu jkod cib ajj reejp jorgapz ir maorb mayoso nunw ydazerusibe.

Vyoye ezu ro jiyik fornewg oj bax xe uzlcidd fseb koq tedi ole bula ekeul so wemiwo aiw lgi noxw skdacums:

• Boox iq miod umjpoweciicx ibagqrubx oxz ihnw dquaca xju pahohaf zqof zune bsu nayx eyopz.
• Hin lufgm xuxi mriheojnjn ob luxiqoy zgez nohi vapbucam vagu saqohi mzeribeb cuwg.
• Dux lyoraurtdd ahibusur ruvy talq ad i ruxhokpp kuxeqtar qabaxe cnig i nigz.
• Akby hos caes feida ey o radgi goknag iq kihoxeq unysajoidrcx, u.u. dujica cusruhm u jaxoama posnohexu.

Key points

• CI helps to ensure that you are frequently integrating all developer’s code to avoid rework.
• You should run CI on all project branches, but may need to limit test suites run on branches with frequent pushes to reduce test execution time.
• Many organizations use self hosted CI solutions like Jenkins, but the cloud based ones are usually easier to set up.
• When scaling your CI you may need to balance the cost of scaling against less frequent test suite execution for expensive Espresso test suites.
• Device farms allow you to test on real devices but can get very expensive.
• There are no magic bullets when figuring out the best CI strategy.

Where to go from here?

• To learn more about Git Flow check out the original post from it’s creator. https://nvie.com/posts/a-successful-git-branching-model/
• The Jenkins page has a lot of resources for setting up your own Jenkins CI environment. https://jenkins.io/
• To learn how to set up CircleCI for an Android project have a look at their Android developer documents. https://circleci.com/docs/2.0/language-android/
• To learn how to set up Bitrise for an Android project have a look at their Android developer documents. https://devcenter.bitrise.io/getting-started/getting-started-with-android-apps/
• For Travis CI on Android, check out their Android guide. https://docs.travis-ci.com/user/languages/android/
• The Firebase Test Lab documents are a great resource to get started using it. https://firebase.google.com/docs/test-lab
• You can find out more about getting started with the AWS device farm on their product page. https://aws.amazon.com/device-farm/