Using a CI Server is a programming practice that is well established and not opened to debate anymore. Sometimes, it's even a topic on which the IT Department has regained control of, managing and rationalizing the servers. Yet, as is often the case, mobile is following its own way: the technologies used can be considered non-standard among the company, the ecosystem is updated way more frequently than in other computing areas, and the need of running on a specific OS for iOS can be the fatal blow that leaves mobile developers on their own. Therefore, they frequently end up installing a mac mini in the open-space, in order to run their builds on it.
Whereas the problems related to using such installations are mostly shared between Android and iOS, the solutions differ: this article focuses on Android.
When he sets its Jenkins Server up, the mobile developer will have to deal with the following issues:
The mobile developer is not a devops! If he's lucky enough, he will manage to deal with these issues and end up with a fair install. Occasionally, a weakness of this setup will arise and mess with the team, but overall, it'll be livable.
Regardless the configuration and quality of the installation, a serious limitation will be present on this in-house CI Server: the need to have an Android simulator, emulator or device connected to run automated tests.
Why this need for an emulator? Android does not run on a standard JVM, but on Dalvik (or more recently, ART). Before a relatively recent update (and still incomplete to this day), tests written by the developers had to run on a Dalvik/ART VM, thus on an emulator or a device. (Note 1)
Using a device is hard to industrialize: it has to continuously stay on and plugged in. Data transfer through the USB cable is slow, but more important, the computer sometimes loses its connection with the device, requiring someone to physically disconnect and reconnect it to the machine. Unthinkable for a CI Server. The Android emulator is, therefore, the only viable solution. As it is quite slow to boot (it can take up to several minutes), developers often let an emulator run continuously on the server. If this allows a saving of a few minutes on each build, it also brings a few limitations:
For the last few years, we have made it a habit to move to the cloud all the vital services we use, but find painful to maintain. Does the cloud have anything to offer from an "Android CI Server" perspective?
Mobile specialized CI solutions can be found, alongside more standard platforms supporting Android. The following table was assembled by running the following project (available on github) on different platforms. This project contains a non-instrumented (JVM) test, and an instrumented espresso test, checking that a TextView contains the text "Hello world!".
On the following table, the "Git" row tells whether the platform can read from any git repository accessible using SSH/HTTPS, or if it's tied to Github. "Job count" states the number of different jobs it is possible to define then launch separately. "Distinction CI / Deploy" indicates if it is possible to differentiate CI jobs from jobs that go as far as deploying the APK. "Emulator configuration" tells the different Android emulator versions available, if the configuration has to be done manually (with code) or through a web interface. "Deploy" lists the plugins available from a web interface to deploy the app. Manually indicates that deploying is possible, but needs a gradle task or a shell script.
Note: Configuring the emulator on SnapCI must be done manually through code. I could not manage to start an emulator within a one-hour delay. The interface test could therefore not be run on this platform.
The first striking point is the lack of maturity from the different players, even if they all advertise Android support as being one of their feature.
GreenHouseCI and Travis might be sufficient on "basic" projects. The main limitation comes from the inability to differentiate a CI job from a delivery job. It is also impossible to follow any code quality metrics on those platforms.
CloudBees, which simply offers a Jenkins instance "as a service", is the only platform that seems mature enough today to host a professional project.
To differentiate the players regarding the price, the example project ran before is not relevant anymore. To assemble the following table, I considered an existing project, of about 140k LOC, on which every build takes 15 minutes, and is run 10 times per working day. The plans I selected are the cheapest ones allowing at least 5 users, and the execution of a minimum of 2 builds in parallel.
Note: Prices are in USD.
Every player but CloudBees offers fixed-price plans, depending on the user number and the number of builds that can be run in parallel.
On CloudBees, the monthly plan starts at $60. You then have to add $1,32 per build hour. The price thus varies with the load. With the same parameters as told previously, the total monthly price would drop to $104 with 10 minutes builds, and climb to $148 with 20 minutes builds.
It is not really a surprise to find that Jenkins, the reference solution regarding continuous integration, is still the most powerful option when hosted on the cloud. In its "Dev@Cloud" offer, CloudBees brings you a hosted Jenkins master instance. It will launch each of your builds on a dedicated virtual machine instantiated specifically for this occasion. As you are probably already using a Jenkins server today, migrating your jobs and plugins is easy.
It's important to ensure that every instance booted to run one of your build has an up-to-date Android SDK installed. The use of Jake Wharton's "sdk-manager" gradle-plugin is thus mandatory. The plugin will check that the sdk, build tools, support repositories and google libraries are up to date before letting you compile your project.
All the "OPS" issues that we enumerated in the first half of this article disappear on CloudBees. The master Jenkins never crashes (unless, of course, the whole platform experiences downtimes). Everything is siloed: each build leads to a slave instance being booted. This instance can be as fast as you want. Finally, you can choose, for each and every job, on which Android version you want the emulator to run to play your test suite.
This last bullet-point is a strength and a weakness at the same time.
Each project can now run its instrumented and interface tests on the lowest Android version you support on your app. But you can also choose to run those tests on a configuration matrix mixing for example the Android version, the device's locale, and the screen size.
Playing with the emulator versions leads to reaching one of the limitations of CloudBees: the difficulty to boot emulators running on the latests Android versions. Official Android emulators are known to be "heavy", slow to boot and laggy. These flaws worsen as you increase the Android version. Additionally, it is impossible to benefit from hardware acceleration (whether GPU or HAXM) or from x86/64 versions on CloudBees. It is thus very difficult to reliably boot an emulator starting from version 4.4.4: the success rate is lower that 25%. (Note 2)
This limitation is frankly annoying, but is mitigated by various factors:
On my previous Android project (140k lines of code, 3 years of development), the migration from an on-premise Jenkins server to CloudBees halved the build time of the CI job, and split by 4 the build time for the delivery job. Despite a few troubles discussed before with emulators during the migration, it has clearly been beneficial to the project.
The caveats around emulators should only slow down projects that are currently in production, supporting "very recent" Android versions (such as KitKat) as their minimum, and relying on large instrumented tests harnesses. I hope that the rise of better Android emulators or more mature solutions to access emulators and devices "as-a-service" should allow me to reconsider this limit soon.
Notes :