Revamping our iOS Continuous Integration System

We employ Continuous integration (CI) in our mobile development process. When someone commits code to one of our our project repositories this pipeline is triggered: 

1. Check out code onto a build machine 
2. Run static code analysis
3. Run unit tests
4. Run integration tests
5. Deploy to TestFlight for testing / submission to App Store

We‘ve been using Jenkins to manage the CI pipeline and Fastlane to perform code signing, testing and ad-hoc build processes.

This approach has served us well over the last few years and has ensured we have good end to end testing on our mobile products. 

But recently our CI has been less reliable. Builds and tests would randomly fail for various reasons – sometimes simulators would fail to start, or the build environment wouldn’t have the correct code signing certificates, increasing the amount of time spent diagnosing and fixing CI issues.

CI Alternatives

We looked into ways we could improve this and came up with the following options:

• Attempt to rebuild our Jenkins pipelines.
• Use Xcode Server and automatic signing.
• Use a 3rd party solution such as Bitrise.

We decided to try Xcode Server for several reasons:

1. We have dedicated hardware for CI so installing Xcode Server on our existing Mac mini was very straightforward.
2. We’d no longer need Fastlane. While it has proved useful over the years it’s suffered from bugs, and added complexity to the process.
3. Automatic code signing: Apple’s own solution, introduced a couple of years ago, supersedes that of Fastlane and is much more simple.
4. Keeping builds and tests local means we can easily integrate with our existing integration test API servers.
5. By not using a remote 3rd party service, we keep control of our codebase.

Xcode Server Setup

One of the main benefits from using Xcode Server is the automatic provisioning profiles and certificate management. For a long time we used Fastlane’s Match but even this became tedious and a constant source of problems when deploying. We hoped the signing features announced at WWDC 17 would allow us to hand off this responsibility to Xcode.

There are a lot of great blog posts on Xcode Server set up but we found this one to be the best: https://www.toptal.com/xcode/ios-continuous-integration-xcode-server. 

Due to a few issues, our setup experience wasn’t perfect.

Automatic Codesigning

Setting automatic signing up in Xcode is very straightforward and is enabling by checking the “Automatically manage signing” tickbox in the project options.

We found that despite this option being selected, archives created by Xcode Server weren’t being signed correctly with any provisioning profiles. This meant that we were unable to deploy our builds to TestFlight.

To fix this and to get Xcode to create and install the correct certificates and provisioning profiles we needed to open Xcode on our CI server and publish a build manually. The process is outlined in this tutorial.

Deploying manually once is enough to create all the correct certificates and profiles. Unfortunately this isn’t documented anywhere and isn’t quite as automatic as we’d been led to believe.

Deploying to the App Store / TestFlight

This is the final step in our pipeline and enables our builds to be delivered for testing or deploying. I imagine this is a standard step for most people using CI yet Xcode Server does not support this.

This means we had to build our app and deploy via a shell script using a similar method to the one in the tutorial linked above.

Pipelines

Our existing Jenkins pipeline ran every step in order. If a step fails the whole pipeline fails and we get notified. The original aim was to replicate this in one integration. Due to hardware limitations (an old Mac mini) we were still getting UI tests failing incorrectly and occasionally the simulator would crash to a grey screen. Restarting the simulator before each run of tests seemed to resolve this.

There’s no easy or efficient way to restart the simulator between test runs within Xcode, so to get around this we tried adding the UI tests as an xcodebuild command in a post run trigger. This works well but means we don’t get to leverage the great testing UI that’s part of Xcode Server.

The only other alternative was to add each set of UI tests (iPad and iPhone) as separate integrations and chain each integration if the last step was successful. There is no way to do this within Xcode or bot set up. However, you are able to trigger an integration via the Xcode Server API.

Using the following curl command in our post run triggers mean we are able to chain all of our integrations.

curl -H "Content-Type: application/json" -X POST -d '{}' https://server-address:20343/api/bots/BOTID/integrations --insecure

Summary

It took several days for us to transfer all of our build pipelines over from Jenkins to Xcode Server. At times it was pretty frustrating due to the lack of Apple’s documentation and Xcode Server not offering some standard CI features.

Things we like:

• Xcode integration. 
• Good feedback on what tests have failed.
• Good comparisons between integrations to see what has changed since last run.
• Much quicker and more reliable at running UI tests than through Jenkins.
• Faster build times – they’ve gone from around 30 minutes to 22 minutes to run our entire test suite and send a new version to TestFlight.
• No more manual code signing or using Fastlane.

Things we don’t like:

• Initial set up is slow and not very well documented.
• Logs aren’t immediately accessible through Xcode (they can be accessed via the Xcode Server web app).
• No plugin ecosystem to match Jenkins.
• Missing basic features like App Store deployment.

Overall this has been an improvement in our CI process and was worth the effort. We’re also hopeful of further improvements to Xcode Server from Apple due to their acquisition of BuddyBuild last year.