iOS development projects are not first-in-class when it comes to managing the quality of the software produced.
Very short projects, very short time-to-market, it is not the kind of projects where you see a lot of attention towards quality, unfortunately.
Here at OCTO we try to do it differently, even for this kind of projects. Or above all, for that matter. But here comes another issue: the lack of tooling.
Here is one of our latest attempt to setup quality metrics on a short and budget constrained iOS project. And yes, a teaser, look at our dashboard at the end of this six weeks (six Agile iterations) project:
The following article will detail how to setup all these quality metrics in a integrated report in Jenkins (Continuous Integration).
Here is the breakdown of this article:
When it comes to tooling, Objective-C is a mixed case because it is at the same time a proprietary platform and a language derived from C. In practice, here is the current situation on quality tools for Objective-C:
Let's look at how to setup all those metrics.
Note: If you get lost on the way, do not hesitate to check the end of the article, I have put links to sample configuration files
We will not detail this point as there are a lot of posts around on doing this. For example you can go with this one in case Jenkins is not already installed at your place. I recommend using the same user as your CI user because otherwise you will run into issues with Keychain certificates (the certificates you will install on your server will not be available to the Jenkins user).
As above, I refer you to this article if it is something new for you. I also recommend using the Xcode plugin for Jenkins.
Let's dive in interesting things. Setting up code metrics and code duplication is quite easy, so let's start with it.
I decided to go for SLOCCount as it already provides a Jenkins plugin. The documentation for it is here.
I followed the instructions on both the SLOCCount page and Jenkins plugin page, that is basically:
sloccount --duplicates --wide --details YOUR-PROJECT-ROOT-DIRECTORY | grep -v -e '.*ExternalFrameworks.*' | grep -v 'top_dir' > build/sloccount.sc This is how it looks in my Jenkins job configuration: 
Once this done, relaunch your build. You might have to modify your Jenkins configuration to have /usr/local/bin in Jenkins path (unfortunately Jenkins do not use your system path). I fixed this by setting the PATH variable in Jenkins configuration with the content of my path ('echo $PATH'). You can do this in the 'Global properties' section of the configuration (not in your job configuration but in Jenkins configuration):
If you go back on your dashboard, you should see the trend graph on it and have also access to the full report through the left menu. Actually, depending on the plugins, the trend graph does not appear from the start but once it has enough data to display (typically after 3 or 5 builds).
The detailed report gives you the details of LOC by file and by folder. You can do a first sanity check here to know if no file has been forgotten or is present by mistake. Here is an example of the report by folder:
The tool chosen here is PMD because it is free to use. In fact the Objective-C support is more 'beta' than it is in Simian, so you might choose the later.
Objective-C support in PMD is made possible thanks to Joshua Kennedy. Announcements and details on how to setup it are posted on his blog so I refer you to his post.
Basically you have to:
java -Xmx512m -classpath /Users/udd/PMD/pmd-4.2.5/lib/pmd-4.2.5.jar:/Users/udd/PMD/ObjCLanguage-0.0.6-SNAPSHOT.jar net.sourceforge.pmd.cpd.CPD --minimum-tokens 100 --language ObjectiveC --encoding UTF-8 --format net.sourceforge.pmd.cpd.XMLRenderer --files YOUR-PROJECT-ROOT-DIRECTORY --files YOUR-TEST-ROOT-DIRECTORY > build/cpd-output.xml This is how it looks in my Jenkins job configuration: 
Just one comment: I had trouble to make it work and always get a blank report file first (so as few other if you read the comments).
<?xml version="1.0" encoding="MacRoman"?>
<pmd-cpd>
</pmd-cpd>
If your are in this case, here is the hint: you have to specify a full path for the JARs in the classpath, for some reason I ignore.
Well, if things went right, you should now have on your dashboard a nice trend graph like the following:
and you can access from the left menu the detailed report, including the report by file which is quite interesting:
and if you drill down to the duplication, the detail of each duplication per file:
Nice, isn't it?
Let's step to the harder part, getting test results and code coverage metrics in Jenkins. Getting test results is pretty easy thanks to the Xcode Jenkins plugin though, because the plugin will take care to generate the test results XML file in the expected JUnit format.
To achieve it, you need to, basically:
Once done, if you run your build again, you will get a new trend graph on your dashboard:
And you will be able to drill down to the individual tests or failures, in case there are some.
Some of you might be using GHUnit instead of OCUnit (Xcode default).
Here is a summary of the differences:
GHUNIT_CLI=1 WRITE_JUNIT_XML=YES xcodebuild -target YOUR-TEST-TARGET-NAME -configuration Debug -sdk iphonesimulator clean buildThis is how it looks in my Jenkins job configuration: 
Take a breath now (or a coffee), this is the harder part. There is a few good posts around on how to compute code coverage for iOS, but not always up-to-date The way to do it has changed drastically with the different versions of Xcode. The following has been tested with Xcode 4.3.3.
To summarize, you should:
enable the two build settings 'Generate Test Coverage Files' and 'Instrument Program Flow' in the test target of your project in Xcode
enable the two build settings 'Generate Test Coverage Files' and 'Instrument Program Flow' in the main target of your project in Xcode, but only for Debug 
add code for missing fopen$UNIX2003 and fwrite$UNIX2003 functions. This is more an Xcode bug. I added this code in my AppDelegate.h:
#ifdef DEBUG
FILE *fopen$UNIX2003(const char *filename, const char *mode);
size_t fwrite$UNIX2003(const void *ptr, size_t size, size_t nitems, FILE *stream);
#endif
and this one in my AppDelegate.m:
#ifdef DEBUG
FILE *fopen$UNIX2003(const char *filename, const char *mode) {
return fopen(filename, mode);
}
size_t fwrite$UNIX2003(const void *ptr, size_t size, size_t nitems, FILE *stream) {
return fwrite(ptr, size, nitems, stream);
}
#endif
Once done, I recommend to check everything is working locally first. For that you should run your tests and go to: ~/Library/Developer/Xcode/DerivedData/YOUR-PROJECT-NAME/Build/Intermediates/YOUR-TEST-TARGET-NAME.build/Objects-normal/i386' and ~/Library/Developer/Xcode/DerivedData/YOUR-PROJECT-NAME/Build/Intermediates/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386'Both locations should contain several .gcno files if the project is properly setup and some .gcda files if your tests did generate coverage data. You can check that the coverage is correct by opening one of the files with CoverStory, a GUI for analyzing the coverage data.
Once your project properly set up, you have to configure Jenkins to generate this report automatically. This is explained in this blog post.
Basically you need to:
scripts/gcovr -r . --object-directory build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386 --exclude '.*Tests.*' --exclude '.*ExternalFrameworks.*' --xml > build/coverage.xmlThis is how it looks in my Jenkins job configuration: 
Launch your Jenkins job! It works? Congratulations, but don't worry if it doesn't, this part is a bit a makeshift job, it might need some trial-and-error.
Once done, you will get a new trend graph on your dashboard as below:
and you will be able to drill down to see the coverage of all your different files:
and event of the specific lines of your source code:
Satisfied? If you try this step, you will notice that you don't really get the same dashboard as the one shown above, right? In fact there is plenty to say about getting an useful code coverage measurement, and as this article was already long enough, I have detailed this in another blog article with all the tips and tricks that lead me here.
For adding code coverage, the main differences with OCUnit (Xcode standard) are:
add the code for fopen$UNIX2003 and fwrite$UNIX2003 functions in the main.m file of your test target, not in the AppDelegate. This is how my main.m file looks like:
#import <UIKit/UIKit.h>
int main(int argc, char *argv[])
{
@autoreleasepool {
return UIApplicationMain(argc, argv, nil, @"GHUnitIOSAppDelegate");
}
}
FILE *fopen$UNIX2003(const char *filename, const char *mode) {
return fopen(filename, mode);
}
size_t fwrite$UNIX2003(const void *ptr, size_t size, size_t nitems, FILE *stream) {
return fwrite(ptr, size, nitems, stream);
}
edit your test target .plist and add the property 'Application does not run in background' with value 'YES'. This is depicted below. Notice that this is useful because the coverage files are only written when the application exits.
add a Shell Build Step in your Jenkins job configuration between the launch of your unit tests and the launch of the GCOVR script, with the following content: cp -n build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386/*.gcno build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-TEST-TARGET-NAME.build/Objects-normal/i386 || true This is how it looks in my Jenkins job configuration: 
Congratulations!
But don't forget that all of this is only useful if you analyze it and track it regularly. This is what the trends are made for, and this is how I use these metrics:
I manually mark important builds as 'Keep forever' in Jenkins and I enabled 'Discard Old Builds' in my Jenkins job configuration. In practice I only keep builds delivered at the end of each Agile iteration, to have a good vision of the coarse-grained variations (trends) 
This is why all the graphs shown in this article looks so nice and clearly outline trends.
I review them at the end of each iteration to add actions to the backlog of the next iteration. Example of such actions:
Honestly this is still a bit long/clumsy to setup, I think for a first installation it will take you about 2-4h. A lot more if you also need to install Jenkins and don't know Jenkins yet.
The good news is that once done for a first project, it's about 30 min of work for all the future mobile projects you will work on. I think this is acceptable even for very short projects.
I have an another good news for those that gave up because of time/pitfalls, we are currently working on a Sonar plugin for Objective-C. This will make all this easier and will bring new metrics. Stay tuned, we will announce it here as soon as we release the first useful version.
Last thing: I have put here the full configuration of my Jenkins job in case you want to compare with yours. I have also put a sample configuration for test and coverage for OCUnit and the same sample configuration for GHUnit, because this is the hardest part and I thought it deserves it.
And yes, I welcome your feedback if I forgot some steps or you think I should clarify a particular step.
