quality metrics for iOS, I wasn't prepared to spend that much time to get something robust and correct.
The part on which I stumbled most was the code coverage, not because it's that difficult to make it work (there is plenty of resources on the Web) but because in all articles I have seen the solution was working but was not reporting accurate and useful metrics (I am sure I have missed some, sorry for this if this is the case).
Note: the fact that Xcode support for it is very fragile and has changed with almost each version of Xcode did not ease this and explain why people were first focused on making it work.
Here are some of the pitfalls I saw in all the articles talking about code coverage:
The setup I proposed in the preceding article is still valid and will avoid you these pitfalls. As the article was already long enough, I decided not to make it longer and keep all the detailed explanations for a new article. Here it is.
Let's see how to tackle these issues one by one. If you tried to setup the code coverage using my last article, it will make all the steps a bit more logic.
The two most popular blog posts on how to get code coverage for the iPhone can be found here and here. Follow them and you will only get the coverage for the files that are actually tested by unit tests.
OK you might know that all other classes are not tested, but how much does it weight? And does other people in your team know? What if your "standard" says that this part of the application should be covered (for example an utility class)?
Why does this happens? This is because the recommended approach is to enable the build settings 'Generate Test Coverage Files' and 'Instrument Program Flow' only on the test target, which means that only the tested files and the tests themselves will have .gcno files (code to cover) and .gcda files (code covered) generated during compilation (for .gcno files) and execution (for .gcda files).
This is why in my setup I also enable the two build settings 'Generate Test Coverage Files' and 'Instrument Program Flow' on the main target. Notice that I only enable them in Debug to avoid having issues with production code.
Is that enough? No, the second difference is to make GCOVR operate on the main bundle and not the test bundle. This is because the test bundle only contains the .gcno/.gcda files of the tests while the main bundle contains both. This is what the --object-directory build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386
option in GCOVR command line is doing.
Note: this is only valid for OCUnit. There are other differences for GHUnit but I cover them later (Pitfall #4)
You can check that everything is working locally first by going to ~/Library/Developer/Xcode/DerivedData/YOUR-PROJECT-NAME/Build/Intermediates/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386'
. It should contain (once you have run your tests) a lot of .gcno and .gcda files. Check that there is a .gcno but no .gcda file for files you know not being tested.
Once done, you will see that your coverage metric in Jenkins will decrease a lot and will show you what your coverage really is!
Now that you cover all your code, you will get a new problem: it will also report coverage for code that is not yours! That's not desirable because this is the kind of noise that make a metric false and quickly abandoned.
This is resolved in two steps:
--exclude '.*ExternalFrameworks.*'
flag to the GCOVR command lineNote: you should also exclude your unit tests themselves with --exclude '.*Tests.*'
but this part was often already covered in the mentioned articles
Once done, your coverage metric in Jenkins should increase significantly as I haven't seen much tests in the iOS frameworks I am using!
The latest pitfall I have seen is that all these reports are flat and give you only a project and a file-by-file view of the coverage. No intermediary view. Very often we structure your application in layers and Apple MVC pattern encourages you to do so. Wouldn't it be great to know the coverage of the different layers? This is particularly true because you don't test all the layers with the same kind of tests and you don't put the same test efforts on the different layers.
My best practice is to structure your project in directories that are both meaningful for day-to-day work and for the code coverage analysis.
This is an example of a typical structure I use:
Notice that you should use real folders (not Xcode logical ones) if you want this to have an impact on Cobertura report.
This is how I get the following report already shown in the preceding post:
In this way I can quickly check that sensitive application layers like domain (Model directory here), services and utilities (Utils here) are correctly tested.
As a rule of thumb, here is what I am targeting today in my applications for unit tests code coverage:
Unfortunately, it's not possible yet to specify specific thresholds by package in Cobertura Jenkins plugin (vote for the open issue in JIRA here!). Someone has posted a solution for this on stackoverflow.com, but I have not tested it yet.
Of course once you start adding UI tests, the coverage of the Controllers/Views should increase significantly. I have not tried yet to include these tests in the coverage report, but I hope to be able soon.
Once done, it should become clear which part of the application you should test next.
Last, but not least: we all have our preferred tools to work. There are strong advisers of OCUnit or GHUnit, and very often not both are covered when it deals to getting test results or coverage.
I won't go back to the explanations already given in my preceding article, but will focus on what is structurally different between the two:
cp -n build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386/*.gcno build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-TEST-TARGET-NAME.build/Objects-normal/i386 || true
is made for and it is important to do this before the GCOVR command line for it to be useful.The unintelligible steps detailed in the preceding article should become logic now, do they?
I hope that this will help you make the most of your improved or new coverage metrics!