Summary
This post introduces the Android JUnit Report Test Runner, a custom instrumentation test runner for Android that produces XML test reports. Using this runner you can integrate your Android test results with tools that understand the Ant JUnit task XML format, e.g. the Pulse Continuous Integration Server.
The motivation and details of the runner are discussed below. For the impatient: simply head on over to the project home page on GitHub and check out the README.
Introduction
If you’ve been following my recent posts you’ll know that I’ve been figuring out the practical aspects of testing Android applications. And if you’ve been following for longer, you might know that my day job is development of the Pulse Continuous Integration Server. So it should come as no surprise that in my latest foray into the world of Android testing I sought to bring the two together :).
Status Quo
Out of the box, the Android SDK supports running functional tests on a device or emulator via instrumentation. Running within Eclipse, you get nice integrated feedback. Unfortunately, though, there are no real options for integrating with other tools such as continuous integration servers. Test output from the standard Ant builds is designed for human consumption, and lacks the level of detail I’d like to see in my build reports.
The Solution
On the upside, having access to the Android source makes it possible to examine how the current instrumentation works, and therefore how it can be customised. I found that the default InstrumentationTestRunner may be fairly easily extended to hook in extra test listeners. So I’ve implemented a custom JUnitReportTestRunner that does just that, with a listener that generates a test report in XML format. The format is designed to be largely compatible with the output of the Ant JUnit task’s XML formatter — the most widely supported format in the Java world. Tools like Pulse can read in this format to give rich test reporting.
How It Works
As mentioned, the JUnitReportTestRunner extends the default InstrumentationTestRunner, so it can act as a drop-in replacement. The custom runner acts identically to the default, with the added side-effect of producing an XML report.
For consistency with the SDK’s support for generating coverage reports, the XML report is generated in the file storage area of the target application. The default report location is something like:
/data/data/<tested application package>/files/junit-report.xml
on the device. To retrieve the report, you can use adb pull, typically as part of your scripted build.
Using the Runner
Full details on using the runner are provided in the README on the project home page. Briefly:
- Add the android-junit-report-<version>.jar to the libraries for your test application.
- Replace all occurrences of android.test.InstrumentationTestRunner with com.zutubi.android.junitreport.JUnitReportTestRunner:
- In the android:name attribute of the instrumentation tag in you test application’s AndroidManifest.xml.
- In the test.runner property in the Ant build for your test application (before calling the Android setup task).
- In the Instrumentation runner field of all Android JUnit Run Configurations in your Eclipse project.
- Add logic to your Ant build to run adb pull to retrieve the report after the tests are run.
As an example for retrieving the report in your Ant build:
<target name="fetch-test-report">
<echo>Downloading XML test report...</echo>
<mkdir dir="${reports.dir}"/>
<exec executable="${adb}" failonerror="true">
<arg line="${adb.device.arg}"/>
<arg value="pull" />
<arg value="/data/data/${tested.manifest.package}/files/junit-report.xml" />
<arg value="${reports.dir}/junit-report.xml" />
</exec>
</target>
In the Wild
You can see a complete example of this in action in my simple DroidScope Android application. The custom runner is applied in the droidscope-test application in the test/ subdirectory. You can even see the test results being picked up by Pulse on our demo server. Note that some of the tests are pure unit tests, which are run on a regular JVM, whereas others are run with the custom runner on an emulator. It’s nice for all the results to be collected together!