Instrumentation Testing

This document describes how to use the Instrumentation Framework to write test cases. Instrumentation testing allows you to verify a particular feature or behavior with an automated JUnit TestCase. You can launch activities and providers within an application, send key events, and make assertions about various UI elements.

You should have a working knowledge of the following:

• Android Application Framework
• Using adb, am and various logging functionality
• A brief understanding of the application of interest, that is, the names of the classes which handle the intents etc.
• JUnit testing.

Each Android application runs in its own process. Instrumentation kills the application process and restarts the process with Instrumentation. Instrumentation gives a handle to the application context used to poke around the application to validate test assertions, allowing you to write test cases to test applications at a much lower level than UI screen shot tests. Note that Instrumentation cannot catch UI bugs.

Instrumentation Framework

Classes

The following classes help glue together Instrumentation with JUnit testing.

Class	Description
InstrumentationTestCase	This extends the standard JUnit TestCase and offers access to an Instrumentation class. Write tests inside your instrumentation class any way you see fit. For example, your test might launch activities and send key events. For this to work properly, the instrumentation needs to be injected into the test case.
InstrumentationTestRunner	The instrumentation test runner is an instrumentation that runs instrumentation test cases and injects itself into each test case. Instrumentation test cases need to be grouped together with an instrumentation test runner with the appropriate target package.
InstrumentationTestSuite	The instrumentation test suite is a simple extension of the standard JUnit TestSuite that keeps a member Instrumentation variable on hand to inject into each TestCase before running them. It is used by InstrumentationTestRunner.

Three additional base classes extend InstrumentationTestCase to allow you to test Activity and Provider classes:

Class	Description
ActivityTestCase	This class can be used to write tests for a specific activity. An activity is launched in its setUp() method and finished with tearDown. If you write a test case that extends ActivityTestCase, you can write tests that access the activity using getActivity() and assume it has been set up properly.
ServiceTestCase	This test case provides a framework in which you can test Service classes in a controlled environment. It provides basic support for the lifecycle of a Service, and hooks by which you can inject various dependencies and control the environment in which your Service is tested.
SingleLaunchActivityTestCase	This class is similar to ActivityTestCase except that the activity is launched once per class instead of every time the test case calls setup.
ProviderTestCase	This class is similar to ActivityTestCase except that it will setup, tear down, and provide access to the Provider of your choice.

Understanding the am Command

The am command is a command-line interface to the ActivityManager (see http://code.google.com/android/reference/android/app/ActivityManager.html for details). am is used to start and instrument activities using the adb shell command, as shown in the snippet below:

> adb shell am
usage: am [start|instrument]
       am start [-a <ACTION>] [-d <DATA_URI>] [-t <MIME_TYPE>]
                [-c <CATEGORY> [-c <CATEGORY>] ...]
                [-e <EXTRA_KEY> <EXTRA_VALUE> [-e <EXTRA_KEY> <EXTRA_VALUE> ...]
                [-n <COMPONENT>] [-D] [<URI>]
       am instrument [-e <ARG_NAME> <ARG_VALUE>] [-p <PROF_FILE>]
                [-w] <COMPONENT>
For example, to start the Contacts application you can use
> adb shell am start -n com.google.android.contacts/.ContactsActivity

Platform Test Suites

This section provides an overview for various unit and functional test cases that can be executed through the instrumentation framework.

Framework Tests

Framework test cases test the Android application framework or specific Android application functionality that requires an Android runtime context. These tests can be found in //device/tests and //device/apps/AndroidTests.

Core Library

Core library test cases test the Android library functionality that does not require an Android runtime context. These tests are split into Android library (android.* package space) tests at //device/java/tests and Java library (java.*, javax.*, etc. packages) tests at //device/dalvik/libcore/.../tests.

Running Tests

Each instrumentation test case is similar to an Android application with the distinction that it starts another application. For example, have a look in the tests/Contacts directory.

• There should be a Makefile and an Android Manifest file.
• Tests are located in tests/Contacts/src/com/google/android/contactstests.
• The Instrumentation Test Runner is located at tests/Contacts/src/com/google/android/contactstests/functional/ContactsInstrumentationTestRunner.java.

Suppose you have a makefile with Contactstests as the target.

• make Contactstests: Compiles the test cases.
• adb install Contactstests.apk: Installs the apk on the device.
• Use the adb shell am command to run them.

To run your tests, use the am instrument command with your InstrumentationTestRunner as its argument. Results are printed as a result of the instrumentation. For example, the following snippet displays the output after running the framework tests with one test failing (note the unusual syntax caused by how instrumentations are run via am):

$ adb shell am instrument -w com.google.android.frameworktest/.tests.FrameworkInstrumentationTestRunner
INSTRUMENTATION_RESULT: test results:=.......F.......
Time: 6.837
There was 1 failure:
1) testSetUpConditions(com.google.android.frameworktest.tests.focus.RequestFocusTest)junit.framework.AssertionFailedError: requestFocus() should work from onCreate.
        at com.google.android.frameworktest.tests.focus.RequestFocusTest.testSetUpConditions(RequestFocusTest.java:66)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at android.test.InstrumentationTestSuite.runTest(InstrumentationTestSuite.java:73)
        at android.test.InstrumentationTestSuite.runTest(InstrumentationTestSuite.java:73)
        at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:151)
        at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1088)

FAILURES!!!
Tests run: 14,  Failures: 1,  Errors: 0

<RETURN> to continue

INSTRUMENTATION_CODE: -1
$ 

All Tests with Default TestRunner behavior

If no class or package is passed in to run, InstrumentationTestRunner will automatically find and run all tests under the package of the test application (as defined by the android:targetPackage attribute of the instrumentation defined in its manifest file).

 
$ adb shell am instrument -w \
  com.android.samples.tests/android.test.InstrumentationTestRunner
 
INSTRUMENTATION_RESULT: Test results for InstrumentationTestRunner=..........
Time: 2.317
 
OK (10 tests)
 
 
INSTRUMENTATION_CODE: -1

Running all Tests Under Single Package

If you have many tests under one package, use the -e package <packagename> option to run all tests under that package without having to manually create a test suite.

 
$ adb shell am instrument -w \
  -e package com.android.samples.view \
  com.android.samples.tests/android.test.InstrumentationTestRunner
INSTRUMENTATION_RESULT: Test results for InstrumentationTestRunner=........
Time: 1.587
 
OK (8 tests)

Running a Single Test Suite

If you prefer to explicitly state which tests comprise all of your tests, you can define a test suite and run that directly. By convention, all test packages in your system should have at least one suite called AllTests (see AllTests.java). To run all of the tests using the AllTests suite for the api demos test app:

 
$ adb shell am instrument -w \
  -e class com.android.samples.AllTests \
  com.android.samples.tests/android.test.InstrumentationTestRunner
 
INSTRUMENTATION_RESULT: Test results for AllTests=..........
Time: 2.286
 
OK (10 tests)
 
 
INSTRUMENTATION_CODE: -1

A Single Test Case

 
$ adb shell am instrument -w \
  -e class com.android.samples.view.Focus2ActivityTest \
  com.android.samples.tests/android.test.InstrumentationTestRunner
 
INSTRUMENTATION_RESULT: Test results for Focus2ActivityTest=....
Time: 1.359
 
OK (4 tests)
 
 
INSTRUMENTATION_CODE: -1

A Single Test

 
$ adb shell am instrument -w \
  -e class com.android.samples.view.Focus2ActivityTest#testGoingLeftFromRightButtonGoesToCenter \
  com.android.samples.tests/android.test.InstrumentationTestRunner
 
INSTRUMENTATION_RESULT: Test results for Focus2ActivityTest=.
Time: 0.51
 
OK (1 test)
 
 
INSTRUMENTATION_CODE: -1

Attaching a debugger to your test

In order to debug your test code, instruct the controller to stop and wait for the debugger by adding -e debug true to your command line. This causes the test runner to stop and wait for the debugger just before calling your setUp() method. For example,

 
$ adb shell am instrument -w \
  -e debug true \
  com.android.samples.tests/android.test.InstrumentationTestRunner

Writing Tests

When writing tests, refer to the ApiDemos tests as models (located at //device/samples/ApiDemos). This section provides an overview of the test structure with ApiDemos.

Location of Files

Test packages should use the following structure and include Android.mk, AndroidManifest.xml, AllTests.java, and a src directory that mirrors the src directory of the tested application.

Files are located within a tests directory found in the root directory:

 
$ find samples/ApiDemos/tests
samples/ApiDemos/tests
samples/ApiDemos/tests/Android.mk
samples/ApiDemos/tests/AndroidManifest.xml
samples/ApiDemos/tests/src
samples/ApiDemos/tests/src/com
samples/ApiDemos/tests/src/com/google
samples/ApiDemos/tests/src/com/google/android
samples/ApiDemos/tests/src/com/google/android/samples
samples/ApiDemos/tests/src/com/google/android/samples/AllTests.java
samples/ApiDemos/tests/src/com/google/android/samples/ApiDemosTest.java
samples/ApiDemos/tests/src/com/google/android/samples/os
samples/ApiDemos/tests/src/com/google/android/samples/os/MorseCodeConverterTest.java
samples/ApiDemos/tests/src/com/google/android/samples/view
samples/ApiDemos/tests/src/com/google/android/samples/view/Focus2ActivityTest.java
samples/ApiDemos/tests/src/com/google/android/samples/view/Focus2AndroidTest.java

Contents of makefile

The contents of the makefile are similar to a normal application with the addition of a LOCAL_INSTRUMENTATION_FOR declaration.

 
# Add appropriate copyright banner here
LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)
 
# We only want this apk build for tests.
LOCAL_MODULE_TAGS := tests
 
# Include all test java files.
LOCAL_SRC_FILES := $(call all-java-files-under, src)
 
# Notice that we don't have to include the src files of ApiDemos because, by
# running the tests using an instrumentation targeting ApiDemos, we
# automatically get all of its classes loaded into our environment.
 
LOCAL_PACKAGE_NAME := ApiDemosTests
 
LOCAL_INSTRUMENTATION_FOR := ApiDemos
 
include $(BUILD_PACKAGE)

Content of Manifest

Use the following example to create an AndroidManifest.xml file that declares the instrumentation. Specify that the framework supplied Instrumentation TestRunner targest the package of your application, allowing the tests that are run with the instrumentation to get access to all of the classes of your application without having to build the source into the test app. The name of the test application is typically the same as your target application with .tests appended.

 
# Add appropriate copyright banner here
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.android.samples.tests">
 
    <uses-permission android:name="android.permission.RUN_INSTRUMENTATION" />
 
    <!--
    This declares that this app uses the instrumentation test runner targeting
    the package of com.android.samples.  To run the tests use the command:
    "adb shell am instrument -w com.android.samples.tests/android.test.InstrumentationTestRunner"
    -->
    <instrumentation android:name="android.test.InstrumentationTestRunner"
                     android:targetPackage="com.android.samples"
                     android:label="Tests for Api Demos."/>
 
</manifest>

 

The following snippet will prefix the /android.test.InstrumentationTestRunner when running tests from the command line:

 
$ adb shell am instrument -w \
  com.android.samples.tests/android.test.InstrumentationTestRunner

New Instrumentation TestRunner

Create a class that derives from this class. You must override two abstract methods; one that returns the class loader of the target package, and another that defines all of the tests within the package. For example, the snippet below displays the test runner for the framework tests.

public class FrameworkInstrumentationTestRunner extends InstrumentationTestRunner {

    @Override
    public TestSuite getAllTests() {
        InstrumentationTestSuite suite = new InstrumentationTestSuite(this);

        suite.addTestSuite(FocusAfterRemovalTest.class);
        suite.addTestSuite(RequestFocusTest.class);
        suite.addTestSuite(RequestRectangleVisibleTest.class);
        return suite;
    }

    @Override
    public ClassLoader getLoader() {
        return FrameworkInstrumentationTestRunner.class.getClassLoader();
    }
}

Next, in an appropriate AndroidManifest.xml, define the instrumentation for the derived class with the appropriate android:targetPackage set. For example, the snippet below defines the instrumentation runner for the framework tests.

<uses-permission android:name="android.permission.RUN_INSTRUMENTATION" />

<instrumentation android:name="android.tests.FrameworkInstrumentationTestRunner"
                 android:targetPackage="com.google.android.frameworktest"
                 android:label="framework instrumentation test runner" />

New InstrumentationTestCase

To create a new test case, write a class that extends InstrumentationTestCase in the same application as your test runner. The following snippet illustrates an example ActivityTestCase that tests an activity named MyActivity.

public class ButtonPressTest extends ActivityTestCase<MyActivity> {

    Button mLeftButton;

    public ButtonPressTest() {
        super("com.example", MyActivity.class);
    }

    @Override
    public void setUp() throws Exception {
      super.setUp();
      mLeftButton = (Button) getActivity().findViewById(R.id.leftButton);
    }

    public void testFocusMovesToRight() throws Exception {
        assertTrue(mLeftButton.hasFocus());
        getInstrumentation().sendCharacterSync(KeyEvent.KEYCODE_DPAD_RIGHT);

        Button rightButton = (Button) getActivity().findViewById(R.id.rightButton);
        assertTrue(rightButton.hasFocus());
    }

    // could have several more tests...
}

Exploring a Test Case

The test case described in this section adds and tests a new Contact. Note that you can send intents, register intent receivers, etc.

Instrumentation.java has helper functions that send key events and strings, for example:

• getInstrumentation(): Returns the handle to the instrumentation
• sendCharacterSync: Sends a character.
• sendStringSync: Sends a string to an input box.
• sendKeyDownUpSync: Sends a specific keyevent.
• sendTrackballEventSync: Sends a trackball event.

You can find the test case below at device/tests/Contacts.

private void addNewContact(String name, int star, int phoneType, String number, String label,
		String email, int emailType){
	ContentValues values = new ContentValues();
	Uri phoneUri = null;
	Uri emailUri = null;

	values.put(Contacts.People.NAME, name);
	values.put(Contacts.People.STARRED, star);

	//Add Phone Numbers
	Uri uri = mActivity.getContentResolver().insert(Contacts.People.CONTENT_URI, values);
	phoneUri = Uri.withAppendedPath(uri, Contacts.People.Phones.CONTENT_DIRECTORY);

	values.clear();
	values.put(Contacts.Phones.TYPE, phoneType);
	values.put(Contacts.Phones.NUMBER, number);
	values.put(Contacts.Phones.LABEL, label);
	mActivity.getContentResolver().insert(phoneUri, values);

	//Add Email
	emailUri = Uri.withAppendedPath(uri, ContactMethods.CONTENT_DIRECTORY);

	values.clear();
	values.put(ContactMethods.KIND, Contacts.KIND_EMAIL);
	values.put(ContactMethods.DATA, email);
	values.put(ContactMethods.LABEL, "");
	values.put(ContactMethods.TYPE, emailType);
	mActivity.getContentResolver().insert(emailUri, values);
}


 public void testAddSaveSingleContact(){
	int previousCount = mActivity.getListView().getCount();
	String message;

	addNewContact(INPUT_NAME_1 + "1", "5435754532", "1" + INPUT_EMAIL_1, CONFIRM_OPTION);

	message = "Added 1 to initial length=" + previousCount + ", but resulted with a count=" +
		mActivity.getListView().getCount();
	assertEquals(message, ++previousCount, mActivity.getListView().getCount());

	// Check Content; Name; Num; Starred
	assertEquals(INPUT_NAME_1 + "1", getTextFromView(0, android.R.id.text1));
	assertEquals("5435754532", getTextFromView(0, android.R.id.text2));

	//Check email is saved
	//cursor = returnEmailCursorAtId("1");
	Uri uri = Uri.parse("content://contacts/people/1");
	uri = Uri.withAppendedPath(uri, ContactMethods.CONTENT_DIRECTORY);
	Cursor cursor = mActivity.getContentResolver().query(uri, CONTACTS_COLUMNS, null, null, null);
	assertTrue("returnEmailCursorAtId: Moving cursor to first row has failed", cursor.first());

	int dataIndex = cursor.getColumnIndexOrThrow("data");
	assertEquals("1" + INPUT_EMAIL_1, cursor.getString(dataIndex));
	cursor.deactivate();
}
	

Deciding Kinds of Tests to Write

Once you are bootstrapped with your test application, you can start writing tests. There are three of types of tests you may wish to write:

• TestCase: The standard junit test case.
• AndroidTestCase: A test case with access to a Context object that is injected for you by the instrumentation test runner.
• InstrumentationTestCase: A test case with access to an Instrumentation, which can be used to launch activities, content providers, send key events, etc.

The API Demos test suite includes examples of all three styles and can be used as a guideline for writing each type of test.

There are two utility classes available for the most common uses of InstrumentationTestCase: ActivityTestCase and ProviderTestCase. See their javadoc for more information.

Troubleshooting

If you run your test cases and nothing appears to happen, have a look at adb logcat. The following is a common problem:

I/dalvikvm(  688): threadid=11: attached from native, name=Binder Thread #1
I/dalvikvm(  688): threadid=13: attached from native, name=Binder Thread #2
W/ActivityManager(  469): Unable to find instrumentation info for: ComponentInfo{com.google.android.browser_instrumentation/com.google.android.browser_instrumentation.BrowserWebkitLayoutInstrumentation}
D/AndroidRuntime(  688): Shutting down VM
E/AndroidRuntime(  688): ERROR: thread attach failed

It's possible that the instrumentation apk isn't installed on your device or that the package name is incorrect in the Manifest file.