Using Firebase Test Lab to Improve the Quality of your Mobile Apps

A local setup of a testing infrastructure vs a setup on Firebase Test Lab

Firebase Test Lab provides cloud-based infrastructure for testing Android apps. With one operation, you can initiate testing of your app across a wide variety of devices and device configurations. Test results — including logs, videos, and screenshots — are made available in your project in the Firebase Console. Even if you haven't written any test code for your app, Firebase Test Lab can exercise your app automatically, looking for crashes.

In this codelab, you'll learn how to get started with using Firebase Test Lab to test your mobile apps. You'll run your tests from the web, Android studio, or using the "gcloud" command line interface. You can even integrate it to your Continuous Integration environment!

What you'll learn

Running Firebase Test Lab from the web, Android Studio and the command line
Ways to use Firebase Test Lab even when you don't have instrumentation tests
Integrating Firebase Test Lab with your Continuous Integration environment

What you'll need

Firebase project
gcloud SDK
Android Studio version 2.0+
The sample code

Upgrade your Firebase Project to the Blaze tier

The first step to use Firebase Test Lab is to navigate to your project page in the Firebase console, and upgrade your Firebase account to the Blaze tier. This allows you to access various Firebase capabilities including Firebase Test Lab. To upgrade your account to the Blaze tier, click on the upgrade button on the lower left of the navigation panel and select the Blaze plan.

Get the sample code

You can either download the sample code into your computer...

Download Zip

...or clone the GitHub repository from the command line.

$ git clone https://github.com/googlesamples/android-testing.git

Get your app and test APKs

Navigate to the BasicSample project main directory
In the BasicSample project main directory, run the following commands to build the app binaries and tests:

$ ./gradlew :app:assembleDebug
$ ./gradlew :app:assembleDebugAndroidTest

The binaries should now be available at:

$ ls app/build/outputs/apk/

The app binary is called: app-debug-unaligned.apk
The test binary is called: app-debug-androidTest-unaligned.apk

Install Android Studio 2.0+

Android Studio should be installed on your codelab machine. If you're following this codelab on your personal machine, make sure you download and install Android Studio 2.0+ from here.

Import the sample project

With the code downloaded, the following instructions describe how to open the completed sample app in Android Studio.

Select the BasicSample directory from the sample code folder (Quickstart > Import Project... > android-testing > ui > espresso > BasicSample).
Install any missing packages and resolve suggested conflicts
Click the Gradle sync button. Resolve all issues.
If you encounter any issues, follow the suggestions from Android Studio to install missing packages until you completely resolve the issue.

Run a Firebase Test Lab test from Android Studio

From the top menu, click Edit Configurations go to the Run/Debug configurations dialog to run a test

It opens the Run/Debug configurations menu. Add a test configuration and configure the test parameters as follows:

Under Deployment Target Options, choose Firebase Test Lab Device Matrix.
You can define a different Matrix configuration by clicking on the following icon: and creating a different configuration.
Pick your cloud project by clicking on the following icon:
Click OK to save your changes and close the window.
Now you can run the test by clicking the Run configuration icon in Android Studio:

Install gcloud command-line interface and enable Firebase Test Lab plugin

Now that we've used Firebase Test Lab from Android Studio and ensured that our code works during development, let's take it one step further and use it from the command-line, which is the first step towards a more formal Continuous integration with our deployment workflow.

First, install gcloud SDK.
Add Beta components to enable access to Firebase Test Lab from the Beta repository. Note: gcloud is a beta CLI, even though it is used to run tests in Firebase Test Lab

$ gcloud components install beta

Identify your project ID. The project ID should appear on the full URL to the Firebase console. https://firebase.corp.google.com/project/<your firebase project name>/overview.
Now configure the project for the codelab:

$ gcloud config set project <your firebase project name>

Make sure your authentication credentials are current.

$ gcloud auth login

Run a sample gcloud command to ensure the above steps finished correctly. You should see a table of devices which are available to test against in Firebase Test Lab. We'll come back to this table in a minute.

$ gcloud beta test android devices list

Run a Cloud Test from gcloud command-line interface

Now you're ready to run your first test. Try running the following from the BasicSample example's main folder we used earlier:

$ gcloud beta test android run \
  --type instrumentation \
  --app app/build/outputs/apk/app-debug-unaligned.apk \
  --test app/build/outputs/apk/app-debug-androidTest-unaligned.apk \
  --device-ids flounder,hammerhead \
  --os-version-ids 19,21 \
  --locales en,fr \
  --orientations portrait,landscape

Here we ran a test on Nexus 9 and Nexus 5 physical devices, running APIs 19 and 21 in English and French locales on portrait and landscape orientations. The gcloud command provided the URL to the test results page, where results appear as each test is completed, as well as a summary table showing how many tests passed or failed.
To run the test on different devices, let's first look at a list of available devices by running this command:

$ gcloud beta test android devices list

Now we can pick another device, e.g. Samsung Galaxy S5 (whose device-ID is "k3g") and run the same test on it as follows:

$ gcloud beta test android run \
  --type instrumentation \
  --app app/build/outputs/apk/app-debug-unaligned.apk \
  --test app/build/outputs/apk/app-debug-androidTest-unaligned.apk \
  --device-ids k3g \
  --os-version-ids 19 \
  --locales en,fr \
  --orientations portrait,landscape

Prerequisites

Download and install Jenkins CI.
Create and activate a service account: https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account
Enable Cloud Testing API and Tool Results API from the APIs page in the Google Developers Console:

Using a service account tells gcloud that the user is a robot. This avoids checks for spam and prevents having the account mistakenly blocked or prompted for captchas.

Add a build step to Jenkins

Now you can add all the necessary steps to Jenkins and watch a build start and finish automatically. It's out of scope for this codelab but take 5-10 minutes to try it out anytime. If you're not familiar with Jenkins, take a look at the documentation here.

Connect your Jenkins build to the github repository
Add a trigger on new pull requests
Add a build step to run the gradle build commands mentioned above
Add a build step for the "gcloud test" commands mentioned above
Start a new build and then watch to see if Jenkins marks the build as successfully tested, or if it finds a test failure.

You have now tried different uses for Firebase Test Lab and passed a major milestone towards achieving better quality for your app!

What we've covered

Introduction to Firebase Test Lab and why it's better to test your app on a variety of devices
Using Firebase Test Lab from the web, Android Studio, command-line and integration with Jenkins
Using Firebase Test Lab even when you don't have automated tests within your app

Next Steps

Start using Firebase Test Lab in your own applications by going to https://firebase.google.com/ to create a project.
View our full documentation at: https://firebase.google.com/docs/test-lab/
Join our users group and send us any questions or feedback: https://groups.google.com/forum/#!forum/google-cloud-test-lab-external
Start writing instrumentation tests for your application.

Tell us how we did