A local setup of a testing infrastructure vs a setup on Firebase Test Lab
Firebase Test Lab provides cloud-based infrastructure for testing Android apps. With one operation, you can initiate testing of your app across a wide variety of devices and device configurations. Test results — including logs, videos, and screenshots — are made available in your project in the Firebase Console. Even if you haven't written any test code for your app, Firebase Test Lab can exercise your app automatically, looking for crashes.
In this codelab, you'll learn how to get started with using Firebase Test Lab to test your mobile apps. You'll run your tests from the web, Android studio, or using the "gcloud" command line interface. You can even integrate it to your Continuous Integration environment!
What you'll learn
Running Firebase Test Lab from the web, Android Studio and the command line
Ways to use Firebase Test Lab even when you don't have instrumentation tests
Integrating Firebase Test Lab with your Continuous Integration environment
The first step to use Firebase Test Lab is to navigate to your project page in the Firebase console, and upgrade your Firebase account to the Blaze tier. This allows you to access various Firebase capabilities including Firebase Test Lab. To upgrade your account to the Blaze tier, click on the upgrade button on the lower left of the navigation panel and select the Blaze plan.
Get the sample code
You can either download the sample code into your computer...
The test binary is called: app-debug-androidTest-unaligned.apk
Install Android Studio 2.0+
Android Studio should be installed on your codelab machine. If you're following this codelab on your personal machine, make sure you download and install Android Studio 2.0+ from here.
Import the sample project
With the code downloaded, the following instructions describe how to open the completed sample app in Android Studio.
Select the BasicSample directory from the sample code folder (Quickstart > Import Project... > android-testing > ui > espresso > BasicSample).
Install any missing packages and resolve suggested conflicts
Click the Gradle sync button. Resolve all issues.
If you encounter any issues, follow the suggestions from Android Studio to install missing packages until you completely resolve the issue.
Run a Firebase Test Lab test from Android Studio
From the top menu, click Edit Configurations go to the Run/Debug configurations dialog to run a test
It opens the Run/Debug configurations menu. Add a test configuration and configure the test parameters as follows:
Under Deployment Target Options, choose Firebase Test Lab Device Matrix.
You can define a different Matrix configuration by clicking on the following icon: and creating a different configuration.
Pick your cloud project by clicking on the following icon:
Click OK to save your changes and close the window.
Now you can run the test by clicking the Run configuration icon in Android Studio:
Install gcloud command-line interface and enable Firebase Test Lab plugin
Now that we've used Firebase Test Lab from Android Studio and ensured that our code works during development, let's take it one step further and use it from the command-line, which is the first step towards a more formal Continuous integration with our deployment workflow.
$ gcloud config set project <your firebase project name>
Make sure your authentication credentials are current.
$ gcloud auth login
Run a sample gcloud command to ensure the above steps finished correctly. You should see a table of devices which are available to test against in Firebase Test Lab. We'll come back to this table in a minute.
$ gcloud beta test android devices list
Run a Cloud Test from gcloud command-line interface
Now you're ready to run your first test. Try running the following from the BasicSample example's main folder we used earlier:
Here we ran a test on Nexus 9 and Nexus 5 physical devices, running APIs 19 and 21 in English and French locales on portrait and landscape orientations. The gcloud command provided the URL to the test results page, where results appear as each test is completed, as well as a summary table showing how many tests passed or failed.
To run the test on different devices, let's first look at a list of available devices by running this command:
$ gcloud beta test android devices list
Now we can pick another device, e.g. Samsung Galaxy S5 (whose device-ID is "k3g") and run the same test on it as follows:
Enable Cloud Testing API and Tool Results API from the APIs page in the Google Developers Console:
Using a service account tells gcloud that the user is a robot. This avoids checks for spam and prevents having the account mistakenly blocked or prompted for captchas.
Add a build step to Jenkins
Now you can add all the necessary steps to Jenkins and watch a build start and finish automatically. It's out of scope for this codelab but take 5-10 minutes to try it out anytime. If you're not familiar with Jenkins, take a look at the documentation here.
Connect your Jenkins build to the github repository
Add a trigger on new pull requests
Add a build step to run the gradle build commands mentioned above
Add a build step for the "gcloud test" commands mentioned above
Start a new build and then watch to see if Jenkins marks the build as successfully tested, or if it finds a test failure.
You have now tried different uses for Firebase Test Lab and passed a major milestone towards achieving better quality for your app!
What we've covered
Introduction to Firebase Test Lab and why it's better to test your app on a variety of devices
Using Firebase Test Lab from the web, Android Studio, command-line and integration with Jenkins
Using Firebase Test Lab even when you don't have automated tests within your app