Android Things Image Classifier

Android Things makes developing connected embedded devices easy by providing the same Android development tools, best-in-class Android framework, and Google APIs that make developers successful on mobile. With the TensorFlow Lite inference library for Android, developers can easily integrate TensorFlow and machine learning into their apps on Android Things.

TensorFlow™ is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. TensorFlow has become a popular framework for training machine learning models and using those models to solve problems.

What you'll build

In this codelab, you will use the TensorFlow inference library for Android to build a device that captures images from the device camera and locally classifies them against a pre-trained ImageNet model.

What you'll learn

How to integrate the TensorFlow Lite inference library for Android
How to feed data into a TensorFlow model
How to capture an image using the Android camera APIs

What you'll need

Raspberry Pi 3 or NXP i.MX7D developer board
Camera module
Rainbow HAT for Android Things
Android Studio 3.0+

The expansion connector on the development board exposes Peripheral I/O signals for application use. The Rainbow HAT sits on top of the expansion connector, providing a variety of inputs and outputs for developers to interact with.

The Peripheral I/O ports on the Rainbow HAT used in this codelab are connected to the following signals. These are also listed on the back of the Rainbow HAT:

Peripheral Device	Raspberry Pi 3	i.MX7D
'C' Button	`BCM16`	`GPIO2_IO07`

Camera Module

The developer boards are equipped with a Camera Serial Interface (CSI) connector to integrate supported camera modules with Android Things. The CSI bus is a high-speed, dedicated interface for capturing camera data. Supported camera modules connected to the developer board are accessed using the standard Android Camera APIs.

Click the following link to download the starter project for this codelab:

Download source code

...or you can clone the GitHub repository from the command line:

$ git clone https://github.com/googlecodelabs/androidthings-imageclassifier

About the project

The starter project contains the following:

An Android Activity (ImageClassifierActivity) with the basic code, button handling and a few placeholders for your code
Camera helper classes (ImagePreprocessor and CameraHandler)
TensorFlow helper classes (Recognition and TensorFlowHelper)
A sample photo of a dog
A serialized TensorFlow neural network already trained to recognize images

This sample photo for this project is located at res/drawable-nodpi/sampledog_224x224.png:

This is "Proto", an adorable Portuguese Water Dog. I know you want to spend some time looking at him, so take your time...

Import and run the starter project

Open the imageclassifier-start project in Android Studio and run it:

Open Android Studio, and close any existing projects you may have opened with File → Close Project.
Choose Import project from the welcome screen.
Navigate to the project directory you downloaded in the previous step.
Select the imageclassifier-start subdirectory.
Click Open. The project will take a few moments to import and build.
Select Run → Run 'app' from the menu, or click the Run icon in the toolbar.

Once the app launches on the device, look for the following line in the Android Logcat output:

Android Logcat

... D/ImageClassifierActivity: Initializing...
... D/ImageClassifierActivity: Press the button to take a picture

If you have a graphical display connected, it will display the same result message:

Tap the 'C' button on the HAT, you should see "I don't understand what I see." appear in Android Logcat:

Android Logcat

... D/ImageClassifierActivity: I don't understand what I see

If you have a graphical display connected, it will display the same result message and our sample image of Proto:

Nothing else should happen at this point. In the next steps you will be adding artificial intelligence (TensorFlow) to actually recognize the photo, and camera capturing.

Our current starter project doesn't do anything with the image. If you look at the doRecognize() method, it ignores the image parameter and reports an empty set of results, meaning that nothing was recognized. Let's wire an artificial intelligence engine here, so that the Android Things device can actually recognize what it sees.

Add the TensorFlow Lite Library

TensorFlow is an open-source library for machine learning and deep neural networks created by Google. TensorFlow Lite is TensorFlow's lightweight solution for mobile and embedded devices. It is available as a Gradle dependency on JCenter.

Add the library dependency to your app-level build.gradle file and re-sync the Gradle project.

build.gradle

dependencies {
    ...
    implementation 'org.tensorflow:tensorflow-lite:0.1.7'
}

Add the TensorFlow Lite calls

Open ImageClassifierActivity.java and make the following changes:

Add code to initialize a TensorFlow Interpreter from the model file (the actual deep neural network) in assets. Read the file using the loadModelFile() helper method.
Use the readLabels() helper method to pull in the list of labels that map the neural network results to names/categories recognized:

ImageClassifierActivity.java

import org.tensorflow.lite.Interpreter;

public class ImageClassifierActivity extends Activity {
    private static final String LABELS_FILE = "labels.txt";
    private static final String MODEL_FILE = "mobilenet_quant_v1_224.tflite";

    ...
 
    private Interpreter mTensorFlowLite;
    private List<String> mLabels;

    ...

    /**
     * Initialize the classifier that will be used to process images.
     */
    private void initClassifier() {
        try {
            mTensorFlowLite =
                    new Interpreter(TensorFlowHelper.loadModelFile(this, MODEL_FILE));
            mLabels = TensorFlowHelper.readLabels(this, LABELS_FILE);
        } catch (IOException e) {
            Log.w(TAG, "Unable to initialize TensorFlow Lite.", e);
        }
    }

    /**
     * Clean up the resources used by the classifier.
     */
    private void destroyClassifier() {
        mTensorFlowLite.close();
    }

    ...
}

Now let's implement the doRecognize() method. This method is called whenever the user requests an image classification. It takes the image as a parameter and runs the onPhotoRecognitionReady callback method with a list of Recognition items describing what was recognized in the input image (for example, ["beer bottle", "water bottle"]) and the confidence level of each result:

ImageClassifierActivity.java

public class ImageClassifierActivity extends Activity {
    ...
    private void doRecognize(Bitmap image) {
        // Allocate space for the inference results
        byte[][] confidencePerLabel = new byte[1][mLabels.size()];
        // Allocate buffer for image pixels.
        int[] intValues = new int[TF_INPUT_IMAGE_WIDTH * TF_INPUT_IMAGE_HEIGHT];
        ByteBuffer imgData = ByteBuffer.allocateDirect(
                DIM_BATCH_SIZE * TF_INPUT_IMAGE_WIDTH * TF_INPUT_IMAGE_HEIGHT * DIM_PIXEL_SIZE);
        imgData.order(ByteOrder.nativeOrder());

        // Read image data into buffer formatted for the TensorFlow model
        TensorFlowHelper.convertBitmapToByteBuffer(image, intValues, imgData);

        // Run inference on the network with the image bytes in imgData as input,
        // storing results on the confidencePerLabel array.
        mTensorFlowLite.run(imgData, confidencePerLabel);

        // Get the results with the highest confidence and map them to their labels
        Collection<Recognition> results =
                TensorFlowHelper.getBestResults(confidencePerLabel, mLabels);
        // Report the results with the highest confidence
        onPhotoRecognitionReady(results);
    }
 
    ...
}

Verify the new behavior

Deploy the app to the device by selecting Run → Run 'app' from the menu, or click the Run icon in the toolbar.

Press the "C" button and verify that you see the following messages in logcat:

... D/ImageClassifierActivity: Running photo recognition
... D/ImageClassifierActivity: Using sample photo in res/drawable-nodpi/sampledog_224x224.png
... D/ImageRecognition: ...
... D/ImageClassifierActivity: curly-coated retriever, Border collie or English springer

Understand What's Happening: The magic of deep learning

It is not the goal of this codelab to explain neural networks, but you deserve at least a brief explanation of what is going on.

Google TensorFlow researchers developed a lightweight neural network model called MobileNet and published pre-trained versions of it for image recognition, trained with a bit more than one million images from ImageNet classified within 1000 categories (or "classes", in machine learning parlance).

Your starter project already has the binary file representing this pre-trained neural network in the assets folder. In the initClassifier() method, we read this model and the labels associated with each of the 1000 categories (like "dishwasher", "Border collie" or "English springer").

In the doRecognize() method, the code extracts the pixels of the given image in a way compatible to how the network was trained: 24 bits per pixel of the original image, one for each basic color (red, green and blue). Also, the image is required to have 224x224 pixels, so the image is resized if necessary.

Once the pixels stored on imgData buffer in a format that is compatible with the inputs expected by the neural network, we run the inference that classifies the image — this is where the actual magic happens. The network is massive, and running it involves a lot of numeric computation. This step itself takes around half second on a Raspberry Pi 3.

The results from running the network are returned as an array of float confidence levels ranging from 0 to 1 per category. A higher value means a stronger confidence that the photo is of that category. The getBestResults() method in the TensorFlowHelper class finds the highest values and returns their labels.

If you want to retrain this network for your own purposes, you can learn more here and here.

If you have a graphical display connected, you should see the same output there as well:

Now we will add code to fetch an image from the board's camera, so you will be able to point the camera at objects and test the object recognition more properly.

Add Camera permission

To access the camera, you will need the proper permissions. Add the following permission declaration to your app's manifest file.

AndroidManifest.xml

<!-- TODO: ADD CAMERA SUPPORT -->
<uses-permission android:name="android.permission.CAMERA"/>

Update the activity to use the camera

Let's add the code to manage the camera connection and take photos. Open ImageClassifierActivity and make the following changes:

Add the variables that will hold the camera-related objects:

ImageClassifierActivity.java

import android.media.ImageReader;
...

public class ImageClassifierActivity extends Activity {

    ...
    private CameraHandler mCameraHandler;
    private ImagePreprocessor mImagePreprocessor;

    ...
}

Now update the initCamera() method so that it initializes the ImagePreprocessor and CameraHandler objects.
Implement an OnImageAvailableListener to be called when an image from the camera is ready. The listener should invoke ImagePreprocessor.preprocessImage() and pass the processed Bitmap to onPhotoReady(), which will reroute it to the image recognition method you implemented in the previous section.
Pass the new listener to CameraHandler.initializeCamera():

ImageClassifierActivity.java

/**
 * Initialize the camera that will be used to capture images.
 */
private void initCamera() {
    mImagePreprocessor = new ImagePreprocessor(
            PREVIEW_IMAGE_WIDTH, PREVIEW_IMAGE_HEIGHT,
            TF_INPUT_IMAGE_WIDTH, TF_INPUT_IMAGE_HEIGHT);
    mCameraHandler = CameraHandler.getInstance();
    mCameraHandler.initializeCamera(this,
            PREVIEW_IMAGE_WIDTH, PREVIEW_IMAGE_HEIGHT, null,
            new ImageReader.OnImageAvailableListener() {
                @Override
                public void onImageAvailable(ImageReader imageReader) {
                    Bitmap bitmap = mImagePreprocessor.preprocessImage(imageReader.acquireNextImage());
                    onPhotoReady(bitmap);
                }
            });
}

Finally, implement closeCamera() and loadPhoto() with code that closes the camera and triggers the CameraHandler:

ImageClassifierActivity.java

/**
 * Clean up resources used by the camera.
 */
private void closeCamera() {
    mCameraHandler.shutDown();
}

/**
 * Load the image that will be used in the classification process.
 * When done, the method {@link #onPhotoReady(Bitmap)} must be called with the image.
 */
private void loadPhoto() {
    mCameraHandler.takePicture();
}

Verify the new behavior

Deploy the app to the device by selecting Run → Run 'app' from the menu, or click the Run icon in the toolbar.

Point the device camera at an object and press the "C" button to take a picture. Verify that you see several messages in logcat similar to the following:

... D/ImageClassifierActivity: Running photo recognition
...
... D/CameraHandler: Capture request created.
...
... D/ImageClassifierActivity: I see a window shade
... D/CameraHandler: CaptureSession closed

Understand What's Happening: Android Camera API

Using cameras on Android involves some non-trivial code, due to the optimized pipeline used by the camera hardware and the image surface. For your convenience, we've hidden most of the complexity behind three classes in your starter project: CameraHandler and ImagePreprocessor.

The big picture (no pun intended) is that CameraHandler handles the Camera2 Android APIs, including initializing, configuring the actual camera and requesting a new image. ImagePreprocessor converts the image into a format compatible with the TensorFlow model.

Different Android Things boards may support different image formats. If you want to run this codelab on a board different than the NXP Pico i.MX7D, you should check what format your board supports and, if necessary, write code to convert. For the Raspberry Pi 3, you can use the code in this sample.

The code saves the bitmap for debugging right before sending it to the TensorFlow inference process. You can see it by pulling the file:

adb pull /storage/emulated/0/Pictures/tensorflow_preview.png /tmp/preview.png

Congratulations! You've successfully built an image classifier using TensorFlow Lite and Android Things! Here are some things you can do to go deeper.

Modify the ImageClassifierActivity to include a live camera preview inside the graphical UI (layout) before the image capture is triggered.
Note that this will require some changes in CameraHandler as well.
Train your own TensorFlow neural network to classify other objects.
Use Android TextToSpeech to speak the classification results aloud.

What we've covered

Using TensorFlow inference on an Android Things device
Integrating with the Android Camera APIs