ML Kit is a mobile SDK that brings Google's machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you're new or experienced in machine learning, you can easily implement the functionality you need in just a few lines of code. Follow our Recognize text in images with ML Kit for Firebase codelab to get started with built-in models -- there's no need to have deep knowledge of neural networks or model optimization. On the other hand, if you are an experienced ML developer, complete this codelab to learn how ML Kit makes it easy to use your custom TensorFlow Lite models in your mobile apps.

If you run into any issues (code bugs, grammatical errors, unclear wording, etc.) as you work through this codelab, please report the issue via the Report a mistake link in the lower left corner of the codelab.

How does it work?

ML Kit makes it easy to apply ML techniques in your apps by bringing Google's ML technologies, such as the Google Cloud Vision API, Mobile Vision, and TensorFlow Lite, together in a single SDK. Whether you need the power of cloud-based processing, the real-time capabilities of Mobile Vision's on-device models, or the flexibility of custom TensorFlow Lite models, ML Kit makes it possible with just a few lines of code.

This codelab will walk you through creating your own Android app that can automatically detect and label images using a custom Tensor Flow Lite machine learning model in ML Kit for Firebase.

What you will build

In this codelab, you're going to build an Android app with Firebase ML Kit. You will:

  • Learn how to host your custom pre-trained Tensor Flow Lite model using Firebase
  • Use the ML Kit Custom Model API to download the pre-trained TensorFlow Lite model to your app
  • Use the downloaded model to run inference and label images

What you'll learn

What you'll need

This codelab is focused on ML Kit. Non-relevant concepts and code blocks are glossed over and are provided for you to simply copy and paste.

Download the Code

Click the following link to download all the code for this codelab:

Download source code

Unpack the downloaded zip file. This will unpack a root folder (mlkit-android) with all of the resources you will need. For this codelab, you will only need the resources in the custom-model subdirectory.

The custom-model subdirectory in the "mlkit" repository contains two directories for 2 Android Studio projects:

Download the Tensor Flow Lite model

Click the following link to download the pre-trained Tensor Flow Lite model we will be using in this codelab:

Download model

Unpack the downloaded zip file. This will unpack a root folder (mobilenet_v1_1.0_224_quant) inside which you will find the Tensor Flow Lite custom model we will use in this codelab (mobilenet_v1_1.0_224_quant.tflite).

Add Firebase to the project

  1. Go to the Firebase console.
  2. Select Add Project.
  3. In the drop down, select the existing project.
  4. Click Add Firebase.

Connect your Android app to your firebase console project, inside firebase console

  1. From the overview screen of your new Firebase project,
    click Add Firebase to your Android app.
  2. Enter the codelab's package name: com.google.firebase.codelab.mlkit_custommodel.

Add google-services.json file to your app

After adding the package name and selecting Continue, your browser automatically downloads a configuration file that contains all the necessary Firebase metadata for your Android app. Copy the google-services.json file into the .../custom-model/starter/app directory in your project.

Add the dependencies for ML Kit and the google-services plugin to your app

The google-services plugin uses the google-services.json file to configure your application to use Firebase, and the ML Kit dependencies allow you to integrate the ML Kit SDK in your app. The following lines should already be added to the end of the build.gradle file in the app directory of your project (check to confirm):

dependencies {
  // ...
  implementation 'com.google.firebase:firebase-ml-model-interpreter:22.0.0'
}
apply plugin: 'com.google.gms.google-services'

Sync your project with gradle files

To be sure that all dependencies are available to your app, you should sync your project with gradle files at this point. Select Sync Project with Gradle Files () from the Android Studio toolbar.

Now that you have imported the project into Android Studio and configured the google-services plugin with your JSON file, and added the dependencies for ML Kit, you are ready to run the app for the first time. Connect your Android device, and click Run (execute.png)in the Android Studio toolbar.

The app should launch on your emulator. At this point, you should see a basic layout that has a drop down field which allows you to select between several images. In the next section, you add image detection to your app to identify the objects in the images.

The Tensorflow Lite model receives a list of inputs and will output an array of bytes, one for each input. We will feed our image as input and interpret the model output in the function runModelInference in our code. To do that, let's replace the contents of that function with the following:

MainActivity.java

/** Uses model to make predictions and interpret output into likely labels. */
private fun runModelInference() = selectedImage?.let { image ->

    // Create input data.
    val imgData = convertBitmapToByteBuffer(image)

    try {
        // Create model inputs from our image data.
        val modelInputs = FirebaseModelInputs.Builder().add(imgData).build()

        // Perform inference using our model interpreter.
        modelInterpreter.run(modelInputs, modelInputOutputOptions).continueWith {
            val inferenceOutput = it.result?.getOutput<Array<ByteArray>>(0)!!

            // Display labels on the screen using an overlay
            val topLabels = getTopLabels(inferenceOutput)
            graphic_overlay.clear()
            graphic_overlay.add(LabelGraphic(graphic_overlay, topLabels))
            topLabels
        }

    } catch (exc: FirebaseMLException) {
        val msg = "Error running model inference"
        Toast.makeText(baseContext, msg, Toast.LENGTH_SHORT).show()
        Log.e(TAG, msg, exc)
    }
}

The pre-trained Tensorflow Lite model we will be using in our app is the MobileNet_v1 model, which has been designed to be used in low-latency, low-power environments, and offers a good compromise between model size and accuracy. In this step, we will take the .tflite file we downloaded earlier and put it into the assets directory of the app.

Next, we are going to fill out the implementation of the function createLocalModelInterpreter. Replace the contents of the function with the following code:

MainActivity.java

/** Initialize a local model interpreter from assets file */
private fun createLocalModelInterpreter(): FirebaseModelInterpreter {
    // Select the first available .tflite file as our local model
    val localModelName = resources.assets.list("")?.firstOrNull { it.endsWith(".tflite") }
        ?: throw(RuntimeException("Don't forget to add the tflite file to your assets folder"))
    Log.d(TAG, "Local model found: $localModelName")

    // Create an interpreter with the local model asset
    val localModel =
        FirebaseCustomLocalModel.Builder().setAssetFilePath(localModelName).build()
    val localInterpreter = FirebaseModelInterpreter.getInstance(
        FirebaseModelInterpreterOptions.Builder(localModel).build())!!
    Log.d(TAG, "Local model interpreter initialized")

    // Return the interpreter
    return localInterpreter
}

We are loading a model by looking inside of the assets folder and grabbing the first file ending in .tflite, which is the extension of trained TensorFlow models. Then, we build a FirebaseModelInterpreter object which loads that model into the internal TensorFlow Lite instance.

The createLocalModelInterpreter function is called within onCreate in a coroutine. We just need to uncomment the following line at the end of onCreate:

MainActivity.java

// Load the model interpreter in a coroutine
lifecycleScope.launch(Dispatchers.IO) {
    modelInterpreter = createLocalModelInterpreter() // <-- Uncomment this line.
    //modelInterpreter = createRemoteModelInterpreter()
    runOnUiThread { button_run.isEnabled = true }
}

Run the app on the emulator

Now click Run (execute.png) in the Android Studio toolbar. Once the app loads, make sure that Image 1 is selected in the drop down field and click on the RUN MODEL button.

Your app should now look like image below, showing the model inference results and the detected image labels with their confidence levels.

In this step, we will be hosting this model with Firebase by uploading it to our Firebase project. This enables apps using the MLKit SDK to automatically download the model to our devices, and allows us to do model version management easily in the Firebase Console.

Host the custom model with Firebase

  1. Go to the Firebase console.
  2. Select your project.
  3. Select ML Kit under the DEVELOP section in the left hand navigation.
  4. Click on the CUSTOM tab.
  5. Click on Add another model and use "mobilenet_v1_224_quant" as the name. This is the name we will later use to download our custom model in our Android code.
  6. In the TensorFlow Lite model section, click BROWSE and upload the mobilenet_v1_1.0_224_quant.tflite file you downloaded earlier.
  7. Click PUBLISH.

We are now ready to modify our Android code to use this hosted model.

Next, we are going to fill out the implementation of the function createRemoteModelInterpreter. Replace the contents of the function with the following code:

MainActivity.java

/** Initialize a remote model interpreter from Firebase server */
private suspend fun createRemoteModelInterpreter(): FirebaseModelInterpreter {
    return suspendCancellableCoroutine { cont ->
        runOnUiThread {
            Toast.makeText(baseContext, "Downloading model...", Toast.LENGTH_LONG).show()
        }

        // Define conditions required for our model to be downloaded. We only request Wi-Fi.
        val conditions =
            FirebaseModelDownloadConditions.Builder().requireWifi().build()

        // Build a remote model object by specifying the name you assigned the model
        // when you uploaded it in the Firebase console.
        val remoteModel =
            FirebaseCustomRemoteModel.Builder(REMOTE_MODEL_NAME).build()
        val manager = FirebaseModelManager.getInstance()
        manager.download(remoteModel, conditions).addOnCompleteListener {
            if (!it.isSuccessful) cont.resumeWithException(
                RuntimeException("Remote model failed to download", it.exception))

            val msg = "Remote model successfully downloaded"
            runOnUiThread { Toast.makeText(baseContext, msg, Toast.LENGTH_SHORT).show() }
            Log.d(TAG, msg)

            val remoteInterpreter = FirebaseModelInterpreter.getInstance(
                FirebaseModelInterpreterOptions.Builder(remoteModel).build())!!
            Log.d(TAG, "Remote model interpreter initialized")

            // Return the interpreter via continuation object
            cont.resume(remoteInterpreter)
        }
    }
}

The createRemoteModelInterpreter function is called within onCreate, we just need to comment the line we uncommented earlier, and uncomment the following line at the end of onCreate:

MainActivity.java

// Load the model interpreter in a coroutine
lifecycleScope.launch(Dispatchers.IO) {
    //modelInterpreter = createLocalModelInterpreter()  // <-- Comment this line.
    modelInterpreter = createRemoteModelInterpreter() // <-- Uncomment this line.
    runOnUiThread { button_run.isEnabled = true }
}

We can now run our app, and the model used will be coming from Firebase rather than our assets folder. The first time we run this, it will take some time to download the model from the cloud. For production applications, we should make sure that our app is resilient in cases where the network may be flaky or not available at all.

You have used ML Kit for Firebase to easily add advanced machine learning capabilities to your app.

What we've covered