4. On-device Intelligence with ML Kit

Written by Zahidur Rahman Faisal

Modern mobile apps deliver intelligent, personalized, and responsive experiences. For years, this intelligence was powered by large-scale cloud servers. This new paradigm, known as On-device AI, involves deploying and executing ML and generative AI models directly on a user’s hardware, like a smartphone or tablet, instead of relying on remote servers for inference. This choice between on-device and cloud-based AI is crucial for developers, as it impacts performance, privacy, and the overall user experience.

ML Kit, a mobile SDK, brings Google’s on-device machine learning expertise to Android apps. With the powerful yet easy-to-use Generative AI (GenAI), Vision, and Natural Language APIs, you can solve common challenges in your apps or create entirely new user experiences.

In this chapter, you’ll harness the power of ML Kit and create a sample app that will:

• Scan documents and save them as images or PDFs.
• Extract text from the saved documents and share it online.

Let’s get started with ML Kit!

ML Kit on Android

ML Kit is an easy-to-use SDK that brings Google’s extensive machine learning expertise to mobile developers, abstracting away the complexities of model management and inference. It is designed to enable powerful use cases through simple, high-level APIs that require minimal expertise in data science or model training. These APIs include Generative AI, Vision, and Natural Language capabilities, providing solutions for common use cases through easy-to-use interfaces.

ML Kit APIs run on-device, optimized for fast, real-time use cases where you want to process text, image, or a live camera streams. The ML Kit APIs are categorized as follows, based on their ML models:

GenAI APIs

• Text Summarization: Concisely summarize articles or chat conversations into a bulleted or concise list.
• Proofreading: Polish short content by refining grammar and correcting spelling errors.
• Rewriting: Rephrase short messages in various tones or styles.
• Image Description: Generate a short description for a given image.

Vision APIs

• Text Recognition: Recognize text from images, and fragment it into blocks, lines, elements and symbols.
• Digital Ink Recognition: Recognize hand‑drawn shapes and emojis.
• Face Detection: Detect faces in an image, identify key facial features, and extract contours of detected faces.
• Pose Detection: Detect the pose of a subject’s body in real time from a continuous video stream or static image.
• Object Detection and Tracking: Detect and track objects in an image or a live camera feed.
• Subject Segmentation: Detect and separate multiple subjects from the background of a picture.
• Image Labeling: Detect and extract information about objects and entities in an image.
• Barcode Scanning: Read standard barcode formats without requiring an internet connection.
• Document Scanner: Convert physical documents into digital formats (e.g., images or PDFs).

Natural Language APIs

• Entity Extraction: Identify specific entities within static text and trigger context-based actions for the user depending on the entity type.
• Language Identification: Determine the language of a given piece or string of text.
• Translation: Dynamically translate text between more than 50 languages, even when the device is offline.
• Smart Reply: Automatically generate contextually relevant and concise replies to messages.

The ML Kit APIs are built upon AICore, an Android system service that facilitates the on-device execution of foundation models ensuring these features process data locally and maintain a high level of privacy.

Creating a Document Scanner using ML Kit

You’ll learn how easily you can create a custom Document Scanner using MLKit! You can rely on the Document Scanner module from MLKit. The Document Scanner APIs can provide below capabilities to your app:

• Automatically capture documents when they are detected.
• Optimize cropping results with accurate edge detection.
• Automatically rotate documents to display them in the correct orientation.
• Offer built-in editing features such as cropping, applying filters, removing shadows, and cleaning stains.

The first step is to add the required dependency so you can start using these APIs.

Adding Dependency

1. Open the app level build.gradle file from the starter project.

2. Add the latest version of the Document Scanner at the bottom of the dependencies block.

3. Press ‘Sync Project with Gradle Files’ to download and sync the dependency with your app.

var documentScannerVersion = "16.0.0-beta1"
implementation "com.google.android.gms:play-services-mlkit-document-scanner:$documentScannerVersion"

Preparing the Scanner

You’re now ready to use the dependency and its helper classes. Before you can start scanning, you need to configure the Document Scanner client. To do so, open MainViewModel.kt and update the prepareScanner() function as follows:

fun prepareScanner(): GmsDocumentScanner {
  val options = GmsDocumentScannerOptions.Builder()
    .setPageLimit(3)
    .setResultFormats(RESULT_FORMAT_JPEG)
    .setScannerMode(SCANNER_MODE_FULL)
    .build()
  return GmsDocumentScanning.getClient(options)
}

The options variable allows you to configure the Document Scanner client. These are the configurations you modified here:

• The setPageLimit() function limits the number of pages to be scanned in a session.

• setResultFormats() sets the output format of results. Using RESULT_FORMAT_JPEG as the parameter sets your output as images; you could use RESULT_FORMAT_PDF if you need your documents converted as PDF.

• Setting SCANNER_MODE_FULL in the setScannerMode() function enables all available features of the Document Scanner. If you prefer to restrict certain advanced capabilities, such as image filters, you can use SCANNER_MODE_BASE instead to control the image editing capabilities.

Once the configuration options are provided, the prepareScanner() function returns a GmsDocumentScanner instance, which you’ll be using in the next steps.

Creating the Scanner Launcher

Next, open MainActivity and add the following code snippet above the onCreate() method:

val scannerLauncher = registerForActivityResult(
  contract = ActivityResultContracts.StartIntentSenderForResult()
) { result ->
    if (result.resultCode == RESULT_OK) {
    val scanResult = GmsDocumentScanningResult.fromActivityResultIntent(result.data)
    viewModel.extractPages(scanResult = scanResult)
  }
}

This is the activity launcher for the Document Scanner. It starts the Document Scanner intent and waits for the result.

You’re parsing the scanned data through GmsDocumentScanningResult.fromActivityResultIntent(result.data) and assigning it to scanResult if the scan completes successfully. The next step is to iterate through all the pages in the scanResult object.

You’re not ready to launch the app yet; you still need to implement the extractPages() function to extract data from each page — you’ll do that in the next section.

Handling Result

Go back to MainViewModel.kt again. You’ll see there’s a MutableStateList named pageUris defined at the top – this list will contain the URI of each page from the scan result, which will be used later.

Now update the extractPages() function as follows:

fun extractPages(scanResult: GmsDocumentScanningResult?) {
  viewModelScope.launch(Dispatchers.IO) {
    scanResult?.pages?.let { pages ->
      pageUris.clear()

      for (page in pages) {
        pageUris.add(page.imageUri)
      }
    }
  }
}

It’s a simple function that executes as follows:

• Takes scanResult as an input parameter and checks whether it contains one or more pages.

• If new pages are detected, it clears the existing pageUris list before adding the new results.

• It then iterates through the pages list. Since you set RESULT_FORMAT_JPEG as the output format earlier, each page will contain an imageUri. The function adds the imageUri of each page to the pageUris list.

Scanning Documents

You’re all set to launch the Document Scanner at this point and use the resultant data. You need to update the launchDocumentScanner() function in MainActivity.kt as shown below:

private fun launchDocumentScanner() {
  viewModel
    .prepareScanner()
    .getStartScanIntent(this@MainActivity)
    .addOnSuccessListener { it ->
      val scannerIntent = IntentSenderRequest.Builder(it).build()
      scannerLauncher.launch(scannerIntent)
  }
}

This function prepares the Document Scanner intent using the configurations provided in the prepareScanner() function. Once it’s ready, it creates an IntentSenderRequest using that intent and launches scannerLauncher to start scanning.

The final step is to trigger the above function on ScanButton click:

ScanButton(
  onClickScan = {
    launchDocumentScanner()
  }
)

Note: To compile the app properly, make sure you’ve added all the required imports after your code changes in both MainActivity.kt and MainViewModel.kt files.

Finally, Build and run the app. Tap the Scan icon at the bottom of the screen and your app will become a fully functional Document Scanner!

Extracting Text using Text Recognizer

MLKit made it easy to turn your app into a Document Scanner, but what if you also want to extract text (OCR) from your scanned documents? This part of the chapter will teach you how to do exactly that!

The Text Recognition API can recognize text in various character sets, in real-time, on a wide range of devices. Key capabilities of this API include:

• Recognizing text in Chinese, Devanagari, Japanese, Korean, and Latin scripts.

• Identifying the language of the recognized text.

• Detecting symbols, elements, lines, and paragraphs.

To enable your app for Text Recognition, open the app level build.gradle file and add the following dependency:

var textRecognitionVersion = "16.0.1"
implementation "com.google.mlkit:text-recognition:$textRecognitionVersion"

Then, sync the dependency with your app.

Recognizing Text from Image

Remember saving your scanned pages as images? That’ll come in handy now. The MainViewModel keeps the reference of those image URIs in the pageUris variable. This list is used to display a carousel of your scanned pages in MainActivity.kt, which looks like this:

The “Extract Text” button is intended to start the Text Recognition process on each page and share the extracted text once the operation is complete.

To implement this, open MainViewModel.kt and update the getTextFromImage() function as follows:

fun getTextFromImage(image: Uri, onCompleted: (String?) -> Unit) {
  viewModelScope.launch(Dispatchers.IO) {
    val image = fromFilePath(application, image)
    TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS)
      .process(image)
      .addOnSuccessListener { visionText ->
        val resultText = visionText.text
        onCompleted(resultText)
      }
      .addOnFailureListener { e ->
        onCompleted(null)
      }
  }
}

This function performs all the heavy lifting of Text Recognition. Here’s how it works:

1. It takes the image Uri from the input parameters and converts it into an InputImage from the file path.

2. Next, the TextRecognition client processes the InputImage. It attempts to detect and read the characters as Blocks of Elements.

1. If text recognition is successful, visionText.text returns the recognized text block as a single String, separated by new lines, through the onCompleted() callback. You can use visionText.textBlocks instead, if you prefer a two-dimensional collection of strings to iterate through each line.

2. If the TextRecognition client fails to detect any text from the given image, it returns null in the onCompleted() callback.

The onCompleted() callback passes the result to the MainActivity.kt, where you’ll handle and display the extracted text.

Handling Result

At this point, viewModel.getTextFromImage(uri) will return the extracted text from the image. You might want to use or share this text within your app. To do that, open MainActivity.kt and add the following function:

private fun shareTextFromImage(uri: Uri) {
  viewModel.getTextFromImage(uri) { extractedText ->
    extractedText?.let {
      shareText(text = it)
    }
  }
}

This function performs two tasks at once:

1. Takes an image URI as input, loads the image into memory, and attempts to extract text from it.

2. If text extraction is successful, it shares the extracted text.

The shareText() function is a helper method uses the Android Sharesheet to share text. It looks like this:

private fun shareText(text: String) {
  val intent = Intent().apply {
    action = Intent.ACTION_SEND
    type = "text/plain"
    putExtra(Intent.EXTRA_TEXT, text)
  }
  val shareIntent = Intent.createChooser(intent, "Text from Image")
  startActivity(shareIntent)
}

Add the shareText() function at the bottom of the MainActivity.

Now it’s time to put everything together! Pass shareTextFromImage() function inside the PageCarousel component:

PageCarousel(
  pages = viewModel.pageUris,
  onItemClick = { uri ->
    shareTextFromImage(uri = uri)
  }
)

With this change, shareTextFromImage() will be invoked for each PageItem, using the URI from that page to extract text from the image and share it as needed.

Note: To compile the app properly, make sure you’ve added all the required imports after your code changes in both MainActivity.kt and MainViewModel.kt files.

Build and run the app again. The end result may look like this:

Congratulations! You’ve now turned your app into a full-fledged Document Scanner and OCR tool using MLKit. Feel free to explore other APIs from MLKit to animate your next innovative idea!

On-device AI surely is handy for processing offline, but there are some trade-offs you need to consider when using it. In the next section you’ll learn about those.

The Trade-offs of On-device AI

On-device AI is optimized for scenarios where data processing must be immediate, private, and available without a network connection, but it comes with some strategic trade-offs.

The Benefits

These are the key benefits of using on-device AI:

Privacy and Security

With on-device processing, sensitive personal data, such as images, voice recordings, or private messages, never needs to leave the user’s device. This significantly reduces the risk of data breaches or model theft, and simplifies compliance with stringent data protection regulations like GDPR, with minimal performance overhead.

Latency

By performing inference locally, on-device AI eliminates network round-trip delays, resulting in near-instantaneous responsiveness. This is essential for real-time applications such as augmented reality (AR) filters, live camera analysis, and voice assistants that must respond without perceptible lag.

Offline Functionality

On-device models enable offline functionality, allowing applications to remain fully operational in environments with poor or nonexistent connectivity, which is a critical consideration for a global user base.

Operational Costs

For developers, on-device inference reduces ongoing server and bandwidth expenses associated with repeated cloud API calls. Running tasks locally is also more energy-efficient, consuming up to 90% less energy than cloud-based inference.

The Limitations

Despite these benefits, on-device AI is not without its challenges. The primary limitations are:

Computational Constraint

Even though modern mobile device hardware is becoming increasingly powerful, it cannot match the scale of a cloud data center. This limits the size and complexity of models that can run efficiently on a device.

Model Management

Managing models becomes more complex with on-device AI. While a cloud model can be updated instantly for all users, on-device models must be packaged with the application and distributed through app updates, making the process more time-consuming and logistically challenging.

Battery Consumption

Even optimized on-device inference can contribute to increased battery usage, particularly for computationally intensive tasks. Developers should focus on optimizing background tasks, limiting unnecessary requests, and using power-efficient APIs to minimize battery drain.

App Size

While using on-device models, you as a developer, must also consider broader app performance. Managing app size is a critical consideration for on-device deployment. Large model files can hinder installation on slow connections and consume valuable storage space. Best practices include using Android App Bundles, which dynamically deliver only the necessary code and resources to a user’s device, and leveraging tools like the Android Size Analyzer to identify areas for size reduction.

Conclusion

A comprehensive analysis of these trade-offs reveals that the architectural decision for AI-powered features is rarely a simple, binary decision. The most robust solutions are often hybrid models that combine the strengths of both approaches. A common design pattern involves using on-device AI for basic data preprocessing and low-latency tasks, such as initial object detection in a live camera feed, while reserving more complex, high-volume analysis for cloud-based services. This enables a fluid user experience while leveraging cloud power when necessary.

A closer look at the on-device AI landscape shows a strong emphasis on privacy and security. This focus is not merely a technical advantage but a core strategic principle. With growing public awareness and increasing regulatory pressure, a product’s ability to keep user data private is becoming a key market differentiator.

The design of ML Kit, with its “no training needed” philosophy and simple APIs, represents a deliberate strategy to democratize AI development. By providing turn-key solutions, Google is lowering the barrier to entry, allowing developers to integrate sophisticated AI-powered features without the need for specialized data science expertise or the infrastructure required for training custom models. This shifts the focus from the mechanics of model development to the creative application of pre-trained intelligence.

For developers, this means the future of AI on Android is becoming both more powerful and more accessible. High-level abstractions like ML Kit will continue to democratize intelligence, while the managed runtime ensures consistent, performant, and secure execution across Android’s diverse ecosystem. The new era of mobile computing is not just about smarter applications, but about a fundamentally smarter operating system that provides a reliable and managed platform for the next generation of intelligent, private, and context-aware user experiences.