OCR Archives - Truiton

Introducing Android Mobile Vision API

Mohit Gupt — Sun, 28 May 2017 17:29:27 +0000

In August 2015, Google announced the release of Android Mobile Vision API. At that time this API had mainly three components Face Detection, Barcode scanner and ways to capture the tracking of objects in real time. But later on some bugs were found in its implementation, due to which the access to this API was restricted by Google to the existing users only. Then recently in June 2016 these issues were resolved and Mobile Vision APIs were re-launched. But this time it had a few new features like OCR and scanning of Aztec barcodes. In the next sections lets understand what exactly is the new Mobile Vision API and how can we use it to build an efficient code base.

What is Android Mobile Vision API ?

Ever wondered how to detect a face, a QR code or a Bar code on an Android Device? If yes; you might have heard of, or used the FaceDetector.Face API of the Android framework, or the OpenCV SDK, or maybe you had opted for a cloud based solution like Cloud Vision API, which makes the requests to a web server, fetching the results for scans. But this new Android Mobile Vision API does not make any requests to a web server. Instead it performs real time image/video scanning on the device itself. Although this may sound a little in-effective, but its not. The Mobile Vision API is very efficient and deeply integrated in to the Android system by the means of Google Play Services SDK. This gives this API an added advantage over all other solutions, as being a developer you do not need to integrate any third party SDK to perform media analysis. All you may need to do is integrate the Google Play Services properly and start building on it. Android Mobile Vision API as of now performs three types of image/stream detection as shown in next sections.

Setting up Android Mobile Vision Library

When speaking of this new multipurpose Android offline image recognition library, the only downside is that one has to set it up before the actual usage. Although it does not mean any manual configuration but a tag needs to be added in the application manifest. So that the installer knows that an extra package is needed to be downloaded as the app is being installed. As of now the mobile vision API has three components, therefore all three dependencies can be included in a single manifest tag under the application tag as shown:

This would instruct the OS to download the packages for all three types of offline image analysis. Next lets have a brief understanding of all the features of this powerful Mobile Vision API.

1. Barcode and QR Code scanning on Android

The Barcode Scanner API is used to scan various types of bar codes and qr codes. Some of them are:

1D barcodes: EAN-13, EAN-8, UPC-A, UPC-E, Code-39, Code-93, Code-128, ITF, Codabar
2D barcodes: QR Code, Data Matrix, PDF-417, AZTEC

Using this Mobile Vision API for Bar codes is very simple, even with so many supported formats. All you need to do is write around 10 lines of code to parse a Barcode or QR Code. One of the most interesting things about this API is that, it also parses the type of barcode/QR code it is scanning. All the information about the scanned barcode or QR Code type is found in the valueFormat field of Barcode Object besides the scanned barcode. To learn more about the usage of Barcode Scanner API of the Android Mobile Vision API please refer to this tutorial:

Android Barcode Tutorial

2. Optical Character Recognition (OCR) on Android

Mobile Vision Text API is another great feature packed into this library. Its purpose is straight forward, i.e. text recognition in an image or a stream of frames. It can recognize a majority of Latin character based languages like English, French, Italian Dutch etc. Therefore if your app needs to extract text out of an image, this is the perfect solution for you. As once its dependencies are downloaded, you no more required to have an internet connection, all the character recognition takes place on device itself.

Incase you wonder; how would this Android Mobile Vision Text API deliver large results, as when scanning a page of a book- a lot of words could be scanned. But thankfully this API returns text in a structured way. It mainly divides text into three sections:

Block: Top level structure, contains all the paragraphs.
Line: Mid level structure, contains lines in a block.
Word: Low level structure, contains a single word from line.

To learn more about the Text API, please refer to this tutorial:

Android OCR Tutorial

3. Face Detection on Android

This is the most powerful API in all, as it has human Face Detection capabilities. It is a perfectly suited API for any face filter or camera app, as it performs the analysis on the device itself once the package is downloaded. Interestingly it not only recognizes a face but can also extract the facial features of that face, including eyes nose and mouth etc. Since this API as of now does not support face recognition it cannot identify similarity between two faces, but can still classify the features like, if the eyes are open or not. The functionalities that Mobile Vision Face Detection API supports are:

Landmark Detection: Face API understands the human face in terms of landmarks. When a face is scanned via this API, it identifies that face via landmarks. In simple terms face landmarks are: nose, mouth, left eye, and right eye etc. By using this API you can actually extract the position of all these landmarks.
Classification: This API not only scans a face but can also apply some basic logic and identify certain characteristics on the scanned face. For ex. with this feature we can find out whether the face has its eyes open or not. Also we can find out the probability of a smile on that face.
Tracking: This is the most interesting feature of this API, you can actually track a face in a video sequence through this API. This feature of Mobile Vision API can be used to identify and track a face in a video. Once again this is not an application of face recognition, instead it tracks the face through movement of that particular face in the video.

To learn more in detail about Face Detection, please refer to this tutorial:

Face Detection Tutorial

Tracking Faces, Bar codes and QR Codes Simultaneously

Interestingly, if you want to scan and track multiple faces simultaneously, even that is possible through this API. All you need to do is initialize a MultiProcessor and track multiple faces through this API. Also if you want to track multiple Bar codes or QR codes, same thing is applicable, as all of this is a part of Mobile Vision API, backed by Google Play Services. But even more interesting thing is that you can track Bar codes, QR codes and Faces in a single frame, by using the MultiDetector class of the same API suite. This makes the mobile vision APIs an outstanding leader in terms of features when compared to any other such library in the market. As its light, easy to integrate and very easy to use. For more updates, please connect with us on Twitter, Facebook and Google+.

Mohit Gupt

Born in New Delhi, India. A software engineer by profession, an android enthusiast and mobile development evangelist. My motive here is to create a group of skilled engineers, who can build better software. Reason being programming is my passion, and also it feels good to make a device do something you want. Professionally I have worked with many software engineering and product development firms. As of now too, I am employed as a senior engineer in a leading tech company. In total I may have worked on more than 20 projects professionally, but whenever I get spare time I share my thoughts here at Truiton.

www.truiton.com

The post Introducing Android Mobile Vision API appeared first on Truiton.

Optical Character Recognition on Android – OCR

Mohit Gupt — Sun, 06 Nov 2016 15:12:47 +0000

Android itself is a smart OS, still it lacked a very basic feature of text recognition. But not anymore; with the official Optical Character Recognition API of Android and the Mobile Vision library, now Android can perform OCR very efficiently and correctly. I did a very basic feature test to have a look at the new functionality and found out its very fast and easy to use. Here in this Optical Character Recognition(OCR) example of Android, I would simply import the library, click a picture of a piece of text and look for text blocks in it. But before doing so lets take an overview of Android Mobile Vision API to understand the working of Text API better.

Android Mobile Vision Library

Consider a scenario where you wish to scan and an image or a video stream and detect faces, barcodes, QR codes or Text in it on Android. Astonishingly till now no such framework existed on Android. But now Google has introduced Mobile Vision APIs. These set of APIs provide a very easy to use programming interfaces through which we can scan Faces, Barcodes, QR Codes and Text without writing huge amount of code. And the best part is that these can be used offline as well, i.e. only once when the app is downloaded the required dependencies are downloaded, post that an internet connection is not required any more. As this feature is introduced on Android through the Google Play services. To enable your app to use Mobile Vision APIs, you need to add this dependency in your build.gradle file:

compile 'com.google.android.gms:play-services-vision:11.4.0'

Please Note: Learn more on how to setup Google Play Services. Or learn more about Mobile Vision API.

Introducing an Android OCR Library – Text Recognition API

Since the Android OS was brought on to production devices, Optical Character Recognition has been a common area of research. But this Text Recognition API of Mobile Vision suite would bring all these researches to a stop. As this Google powered API contains features like multiple language recognition where languages are like : English, French, German, Spanish or any other Latin based text. Also the text can be parsed from a stream of frames i.e. a video and displayed on the screen in real time as displayed in the image above. But due to the scope of this Android OCR Library example we would keep things simple and scan the text from an image only, as this tutorial is targeted for beginners. Apart from this the interesting part is, all this is done offline by the Google Play services itself, i.e. no internet connection is required after once it has been set up in the app (shown in steps ahead). Now when it comes to structuring the text, this Android OCR library not only recognizes the text but can also divide the captured text into the following categories:

Block – TextBlock – A top level object where a scanned paragraph or column is captured.
Line – Line – A line of text captured from a block of text.
Word – Element – A single word recognized in a Line.

Android OCR Example

Now that we have a basic understanding of Android OCR library, particularly the Text Recognition API. I will demonstrate by an example where we would simply take a picture and scan for text in it. Now as a thumb rule to do so first we may need to set up our Android app to download the play services dependency for Optical Character Recognition. Therefore please include the block of code below in your manifest to instruct installer to download the OCR dependency at the time of installing the app.

Please Note: This is not a mandatory step, but helps in downloading the dependencies beforehand. Also the link to full source code is at the end of this tutorial.

Next lets define a layout to display the scanned results:

I was playing around with ConstraintLayout in Android. Hence made it in Constraint layout, although its not a requirement to use ConstraintLayout for this Android OCR Library example. The above layout basically contains a ScrollView to accurately display the scanned text. Next lets define the main Activity:

package com.truiton.mobile.vision.ocr;

import android.Manifest;
import android.content.Context;
import android.content.Intent;
import android.content.pm.PackageManager;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.provider.MediaStore;
import android.support.annotation.NonNull;
import android.support.v4.app.ActivityCompat;
import android.support.v4.content.FileProvider;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.util.SparseArray;
import android.view.View;
import android.widget.Button;
import android.widget.TextView;
import android.widget.Toast;

import com.google.android.gms.vision.Frame;
import com.google.android.gms.vision.text.Text;
import com.google.android.gms.vision.text.TextBlock;
import com.google.android.gms.vision.text.TextRecognizer;

import java.io.File;
import java.io.FileNotFoundException;

public class MainActivity extends AppCompatActivity {
    private static final String LOG_TAG = "Text API";
    private static final int PHOTO_REQUEST = 10;
    private TextView scanResults;
    private Uri imageUri;
    private TextRecognizer detector;
    private static final int REQUEST_WRITE_PERMISSION = 20;
    private static final String SAVED_INSTANCE_URI = "uri";
    private static final String SAVED_INSTANCE_RESULT = "result";

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Button button = (Button) findViewById(R.id.button);
        scanResults = (TextView) findViewById(R.id.results);
        if (savedInstanceState != null) {
            imageUri = Uri.parse(savedInstanceState.getString(SAVED_INSTANCE_URI));
            scanResults.setText(savedInstanceState.getString(SAVED_INSTANCE_RESULT));
        }
        detector = new TextRecognizer.Builder(getApplicationContext()).build();
        button.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view) {
                ActivityCompat.requestPermissions(MainActivity.this, new
                        String[]{Manifest.permission.WRITE_EXTERNAL_STORAGE}, REQUEST_WRITE_PERMISSION);
            }
        });
    }

    @Override
    public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults);
        switch (requestCode) {
            case REQUEST_WRITE_PERMISSION:
                if (grantResults.length > 0 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                    takePicture();
                } else {
                    Toast.makeText(MainActivity.this, "Permission Denied!", Toast.LENGTH_SHORT).show();
                }
        }
    }

    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        if (requestCode == PHOTO_REQUEST && resultCode == RESULT_OK) {
            launchMediaScanIntent();
            try {
                Bitmap bitmap = decodeBitmapUri(this, imageUri);
                if (detector.isOperational() && bitmap != null) {
                    Frame frame = new Frame.Builder().setBitmap(bitmap).build();
                    SparseArray textBlocks = detector.detect(frame);
                    String blocks = "";
                    String lines = "";
                    String words = "";
                    for (int index = 0; index < textBlocks.size(); index++) {
                        //extract scanned text blocks here
                        TextBlock tBlock = textBlocks.valueAt(index);
                        blocks = blocks + tBlock.getValue() + "\n" + "\n";
                        for (Text line : tBlock.getComponents()) {
                            //extract scanned text lines here
                            lines = lines + line.getValue() + "\n";
                            for (Text element : line.getComponents()) {
                                //extract scanned text words here
                                words = words + element.getValue() + ", ";
                            }
                        }
                    }
                    if (textBlocks.size() == 0) {
                        scanResults.setText("Scan Failed: Found nothing to scan");
                    } else {
                        scanResults.setText(scanResults.getText() + "Blocks: " + "\n");
                        scanResults.setText(scanResults.getText() + blocks + "\n");
                        scanResults.setText(scanResults.getText() + "---------" + "\n");
                        scanResults.setText(scanResults.getText() + "Lines: " + "\n");
                        scanResults.setText(scanResults.getText() + lines + "\n");
                        scanResults.setText(scanResults.getText() + "---------" + "\n");
                        scanResults.setText(scanResults.getText() + "Words: " + "\n");
                        scanResults.setText(scanResults.getText() + words + "\n");
                        scanResults.setText(scanResults.getText() + "---------" + "\n");
                    }
                } else {
                    scanResults.setText("Could not set up the detector!");
                }
            } catch (Exception e) {
                Toast.makeText(this, "Failed to load Image", Toast.LENGTH_SHORT)
                        .show();
                Log.e(LOG_TAG, e.toString());
            }
        }
    }

    private void takePicture() {
        Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
        File photo = new File(Environment.getExternalStorageDirectory(), "picture.jpg");
        imageUri = FileProvider.getUriForFile(MainActivity.this,
                BuildConfig.APPLICATION_ID + ".provider", photo);
        intent.putExtra(MediaStore.EXTRA_OUTPUT, imageUri);
        startActivityForResult(intent, PHOTO_REQUEST);
    }

    @Override
    protected void onSaveInstanceState(Bundle outState) {
        if (imageUri != null) {
            outState.putString(SAVED_INSTANCE_URI, imageUri.toString());
            outState.putString(SAVED_INSTANCE_RESULT, scanResults.getText().toString());
        }
        super.onSaveInstanceState(outState);
    }

    private void launchMediaScanIntent() {
        Intent mediaScanIntent = new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE);
        mediaScanIntent.setData(imageUri);
        this.sendBroadcast(mediaScanIntent);
    }

    private Bitmap decodeBitmapUri(Context ctx, Uri uri) throws FileNotFoundException {
        int targetW = 600;
        int targetH = 600;
        BitmapFactory.Options bmOptions = new BitmapFactory.Options();
        bmOptions.inJustDecodeBounds = true;
        BitmapFactory.decodeStream(ctx.getContentResolver().openInputStream(uri), null, bmOptions);
        int photoW = bmOptions.outWidth;
        int photoH = bmOptions.outHeight;

        int scaleFactor = Math.min(photoW / targetW, photoH / targetH);
        bmOptions.inJustDecodeBounds = false;
        bmOptions.inSampleSize = scaleFactor;

        return BitmapFactory.decodeStream(ctx.getContentResolver()
                .openInputStream(uri), null, bmOptions);
    }
}

In the above piece of code, I simply initialized a TextRecognizer and asked the user to grant the permission to store the captured image on disk. Post which, when an image like shown below is captured, we resize the image in method decodeBitmapUri to a smaller size so that, it can be scanned faster. Once the image is scaled, we check for operational TextRecognizer. Then after the detector is operational, we scan out the text from picture.

When the Android OCR library – the Mobile Vision, returned the text from the picture above, it was very accurate. Have a look at the result below:

For full source code, please refer to the link below:

Full Source Code

Hence I believe this is one of the best OCR libraries available for Android till date. As it gives the unique capability of offline text scanning, without compromising on quality. Therefore if you have to make an app where it is required to scan the text and process it, use the Text API of Mobile Vision for Optical Character Recognition. For more posts like this, please connect with us on Twitter, Facebook and Google+. Hope this helped.

Mohit Gupt

www.truiton.com

The post Optical Character Recognition on Android – OCR appeared first on Truiton.