In August 2015, Google announced the release of Android Mobile Vision API. At that time this API had mainly three components Face Detection, Barcode scanner and ways to capture the tracking of objects in real time. But later on some bugs were found in its implementation, due to which the access to this API was restricted by Google to the existing users only. Then recently in June 2016 these issues were resolved and Mobile Vision APIs were re-launched. But this time it had a few new features like OCR and scanning of Aztec barcodes. In the next sections lets understand what exactly is the new Mobile Vision API and how can we use it to build an efficient code base.

What is Android Mobile Vision API ?

Ever wondered how to detect a face, a QR code or a Bar code on an Android Device? If yes; you might have heard of, or used the FaceDetector.Face API of the Android framework, or the OpenCV SDK, or maybe you had opted for a cloud based solution like Cloud Vision API, which makes the requests to a web server, fetching the results for scans. But this new Android Mobile Vision API does not make any requests to a web server. Instead it performs real time image/video scanning on the device itself. Although this may sound a little in-effective, but its not. The Mobile Vision API is very efficient and deeply integrated in to the Android system by the means of Google Play Services SDK. This gives this API an added advantage over all other solutions, as being a developer you do not need to integrate any third party SDK to perform media analysis. All you may need to do is integrate the Google Play Services properly and start building on it. Android Mobile Vision API as of now performs three types of image/stream detection as shown in next sections.

Setting up Android Mobile Vision Library

When speaking of this new multipurpose Android offline image recognition library, the only downside is that one has to set it up before the actual usage. Although it does not mean any manual configuration but a tag needs to be added in the application manifest. So that the installer knows that an extra package is needed to be downloaded as the app is being installed. As of now the mobile vision API has three components, therefore all three dependencies can be included in a single manifest tag under the application tag as shown:

<meta-data
            android:name="com.google.android.gms.vision.DEPENDENCIES"
            android:value="barcode,face,ocr"/>

This would instruct the OS to download the packages for all three types of offline image analysis. Next lets have a brief understanding of all the features of this powerful Mobile Vision API.

1. Barcode and QR Code scanning on Android

The Barcode Scanner API is used to scan various types of bar codes and qr codes. Some of them are:

• 1D barcodes: EAN-13, EAN-8, UPC-A, UPC-E, Code-39, Code-93, Code-128, ITF, Codabar
• 2D barcodes: QR Code, Data Matrix, PDF-417, AZTEC

Using this Mobile Vision API for Bar codes is very simple, even with so many supported formats. All you need to do is write around 10 lines of code to parse a Barcode or QR Code. One of the most interesting things about this API is that, it also parses the type of barcode/QR code it is scanning. All the information about the scanned barcode or QR Code type is found in the valueFormat field of Barcode Object besides the scanned barcode. To learn more about the usage of Barcode Scanner API of the Android Mobile Vision API please refer to this tutorial:

Android Barcode Tutorial

2. Optical Character Recognition (OCR) on Android

Mobile Vision Text API is another great feature packed into this library. Its purpose is straight forward, i.e. text recognition in an image or a stream of frames. It can recognize a majority of Latin character based languages like English, French, Italian Dutch etc. Therefore if your app needs to extract text out of an image, this is the perfect solution for you. As once its dependencies are downloaded, you no more required to have an internet connection, all the character recognition takes place on device itself.

Incase you wonder; how would this Android Mobile Vision Text API deliver large results, as when scanning a page of a book- a lot of words could be scanned. But thankfully this API returns text in a structured way. It mainly divides text into three sections:

1. Block: Top level structure, contains all the paragraphs.
2. Line: Mid level structure, contains lines in a block.
3. Word: Low level structure, contains a single word from line.

To learn more about the Text API, please refer to this tutorial:

Android OCR Tutorial

3. Face Detection on Android

This is the most powerful API in all, as it has human Face Detection capabilities. It is a perfectly suited API for any face filter or camera app, as it performs the analysis on the device itself once the package is downloaded. Interestingly it not only recognizes a face but can also extract the facial features of that face, including eyes nose and mouth etc. Since this API as of now does not support face recognition it cannot identify similarity between two faces, but can still classify the features like, if the eyes are open or not. The functionalities that Mobile Vision Face Detection API supports are:

1. Landmark Detection: Face API understands the human face in terms of landmarks. When a face is scanned via this API, it identifies that face via landmarks. In simple terms face landmarks are: nose, mouth, left eye, and right eye etc. By using this API you can actually extract the position of all these landmarks. 
2. Classification: This API not only scans a face but can also apply some basic logic and identify certain characteristics on the scanned face. For ex. with this feature we can find out whether the face has its eyes open or not. Also we can find out the probability of a smile on that face. 
3. Tracking: This is the most interesting feature of this API, you can actually track a face in a video sequence through this API. This feature of Mobile Vision API can be used to identify and track a face in a video. Once again this is not an application of face recognition, instead it tracks the face through movement of that particular face in the video. 

To learn more in detail about Face Detection, please refer to this tutorial:

Face Detection Tutorial 

Tracking Faces, Bar codes and QR Codes Simultaneously

Interestingly, if you want to scan and track multiple faces simultaneously, even that is possible through this API. All you need to do is initialize a MultiProcessor and track multiple faces through this API. Also if you want to track multiple Bar codes or QR codes, same thing is applicable, as all of this is a part of Mobile Vision API, backed by Google Play Services. But even more interesting thing is that you can track Bar codes, QR codes and Faces in a single frame, by using the MultiDetector class of the same API suite. This makes the mobile vision APIs an outstanding leader in terms of features when compared to any other such library in the market. As its light, easy to integrate and very easy to use. For more updates, please connect with us on Twitter, Facebook and Google+.

Mohit Gupt

Born in New Delhi, India. A software engineer by profession, an android enthusiast and mobile development evangelist. My motive here is to create a group of skilled engineers, who can build better software. Reason being programming is my passion, and also it feels good to make a device do something you want. Professionally I have worked with many software engineering and product development firms. As of now too, I am employed as a senior engineer in a leading tech company. In total I may have worked on more than 20 projects professionally, but whenever I get spare time I share my thoughts here at Truiton.

www.truiton.com

The post Introducing Android Mobile Vision API appeared first on Truiton.

Android itself is a smart OS, still it lacked a very basic feature of text recognition. But not anymore; with the official Optical Character Recognition API of Android and the Mobile Vision library, now Android can perform OCR very efficiently and correctly. I did a very basic feature test to have a look at the new functionality and found out its very fast and easy to use. Here in this Optical Character Recognition(OCR) example of Android, I would simply import the library, click a picture of a piece of text and look for text blocks in it. But before doing so lets take an overview of Android Mobile Vision API to understand the working of Text API better.

Android Mobile Vision Library

Consider a scenario where you wish to scan and an image or a video stream and detect faces, barcodes, QR codes or Text in it on Android. Astonishingly till now no such framework existed on Android. But now Google has introduced Mobile Vision APIs. These set of APIs provide a very easy to use programming interfaces through which we can scan Faces, Barcodes, QR Codes and Text without writing huge amount of code. And the best part is that these can be used offline as well, i.e. only once when the app is downloaded the required dependencies are downloaded, post that an internet connection is not required any more. As this feature is introduced on Android through the Google Play services. To enable your app to use Mobile Vision APIs, you need to add this dependency in your build.gradle file:

compile 'com.google.android.gms:play-services-vision:11.4.0'

Please Note: Learn more on how to setup Google Play Services. Or learn more about Mobile Vision API.

Introducing an Android OCR Library – Text Recognition API

Since the Android OS was brought on to production devices, Optical Character Recognition has been a common area of research. But this Text Recognition API of Mobile Vision suite would bring all these researches to a stop. As this Google powered API contains features like multiple language recognition where languages are like : English, French, German, Spanish or any other Latin based text. Also the text can be parsed from a stream of frames i.e. a video and displayed on the screen in real time as displayed in the image above. But due to the scope of this Android OCR Library example we would keep things simple and scan the text from an image only, as this tutorial is targeted for beginners. Apart from this the interesting part is, all this is done offline by the Google Play services itself, i.e. no internet connection is required after once it has been set up in the app (shown in steps ahead).  Now when it comes to structuring the text, this Android OCR library not only recognizes the text but can also divide the captured text into the following categories:

1. Block – TextBlock – A top level object where a scanned paragraph or column is captured.
2. Line – Line – A line of text captured from a block of text.
3. Word – Element – A single word recognized in a Line.

Android OCR Example

Now that we have a basic understanding of Android OCR library, particularly the Text Recognition API. I will demonstrate by an example where we would simply take a picture and scan for text in it. Now as a thumb rule to do so first we may need to set up our Android app to download the play services dependency for Optical Character Recognition. Therefore please include the block of code below in your manifest to instruct installer to download the OCR dependency at the time of installing the app.

<meta-data
            android:name="com.google.android.gms.vision.DEPENDENCIES"
            android:value="ocr"/>

Please Note: This is not a mandatory step, but helps in downloading the dependencies beforehand. Also the link to full source code is at the end of this tutorial.

Next lets define a layout to display the scanned results:

<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout
    android:id="@+id/activity_main"
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context="com.truiton.mobile.vision.ocr.MainActivity">

    <ImageView
        android:id="@+id/imageView"
        android:layout_width="200dp"
        android:layout_height="200dp"
        android:layout_marginTop="16dp"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent"
        app:srcCompat="@mipmap/truiton"
        tools:layout_constraintLeft_creator="1"
        tools:layout_constraintRight_creator="1"/>

    <Button
        android:id="@+id/button"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginBottom="8dp"
        android:text="Scan Text"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        tools:layout_constraintLeft_creator="1"
        tools:layout_constraintRight_creator="1"
        />

    <TextView
        android:id="@+id/textView"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginTop="40dp"
        android:text="Scan Results:"
        android:textAllCaps="false"
        android:textStyle="normal|bold"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/imageView"
        tools:layout_constraintLeft_creator="1"
        tools:layout_constraintRight_creator="1"/>

    <ScrollView
        android:layout_width="0dp"
        android:layout_height="0dp"
        android:layout_marginTop="8dp"
        android:paddingLeft="5dp"
        android:paddingRight="5dp"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintHorizontal_bias="1.0"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/textView"
        tools:layout_constraintTop_creator="1"
        tools:layout_constraintRight_creator="1"
        tools:layout_constraintBottom_creator="1"
        tools:layout_constraintLeft_creator="1">

        <LinearLayout
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:orientation="vertical">

            <TextView
                android:id="@+id/results"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:layout_marginTop="8dp"
                app:layout_constraintLeft_toLeftOf="parent"
                app:layout_constraintRight_toRightOf="parent"
                tools:layout_constraintLeft_creator="1"
                tools:layout_constraintRight_creator="1"
                tools:layout_constraintTop_creator="1"/>
        </LinearLayout>
    </ScrollView>
</android.support.constraint.ConstraintLayout>

I was playing around with ConstraintLayout in Android. Hence made it in Constraint layout, although its not a requirement to use ConstraintLayout for this Android OCR Library example. The above layout basically contains a ScrollView to accurately display the scanned text. Next lets define the main Activity:

package com.truiton.mobile.vision.ocr;

import android.Manifest;
import android.content.Context;
import android.content.Intent;
import android.content.pm.PackageManager;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.provider.MediaStore;
import android.support.annotation.NonNull;
import android.support.v4.app.ActivityCompat;
import android.support.v4.content.FileProvider;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.util.SparseArray;
import android.view.View;
import android.widget.Button;
import android.widget.TextView;
import android.widget.Toast;

import com.google.android.gms.vision.Frame;
import com.google.android.gms.vision.text.Text;
import com.google.android.gms.vision.text.TextBlock;
import com.google.android.gms.vision.text.TextRecognizer;

import java.io.File;
import java.io.FileNotFoundException;

public class MainActivity extends AppCompatActivity {
    private static final String LOG_TAG = "Text API";
    private static final int PHOTO_REQUEST = 10;
    private TextView scanResults;
    private Uri imageUri;
    private TextRecognizer detector;
    private static final int REQUEST_WRITE_PERMISSION = 20;
    private static final String SAVED_INSTANCE_URI = "uri";
    private static final String SAVED_INSTANCE_RESULT = "result";

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Button button = (Button) findViewById(R.id.button);
        scanResults = (TextView) findViewById(R.id.results);
        if (savedInstanceState != null) {
            imageUri = Uri.parse(savedInstanceState.getString(SAVED_INSTANCE_URI));
            scanResults.setText(savedInstanceState.getString(SAVED_INSTANCE_RESULT));
        }
        detector = new TextRecognizer.Builder(getApplicationContext()).build();
        button.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view) {
                ActivityCompat.requestPermissions(MainActivity.this, new
                        String[]{Manifest.permission.WRITE_EXTERNAL_STORAGE}, REQUEST_WRITE_PERMISSION);
            }
        });
    }

    @Override
    public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults);
        switch (requestCode) {
            case REQUEST_WRITE_PERMISSION:
                if (grantResults.length > 0 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                    takePicture();
                } else {
                    Toast.makeText(MainActivity.this, "Permission Denied!", Toast.LENGTH_SHORT).show();
                }
        }
    }

    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        if (requestCode == PHOTO_REQUEST && resultCode == RESULT_OK) {
            launchMediaScanIntent();
            try {
                Bitmap bitmap = decodeBitmapUri(this, imageUri);
                if (detector.isOperational() && bitmap != null) {
                    Frame frame = new Frame.Builder().setBitmap(bitmap).build();
                    SparseArray<TextBlock> textBlocks = detector.detect(frame);
                    String blocks = "";
                    String lines = "";
                    String words = "";
                    for (int index = 0; index < textBlocks.size(); index++) {
                        //extract scanned text blocks here
                        TextBlock tBlock = textBlocks.valueAt(index);
                        blocks = blocks + tBlock.getValue() + "\n" + "\n";
                        for (Text line : tBlock.getComponents()) {
                            //extract scanned text lines here
                            lines = lines + line.getValue() + "\n";
                            for (Text element : line.getComponents()) {
                                //extract scanned text words here
                                words = words + element.getValue() + ", ";
                            }
                        }
                    }
                    if (textBlocks.size() == 0) {
                        scanResults.setText("Scan Failed: Found nothing to scan");
                    } else {
                        scanResults.setText(scanResults.getText() + "Blocks: " + "\n");
                        scanResults.setText(scanResults.getText() + blocks + "\n");
                        scanResults.setText(scanResults.getText() + "---------" + "\n");
                        scanResults.setText(scanResults.getText() + "Lines: " + "\n");
                        scanResults.setText(scanResults.getText() + lines + "\n");
                        scanResults.setText(scanResults.getText() + "---------" + "\n");
                        scanResults.setText(scanResults.getText() + "Words: " + "\n");
                        scanResults.setText(scanResults.getText() + words + "\n");
                        scanResults.setText(scanResults.getText() + "---------" + "\n");
                    }
                } else {
                    scanResults.setText("Could not set up the detector!");
                }
            } catch (Exception e) {
                Toast.makeText(this, "Failed to load Image", Toast.LENGTH_SHORT)
                        .show();
                Log.e(LOG_TAG, e.toString());
            }
        }
    }

    private void takePicture() {
        Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
        File photo = new File(Environment.getExternalStorageDirectory(), "picture.jpg");
        imageUri = FileProvider.getUriForFile(MainActivity.this,
                BuildConfig.APPLICATION_ID + ".provider", photo);
        intent.putExtra(MediaStore.EXTRA_OUTPUT, imageUri);
        startActivityForResult(intent, PHOTO_REQUEST);
    }

    @Override
    protected void onSaveInstanceState(Bundle outState) {
        if (imageUri != null) {
            outState.putString(SAVED_INSTANCE_URI, imageUri.toString());
            outState.putString(SAVED_INSTANCE_RESULT, scanResults.getText().toString());
        }
        super.onSaveInstanceState(outState);
    }

    private void launchMediaScanIntent() {
        Intent mediaScanIntent = new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE);
        mediaScanIntent.setData(imageUri);
        this.sendBroadcast(mediaScanIntent);
    }

    private Bitmap decodeBitmapUri(Context ctx, Uri uri) throws FileNotFoundException {
        int targetW = 600;
        int targetH = 600;
        BitmapFactory.Options bmOptions = new BitmapFactory.Options();
        bmOptions.inJustDecodeBounds = true;
        BitmapFactory.decodeStream(ctx.getContentResolver().openInputStream(uri), null, bmOptions);
        int photoW = bmOptions.outWidth;
        int photoH = bmOptions.outHeight;

        int scaleFactor = Math.min(photoW / targetW, photoH / targetH);
        bmOptions.inJustDecodeBounds = false;
        bmOptions.inSampleSize = scaleFactor;

        return BitmapFactory.decodeStream(ctx.getContentResolver()
                .openInputStream(uri), null, bmOptions);
    }
}

In the above piece of code, I simply initialized a TextRecognizer and asked the user to grant the permission to store the captured image on disk. Post which, when an image like shown below is captured, we resize the image in method decodeBitmapUri to a smaller size so that, it can be scanned faster. Once the image is scaled, we check for operational TextRecognizer. Then after the detector is operational, we scan out the text from picture.

When the Android OCR library – the Mobile Vision, returned the text from the picture above, it was very accurate. Have a look at the result below:

For full source code, please refer to the link below:

Full Source Code

Hence I believe this is one of the best OCR libraries available for Android till date. As it gives the unique capability of offline text scanning, without compromising on quality. Therefore if you have to make an app where it is required to scan the text and process it, use the Text API of Mobile Vision for Optical Character Recognition. For more posts like this, please connect with us on Twitter, Facebook and Google+. Hope this helped.

Mohit Gupt

Born in New Delhi, India. A software engineer by profession, an android enthusiast and mobile development evangelist. My motive here is to create a group of skilled engineers, who can build better software. Reason being programming is my passion, and also it feels good to make a device do something you want. Professionally I have worked with many software engineering and product development firms. As of now too, I am employed as a senior engineer in a leading tech company. In total I may have worked on more than 20 projects professionally, but whenever I get spare time I share my thoughts here at Truiton.

www.truiton.com

The post Optical Character Recognition on Android – OCR appeared first on Truiton.

Often when building Android apps, we encounter situations where it is required to scan a bar code or QR code. Now as difficult as it sounds, the complex code it has if you plan to write your own scanning algorithm. But this is the place where new and improved Mobile Vision from Google steps in. This API was initially introduced in August 2015, but now it as been further improved and re-introduced including features like OCR and object tracking along with the existing features of scanning qr code and barcodes. Therefore here lets understand the usage of Google’s Mobile Vision with an example where we scan a QR code on Android.

Programmatically scanning a QR Code or Bar Code on Android has always been a topic of great discussion. There are many ways through which this can be done:

• Using a web based API solution, where image is uploaded to a server and it returns back the results.
• Use a hybrid solution where web based app accesses your camera and scans Bar Code or QR Code.
• Write your own image scanning algorithm.
• Use the Google Play Services integrated Mobile Vision.

In my opinion using the Android Mobile Vision for scanning QR codes is the most favorable option in comparison to other options because it offers offline scanning of images, resulting a fast scan and an immediate result. This being said its also the easiest option available. As it just requires a few lines of code to make it work. Besides that; when using this library, you don’t need to import any other android scanning library as this API also takes care of OCR and facial scanning. To read more about it please visit my article on Android Mobile Vision API (coming soon).

Introducing the Official Android QR and Barcode Scanning Library : Mobile Vision

There are many third party libraries available for parsing of barcodes and QR codes, some of them are even endorsed by Google I believe. But nowdays reading of QR codes and barcodes on Android has become so common that, it demanded an official solution. Hence the Mobile Vision APIs were introduced with QR code and barcode scanning capabilities for Android. While most of the opensource libraries scan only one QR code or barcode at a time. These Mobile Vision APIs can scan multiple QR codes at the same time. Also if you want, you can directly integrate them with android’s camera and scan for barcodes and QR codes on the fly while streaming on a surface.

As of now Mobile Vision APIs can scan following types of Bar codes and QR codes on Android:

• 1D barcodes: EAN-13, EAN-8, UPC-A, UPC-E, Code-39, Code-93, Code-128, ITF, Codabar
• 2D barcodes: QR Code, Data Matrix, PDF-417, AZTEC

Setting up Android QR Code Scan Library : Mobile Vision

As I have mentioned before, setting up Mobile Vision API is very easy, all of its downloads are managed internally by Google Play services. Although to properly integrate it and make it functional before the actual usage, it is advised to reference the used features of Mobile Vision API in the Manifest of your app. This way the dependencies are downloaded before you can actually use them i.e. at the time of installation. Also please don’t forget to follow the steps for Google Play Services Integration. But in our case we only need to compile com.google.android.gms:play-services-vision, therefore make sure your build.gradle (app) has following dependencies:

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    testCompile 'junit:junit:4.12'
    compile 'com.android.support:appcompat-v7:26.1.0'
    compile 'com.google.android.gms:play-services-vision:11.4.0'
}

And build.gradle(global) should have this:

allprojects {
    repositories {
        jcenter()
        maven {
            url "https://maven.google.com"
        }
    }
}

Also as mentioned to make Mobile Vision APIs functional for an Android Scan QR Code Example before installation, please include these lines in your Manifest inside the application tag:

<meta-data
            android:name="com.google.android.gms.vision.DEPENDENCIES"
            android:value="barcode"/>

Here the android:value tag can have following comma separated values:

• barcode
• face
• ocr

Android Scan QR Code Example

One of the interesting features of Mobile Vision API is that it performs a scan very fast. Also its documentation states that you can perform multiple types of scans on the same frame. This means you can scan multiple barcodes, QR Codes and faces together. Be it a video, a picture, or a video stream. But covering all of this in a single article is not possible. Therefore to start off we will build a basic example where we would use native phone’s camera to take a picture and scan it for barcodes and qr codes, and building a basic Android Scan QR code Example.

To start off, have a look at the Manifest, where we define the camera and barcode dependency for this Android Scan QR Code example:

<?xml version="1.0" encoding="utf-8"?>
<manifest package="com.truiton.mobile.vision.qrcode"
          xmlns:android="http://schemas.android.com/apk/res/android">

    <uses-feature
        android:name="android.hardware.camera"
        android:required="true"/>
    <uses-permission
        android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
        <meta-data
            android:name="com.google.android.gms.vision.DEPENDENCIES"
            android:value="barcode"/>
        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>

                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
        <provider
            android:name="android.support.v4.content.FileProvider"
            android:authorities="${applicationId}.provider"
            android:exported="false"
            android:grantUriPermissions="true">
            <meta-data
                android:name="android.support.FILE_PROVIDER_PATHS"
                android:resource="@xml/provider_paths"/>
        </provider>
    </application>

</manifest>

Please Note: full source code is available at the end of the tutorial.

Next lets define a basic layout to show the QR code scan results.

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:paddingBottom="@dimen/activity_vertical_margin"
    android:paddingLeft="@dimen/activity_horizontal_margin"
    android:paddingRight="@dimen/activity_horizontal_margin"
    android:paddingTop="@dimen/activity_vertical_margin"
    tools:context="com.truiton.mobile.vision.qrcode.MainActivity">

    <ImageView
        android:id="@+id/image"
        android:layout_width="200dp"
        android:layout_height="200dp"
        android:layout_centerHorizontal="true"
        android:src="@mipmap/truiton"/>

    <TextView
        android:id="@+id/scan_header"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_below="@+id/image"
        android:layout_centerHorizontal="true"
        android:text="Scan Results:"
        android:textStyle="bold"/>

    <TextView
        android:id="@+id/scan_results"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_below="@+id/scan_header"
        android:layout_centerHorizontal="true"
        android:layout_marginTop="10dp"/>

    <Button
        android:id="@+id/button"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_alignParentBottom="true"
        android:layout_centerHorizontal="true"
        android:layout_marginBottom="10dp"
        android:layout_marginTop="20dp"
        android:text="Take Picture"/>

</RelativeLayout>

Now lets define the main activity where the Android QR/Bar code scanning takes place:

package com.truiton.mobile.vision.qrcode;

import android.Manifest;
import android.content.Context;
import android.content.Intent;
import android.content.pm.PackageManager;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.provider.MediaStore;
import android.support.annotation.NonNull;
import android.support.v4.app.ActivityCompat;
import android.support.v4.content.FileProvider;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.util.SparseArray;
import android.view.View;
import android.widget.Button;
import android.widget.TextView;
import android.widget.Toast;

import com.google.android.gms.vision.Frame;
import com.google.android.gms.vision.barcode.Barcode;
import com.google.android.gms.vision.barcode.BarcodeDetector;

import java.io.File;
import java.io.FileNotFoundException;

public class MainActivity extends AppCompatActivity {
    private static final String LOG_TAG = "Barcode Scanner API";
    private static final int PHOTO_REQUEST = 10;
    private TextView scanResults;
    private BarcodeDetector detector;
    private Uri imageUri;
    private static final int REQUEST_WRITE_PERMISSION = 20;
    private static final String SAVED_INSTANCE_URI = "uri";
    private static final String SAVED_INSTANCE_RESULT = "result";

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Button button = (Button) findViewById(R.id.button);
        scanResults = (TextView) findViewById(R.id.scan_results);
        if (savedInstanceState != null) {
            imageUri = Uri.parse(savedInstanceState.getString(SAVED_INSTANCE_URI));
            scanResults.setText(savedInstanceState.getString(SAVED_INSTANCE_RESULT));
        }
        button.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view) {
                ActivityCompat.requestPermissions(MainActivity.this, new
                        String[]{Manifest.permission.WRITE_EXTERNAL_STORAGE}, REQUEST_WRITE_PERMISSION);
            }
        });

        detector = new BarcodeDetector.Builder(getApplicationContext())
                .setBarcodeFormats(Barcode.DATA_MATRIX | Barcode.QR_CODE)
                .build();
        if (!detector.isOperational()) {
            scanResults.setText("Could not set up the detector!");
            return;
        }
    }

    @Override
    public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults);
        switch (requestCode) {
            case REQUEST_WRITE_PERMISSION:
                if (grantResults.length > 0 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                    takePicture();
                } else {
                    Toast.makeText(MainActivity.this, "Permission Denied!", Toast.LENGTH_SHORT).show();
                }
        }
    }

    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        if (requestCode == PHOTO_REQUEST && resultCode == RESULT_OK) {
            launchMediaScanIntent();
            try {
                Bitmap bitmap = decodeBitmapUri(this, imageUri);
                if (detector.isOperational() && bitmap != null) {
                    Frame frame = new Frame.Builder().setBitmap(bitmap).build();
                    SparseArray<Barcode> barcodes = detector.detect(frame);
                    for (int index = 0; index < barcodes.size(); index++) {
                        Barcode code = barcodes.valueAt(index);
                        scanResults.setText(scanResults.getText() + code.displayValue + "\n");

                        //Required only if you need to extract the type of barcode
                        int type = barcodes.valueAt(index).valueFormat;
                        switch (type) {
                            case Barcode.CONTACT_INFO:
                                Log.i(LOG_TAG, code.contactInfo.title);
                                break;
                            case Barcode.EMAIL:
                                Log.i(LOG_TAG, code.email.address);
                                break;
                            case Barcode.ISBN:
                                Log.i(LOG_TAG, code.rawValue);
                                break;
                            case Barcode.PHONE:
                                Log.i(LOG_TAG, code.phone.number);
                                break;
                            case Barcode.PRODUCT:
                                Log.i(LOG_TAG, code.rawValue);
                                break;
                            case Barcode.SMS:
                                Log.i(LOG_TAG, code.sms.message);
                                break;
                            case Barcode.TEXT:
                                Log.i(LOG_TAG, code.rawValue);
                                break;
                            case Barcode.URL:
                                Log.i(LOG_TAG, "url: " + code.url.url);
                                break;
                            case Barcode.WIFI:
                                Log.i(LOG_TAG, code.wifi.ssid);
                                break;
                            case Barcode.GEO:
                                Log.i(LOG_TAG, code.geoPoint.lat + ":" + code.geoPoint.lng);
                                break;
                            case Barcode.CALENDAR_EVENT:
                                Log.i(LOG_TAG, code.calendarEvent.description);
                                break;
                            case Barcode.DRIVER_LICENSE:
                                Log.i(LOG_TAG, code.driverLicense.licenseNumber);
                                break;
                            default:
                                Log.i(LOG_TAG, code.rawValue);
                                break;
                        }
                    }
                    if (barcodes.size() == 0) {
                        scanResults.setText("Scan Failed: Found nothing to scan");
                    }
                } else {
                    scanResults.setText("Could not set up the detector!");
                }
            } catch (Exception e) {
                Toast.makeText(this, "Failed to load Image", Toast.LENGTH_SHORT)
                        .show();
                Log.e(LOG_TAG, e.toString());
            }
        }
    }

    private void takePicture() {
        Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
        File photo = new File(Environment.getExternalStorageDirectory(), "picture.jpg");
        imageUri = FileProvider.getUriForFile(MainActivity.this,
                BuildConfig.APPLICATION_ID + ".provider", photo);
        intent.putExtra(MediaStore.EXTRA_OUTPUT, imageUri);
        startActivityForResult(intent, PHOTO_REQUEST);
    }

    @Override
    protected void onSaveInstanceState(Bundle outState) {
        if (imageUri != null) {
            outState.putString(SAVED_INSTANCE_URI, imageUri.toString());
            outState.putString(SAVED_INSTANCE_RESULT, scanResults.getText().toString());
        }
        super.onSaveInstanceState(outState);
    }

    private void launchMediaScanIntent() {
        Intent mediaScanIntent = new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE);
        mediaScanIntent.setData(imageUri);
        this.sendBroadcast(mediaScanIntent);
    }

    private Bitmap decodeBitmapUri(Context ctx, Uri uri) throws FileNotFoundException {
        int targetW = 600;
        int targetH = 600;
        BitmapFactory.Options bmOptions = new BitmapFactory.Options();
        bmOptions.inJustDecodeBounds = true;
        BitmapFactory.decodeStream(ctx.getContentResolver().openInputStream(uri), null, bmOptions);
        int photoW = bmOptions.outWidth;
        int photoH = bmOptions.outHeight;

        int scaleFactor = Math.min(photoW / targetW, photoH / targetH);
        bmOptions.inJustDecodeBounds = false;
        bmOptions.inSampleSize = scaleFactor;

        return BitmapFactory.decodeStream(ctx.getContentResolver()
                .openInputStream(uri), null, bmOptions);
    }
}

Please Note: Images to be scanned, should not be too big.

When it comes to scanning of QR code from a picture on an Android app using the Mobile Vision APIs, the process is quite simple. All we need to do is, pass on a Frame to the BarcodeDetector, this will return an array of type Barcode, which would contain the actual barcode value. But there are prerequisites to this implementation, like declaring all permissions and getting them on run time, which is shown in the android scan qr code example above. To view full source code for this example please refer to the link below:

Full Source Code

Also please have a look at the screens for this Android Scan QR code scan example:

By using these Mobile Vision APIs for barcodes and QR codes, we can also determine the type of barcode or QR code is being scanned. This is one of the most interesting parts about this API, it reads the data, as well as it understands what type of data it is. For e.g. when an event QR code is scanned, we can use Barcode.CalendarEvent class to extract out the exact data in required format. This can help us in reducing our code bases very efficiently.

When speaking of Android Barcode scanning libraries, Mobile Vision API is the latest addition in the list. Also it is one of the most stable and reliable options as its a library provided by Google, through play services. Which definitely gives it an added advantage over other libraries. That being said, would like to say, we have finally succeeded in offline scanning of barcodes and QR Codes. As most of the methods require uploading of images to a server, or including huge code bases or libraries into our project for offline scanning. But with Mobile Vision APIs nothing of this sort needs to be done, as all this is taken care by Google Play Services once they are downloaded. Hope this helped you. Connect with us on Twitter, Facebook and Google+ for more updates.

Mohit Gupt

Born in New Delhi, India. A software engineer by profession, an android enthusiast and mobile development evangelist. My motive here is to create a group of skilled engineers, who can build better software. Reason being programming is my passion, and also it feels good to make a device do something you want. Professionally I have worked with many software engineering and product development firms. As of now too, I am employed as a senior engineer in a leading tech company. In total I may have worked on more than 20 projects professionally, but whenever I get spare time I share my thoughts here at Truiton.

www.truiton.com

The post Android Example – Programmatically Scan QR Code and Bar Code appeared first on Truiton.

Google introduced a new API called the Google Cloud Vision API. This API has full potential to understand the contents of an image by using the Google’s machine learning platform and TensorFlow. It allows the API to process individual pieces of an image separately and return the results really fast in a unified format. Therefore in simple terms now you can submit an image to the Google Cloud Vision API and find out what’s in the image. What’s more; is that, when making a request to process an image; Google gives us the capability to specify the types of analysis that should be performed on this image. For example, a simple object identification, landmark detection, facial detection, sentimental analysis and many more such analysis could be performed on an image. But what’s even more great is that, this API can also be integrated directly into the Android apps. Making Android image recognition very simple. Also this is another way, Google lets us perform image recognition on Android, powered by Google cloud platform. Imagine the power an end user can have through this API.

The new Google Cloud Vision API is a multi-platform solution for image recognition, weather its an Android app, iOS app or cloud storage, this API is available for image analysis. As this Cloud Vision API has SDK support for Java, Go Lang, Node.js, Python, and more importantly JSON format. But here in this article we would mainly discuss on how to do image recognition on android. Therefore continuing on Java/Android, you might be aware that, there are many ways to perform android image recognition like OpenCV, OCR reading libraries and facial recognition APIs. But none of them are as accurate and light weight as this new Android Cloud Vision API as they require a huge amount of data to be present on the app beforehand to perform object matching. Which in turn increases the APK size and still result in in-accurate data. Also to use this new Android Cloud Vision API and perform all sorts of image analysis; no heavy gradle or project files are required to be included in your project. Just the basic google-api-client-android, google-http-client-gson and google-api-services-vision dependencies are required in your gradle for models. Read more about it in the next section:

Enabling yourself to use Google Cloud Vision API on Android

To perform Android image recognition using the Google Cloud Vision API, we must first enable it from the Google Cloud Developer Console, please follow the steps:

1. Create a project in Google Cloud Console or use an existing one.
2. Enable Billing for the project. if you have a new or un-used account you can start a free trial (It might ask your credit card info to validate your identity but not charge you).
3. Enable the Google Cloud Vision API using this link. OR
   • Navigate to “API Manager” section from the hamburger menu.
   • Search and select “Google Cloud Vision API”.
   • Enable it.
4. Then go to the credentials section from the side menu.
5. Click on credentials drop down menu and select OAuth Client ID.
   • Select Application Type as Android.
   • Add a suitable name like Android client for Cloud Vision API
   • Enter your SHA1 fingerprint in the desired format. Using the mentioned command on screen or use this SHA1 fingerprint tutorial to get your fingerprint.
   • Enter the package name for your app, can be located in the defaultConfig block of your gradle.
   • Click on create.
6. That’s it, you’re done.

If you are planning to access Google Cloud Vision API through some other platform you may need to create the credentials in a different manner. As when accessing this API on Android to perform image analysis, we need the end user’s consent that we are accessing their pictures. Although if you are planning to perform image recognition in background, like a server to server interaction. Then things should be handled differently. We may not need to generate an OAuth ID, instead a Service Account key may be required. But lets not discuss that here; as here in this tutorial our main focus is to perform; Android image recognition. Therefore before moving on to next section lets add the following dependencies in your app gradle, although at the end of this example full source code is available:

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    testCompile 'junit:junit:4.12'
    compile 'com.android.support:appcompat-v7:23.4.0'
    compile 'com.google.android.gms:play-services-base:9.0.2'
    compile 'com.google.android.gms:play-services-auth:9.0.2'
    compile 'com.google.apis:google-api-services-vision:v1-rev16-1.22.0'
    compile ('com.google.api-client:google-api-client-android:1.22.0') {
        exclude module: 'httpclient'
    }
    compile ('com.google.http-client:google-http-client-gson:1.20.0') {
        exclude module: 'httpclient'
    }
}

Supported Image Analysis Techniques in Cloud Vision API on Android

In general Google Cloud Vision API support many types of image analysis techniques. Be it optical character recognition, landmark detection or a simple logo detection. Cloud vision API does it very accurately. Also the best part about this API is that; its cross platform and is available via API access, which results in very light weight applications. Also since this technology is comparatively new, there is a lot of scope for improvement. And while that’s happening behind the scenes we don’t need to worry about the changing code-base as basically its just an API call. Moving on lets have a look at the supported Android image recognition techniques.

1. LABEL_DETECTION

One of the most basic types of techniques available in Android Cloud Vision API is the label detection. This allows the API to analyse image content and list out all the items contained in it individually. Have a look at the image I uploaded:

2. TEXT_DETECTION

Another interesting type of image analysis technique in Google Cloud Vision API is this text detection feature. It allows us to extract out the text from an image. My personal opinion is that, this feature alone could turn out to be the torch bearer for whole Cloud Vision API suite.

3. LANDMARK_DETECTION

Google has gathered so much of data that now they can even identify a landmark, by just scanning through its picture. In a way it might just sound a little scary, but this image recognition technique is equally powerful and useful. Imagine how useful it could be in adding caption on your vacation images automatically.

4. LOGO_DETECTION

By using the new Cloud Vision API you can also identify some of the popular logos. Although when I tried this API with some logos, it wasn’t able to identify most of them. But, maybe in future this will get improved.

5. FACE_DETECTION

This is also one the very powerful features of the Google Cloud Vision API. This allows us to mark the number of faces in a picture. It also helps in identifying the placement of individual facial features in an image. By using this image recognition technique on Android we can highlight a face with a polygon very easily.

6. SAFE_SEARCH_DETECTION

This technique allows us to detect inappropriate images. This could majorly help in moderation of images when performing a server to server integration. Currently it supports four types of annotations adult, spoof, medical and violence.

7. IMAGE_PROPERTIES

Android Cloud Vision API also provides a way to identify the dominant colors in an image. In a way this feature would not be very much used on Android as a great alternative in form of Palette library is already available, through which dominant colors can be identified.

Is Google Cloud Vision API Free ?

The answer is NO. To perform image recognition on android via this API, I assume an infrastructure cost is involved. Therefore just like most of the Google products, Cloud Vision API is FREE for initial use on all platforms but when usage increases, they will start charging you. Please find the detailed pricing plan here.

Android Image Recognition using Google’s Cloud Vision API

Behind the scenes Google is harnessing the power of TensorFlow and Machine Learning platforms to perform this powerful image analysis on Android. As I mentioned earlier through this Android image recognition technique, we can categorize our images in to thousands of tags. This will definitely help us in organizing our data in a better way; leading us to build better location aware apps. To build an Android image recognition app using the Cloud Vision API, hope you have enabled the API from the Cloud Console using the steps mentioned in the first section of this tutorial and included all the dependencies in your build.gradle file. Next lets move on to defining a layout where all the results would be displayed. (Although full source code is available at the end of the tutorial.)

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:paddingBottom="@dimen/activity_vertical_margin"
    android:paddingLeft="@dimen/activity_horizontal_margin"
    android:paddingRight="@dimen/activity_horizontal_margin"
    android:paddingTop="@dimen/activity_vertical_margin"
    tools:context="com.truiton.cloudvisionapi.MainActivity">

    <TextView
        android:id="@+id/selected_image_txt"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_alignParentTop="true"
        android:layout_marginTop="10dp"
        android:text="Selected Image: "/>

    <ImageView
        android:id="@+id/selected_image"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_below="@+id/selected_image_txt"
        android:layout_centerHorizontal="true"
        android:layout_marginTop="10dp"/>

    <TextView
        android:id="@+id/result"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_below="@+id/selected_image"
        android:layout_marginTop="10dp"/>

    <Button
        android:id="@+id/select_image_button"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_alignParentBottom="true"
        android:layout_centerHorizontal="true"
        android:text="Select Image"/>
</RelativeLayout>

Next lets define the GET_ACCOUNTS and INTERNET permission in the manifest. As to perform an OAuth request with Android to Could Vision API we need to access the account information on the android device.

<?xml version="1.0" encoding="utf-8"?>
<manifest package="com.truiton.cloudvisionapi"
          xmlns:android="http://schemas.android.com/apk/res/android">

    <uses-permission android:name="android.permission.GET_ACCOUNTS"/>
    <uses-permission android:name="android.permission.INTERNET"/>

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>

                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
    </application>

</manifest>

In this example to make a Google Cloud Vision API request on Android, we will be using Google API Client library for java. And as you might know, to perform a Google API Client OAuth request on Android we first need to get an auth token from google via a client call. Therefore lets first define a class to get an OAuth token.

Please Note: Don’t forget to generate a ClientID for your OAuth token in the Cloud Console, using the steps mentioned in first section.

package com.truiton.cloudvisionapi;

import android.accounts.Account;
import android.app.Activity;
import android.os.AsyncTask;

import com.google.android.gms.auth.GoogleAuthException;
import com.google.android.gms.auth.GoogleAuthUtil;
import com.google.android.gms.auth.UserRecoverableAuthException;

import java.io.IOException;

/**
 * Created by MG on 04-06-2016.
 */
public class GetTokenTask extends AsyncTask<Void, Void, Void> {
    Activity mActivity;
    String mScope;
    Account mAccount;
    int mRequestCode;

    GetTokenTask(Activity activity, Account account, String scope, int requestCode) {
        this.mActivity = activity;
        this.mScope = scope;
        this.mAccount = account;
        this.mRequestCode = requestCode;
    }

    @Override
    protected Void doInBackground(Void... params) {
        try {
            String token = fetchToken();
            if (token != null) {
                ((MainActivity)mActivity).onTokenReceived(token);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }

    /**
     * Gets an authentication token from Google and handles any
     * GoogleAuthException that may occur.
     */
    protected String fetchToken() throws IOException {
        String accessToken;
        try {
            accessToken = GoogleAuthUtil.getToken(mActivity, mAccount, mScope);
            GoogleAuthUtil.clearToken (mActivity, accessToken); // used to remove stale tokens.
            accessToken = GoogleAuthUtil.getToken(mActivity, mAccount, mScope);
            return accessToken;
        } catch (UserRecoverableAuthException userRecoverableException) {
            mActivity.startActivityForResult(userRecoverableException.getIntent(), mRequestCode);
        } catch (GoogleAuthException fatalException) {
            fatalException.printStackTrace();
        }
        return null;
    }
}

Moving on lets define the MainActivity where the Android image recognition would take place. In this example we would perform LABEL_DETECTION, TEXT_DETECTION, and LANDMARK_DETECTION. Although if you wish to perform more analysis on single image. You can add more feature requests on the same image upload. But since this Google Could Vision API is in very nascent stage, therefore as and when more features are added, it gets slower.

package com.truiton.cloudvisionapi;

import android.Manifest;
import android.accounts.Account;
import android.accounts.AccountManager;
import android.content.Intent;
import android.content.pm.PackageManager;
import android.graphics.Bitmap;
import android.net.Uri;
import android.os.AsyncTask;
import android.provider.MediaStore;
import android.support.annotation.NonNull;
import android.support.v4.app.ActivityCompat;
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.util.Log;
import android.view.View;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.TextView;
import android.widget.Toast;

import com.google.android.gms.auth.GoogleAuthUtil;
import com.google.android.gms.common.AccountPicker;
import com.google.api.client.extensions.android.http.AndroidHttp;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.vision.v1.Vision;
import com.google.api.services.vision.v1.model.AnnotateImageRequest;
import com.google.api.services.vision.v1.model.BatchAnnotateImagesRequest;
import com.google.api.services.vision.v1.model.BatchAnnotateImagesResponse;
import com.google.api.services.vision.v1.model.EntityAnnotation;
import com.google.api.services.vision.v1.model.Feature;
import com.google.api.services.vision.v1.model.Image;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;

public class MainActivity extends AppCompatActivity {
    private static String accessToken;
    static final int REQUEST_GALLERY_IMAGE = 10;
    static final int REQUEST_CODE_PICK_ACCOUNT = 11;
    static final int REQUEST_ACCOUNT_AUTHORIZATION = 12;
    static final int REQUEST_PERMISSIONS = 13;
    private final String LOG_TAG = "MainActivity";
    private ImageView selectedImage;
    private TextView resultTextView;
    Account mAccount;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Button selectImageButton = (Button) findViewById(R.id
                .select_image_button);
        selectedImage = (ImageView) findViewById(R.id.selected_image);
        resultTextView = (TextView) findViewById(R.id.result);

        selectImageButton.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                ActivityCompat.requestPermissions(MainActivity.this,
                        new String[]{Manifest.permission.GET_ACCOUNTS},
                        REQUEST_PERMISSIONS);
            }
        });
    }

    private void launchImagePicker() {
        Intent intent = new Intent();
        intent.setType("image/*");
        intent.setAction(Intent.ACTION_GET_CONTENT);
        startActivityForResult(Intent.createChooser(intent, "Select an image"),
                REQUEST_GALLERY_IMAGE);
    }

    @Override
    public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions,
                                           @NonNull int[] grantResults) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults);
        switch (requestCode) {
            case REQUEST_PERMISSIONS:
                if (grantResults.length > 0 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                    getAuthToken();
                } else {
                    Toast.makeText(MainActivity.this, "Permission Denied!", Toast.LENGTH_SHORT).show();
                }
        }
    }

    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        super.onActivityResult(requestCode, resultCode, data);
        if (requestCode == REQUEST_GALLERY_IMAGE && resultCode == RESULT_OK && data != null) {
            uploadImage(data.getData());
        } else if (requestCode == REQUEST_CODE_PICK_ACCOUNT) {
            if (resultCode == RESULT_OK) {
                String email = data.getStringExtra(AccountManager.KEY_ACCOUNT_NAME);
                AccountManager am = AccountManager.get(this);
                Account[] accounts = am.getAccountsByType(GoogleAuthUtil.GOOGLE_ACCOUNT_TYPE);
                for (Account account : accounts) {
                    if (account.name.equals(email)) {
                        mAccount = account;
                        break;
                    }
                }
                getAuthToken();
            } else if (resultCode == RESULT_CANCELED) {
                Toast.makeText(this, "No Account Selected", Toast.LENGTH_SHORT)
                        .show();
            }
        } else if (requestCode == REQUEST_ACCOUNT_AUTHORIZATION) {
            if (resultCode == RESULT_OK) {
                Bundle extra = data.getExtras();
                onTokenReceived(extra.getString("authtoken"));
            } else if (resultCode == RESULT_CANCELED) {
                Toast.makeText(this, "Authorization Failed", Toast.LENGTH_SHORT)
                        .show();
            }
        }
    }

    public void uploadImage(Uri uri) {
        if (uri != null) {
            try {
                Bitmap bitmap = resizeBitmap(
                                MediaStore.Images.Media.getBitmap(getContentResolver(), uri));
                callCloudVision(bitmap);
                selectedImage.setImageBitmap(bitmap);
            } catch (IOException e) {
                Log.e(LOG_TAG, e.getMessage());
            }
        } else {
            Log.e(LOG_TAG, "Null image was returned.");
        }
    }

    private void callCloudVision(final Bitmap bitmap) throws IOException {
        resultTextView.setText("Retrieving results from cloud");

        new AsyncTask<Object, Void, String>() {
            @Override
            protected String doInBackground(Object... params) {
                try {
                    GoogleCredential credential = new GoogleCredential().setAccessToken(accessToken);
                    HttpTransport httpTransport = AndroidHttp.newCompatibleTransport();
                    JsonFactory jsonFactory = GsonFactory.getDefaultInstance();

                    Vision.Builder builder = new Vision.Builder
                            (httpTransport, jsonFactory, credential);
                    Vision vision = builder.build();

                    List<Feature> featureList = new ArrayList<>();
                    Feature labelDetection = new Feature();
                    labelDetection.setType("LABEL_DETECTION");
                    labelDetection.setMaxResults(10);
                    featureList.add(labelDetection);

                    Feature textDetection = new Feature();
                    textDetection.setType("TEXT_DETECTION");
                    textDetection.setMaxResults(10);
                    featureList.add(textDetection);

                    Feature landmarkDetection = new Feature();
                    landmarkDetection.setType("LANDMARK_DETECTION");
                    landmarkDetection.setMaxResults(10);
                    featureList.add(landmarkDetection);

                    List<AnnotateImageRequest> imageList = new ArrayList<>();
                    AnnotateImageRequest annotateImageRequest = new AnnotateImageRequest();
                    Image base64EncodedImage = getBase64EncodedJpeg(bitmap);
                    annotateImageRequest.setImage(base64EncodedImage);
                    annotateImageRequest.setFeatures(featureList);
                    imageList.add(annotateImageRequest);

                    BatchAnnotateImagesRequest batchAnnotateImagesRequest =
                            new BatchAnnotateImagesRequest();
                    batchAnnotateImagesRequest.setRequests(imageList);

                    Vision.Images.Annotate annotateRequest =
                            vision.images().annotate(batchAnnotateImagesRequest);
                    // Due to a bug: requests to Vision API containing large images fail when GZipped.
                    annotateRequest.setDisableGZipContent(true);
                    Log.d(LOG_TAG, "sending request");

                    BatchAnnotateImagesResponse response = annotateRequest.execute();
                    return convertResponseToString(response);

                } catch (GoogleJsonResponseException e) {
                    Log.e(LOG_TAG, "Request failed: " + e.getContent());
                } catch (IOException e) {
                    Log.d(LOG_TAG, "Request failed: " + e.getMessage());
                }
                return "Cloud Vision API request failed.";
            }

            protected void onPostExecute(String result) {
                resultTextView.setText(result);
            }
        }.execute();
    }

    private String convertResponseToString(BatchAnnotateImagesResponse response) {
        StringBuilder message = new StringBuilder("Results:\n\n");
        message.append("Labels:\n");
        List<EntityAnnotation> labels = response.getResponses().get(0).getLabelAnnotations();
        if (labels != null) {
            for (EntityAnnotation label : labels) {
                message.append(String.format(Locale.getDefault(), "%.3f: %s",
                        label.getScore(), label.getDescription()));
                message.append("\n");
            }
        } else {
            message.append("nothing\n");
        }

        message.append("Texts:\n");
        List<EntityAnnotation> texts = response.getResponses().get(0)
                .getTextAnnotations();
        if (texts != null) {
            for (EntityAnnotation text : texts) {
                message.append(String.format(Locale.getDefault(), "%s: %s",
                        text.getLocale(), text.getDescription()));
                message.append("\n");
            }
        } else {
            message.append("nothing\n");
        }

        message.append("Landmarks:\n");
        List<EntityAnnotation> landmarks = response.getResponses().get(0)
                .getLandmarkAnnotations();
        if (landmarks != null) {
            for (EntityAnnotation landmark : landmarks) {
                message.append(String.format(Locale.getDefault(), "%.3f: %s",
                        landmark.getScore(), landmark.getDescription()));
                message.append("\n");
            }
        } else {
            message.append("nothing\n");
        }

        return message.toString();
    }

    public Bitmap resizeBitmap(Bitmap bitmap) {

        int maxDimension = 1024;
        int originalWidth = bitmap.getWidth();
        int originalHeight = bitmap.getHeight();
        int resizedWidth = maxDimension;
        int resizedHeight = maxDimension;

        if (originalHeight > originalWidth) {
            resizedHeight = maxDimension;
            resizedWidth = (int) (resizedHeight * (float) originalWidth / (float) originalHeight);
        } else if (originalWidth > originalHeight) {
            resizedWidth = maxDimension;
            resizedHeight = (int) (resizedWidth * (float) originalHeight / (float) originalWidth);
        } else if (originalHeight == originalWidth) {
            resizedHeight = maxDimension;
            resizedWidth = maxDimension;
        }
        return Bitmap.createScaledBitmap(bitmap, resizedWidth, resizedHeight, false);
    }

    public Image getBase64EncodedJpeg(Bitmap bitmap) {
        Image image = new Image();
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        bitmap.compress(Bitmap.CompressFormat.JPEG, 90, byteArrayOutputStream);
        byte[] imageBytes = byteArrayOutputStream.toByteArray();
        image.encodeContent(imageBytes);
        return image;
    }

    private void pickUserAccount() {
        String[] accountTypes = new String[]{GoogleAuthUtil.GOOGLE_ACCOUNT_TYPE};
        Intent intent = AccountPicker.newChooseAccountIntent(null, null,
                accountTypes, false, null, null, null, null);
        startActivityForResult(intent, REQUEST_CODE_PICK_ACCOUNT);
    }

    private void getAuthToken() {
        String SCOPE = "oauth2:https://www.googleapis.com/auth/cloud-platform";
        if (mAccount == null) {
            pickUserAccount();
        } else {
            new GetTokenTask(MainActivity.this, mAccount, SCOPE, REQUEST_ACCOUNT_AUTHORIZATION)
                    .execute();
        }
    }

    public void onTokenReceived(String token){
        accessToken = token;
        launchImagePicker();
    }
}

To perform a successful Android image recognition, using the new Google Cloud Vision API, we first need to authenticate the our app and generate an OAuth token for making an API call. But before generating an OAuth token we may first need to pick an account through which the OAuth token can be generated. But to pick an account we first need to get a permission called GET_ACCOUNTS. Once we have the permission to get accounts, we can call the method getAuthToken() mentioned above to pick an account and get its OAuth token. This method internally takes consent from the user, using the GetTokenTask; after they select an account using the pickUserAccount() method. Now once we have this API token we can call the launchImagePicker() method to pick an image from gallery and pass it on to the callCloudVision() method which uploads the image to Google Cloud Vision API and applies all the Android image analysis techniques on it. This method internally encodes the bitmap into a jpeg, creates and sends a request to the google cloud vision API. Further we parse the result and display it on screen. View the full source code here:

 Full Source Code

In the above Android image recognition example, we simply picked an account authenticated it, selected and uploaded an image to perform image analysis on it. Although we applied very basic set of image recognition techniques, like LABEL_DETECTION, TEXT_DETECTION, and LANDMARK_DETECTION. Which gave deep insights on how google looks at an image. Maybe in future Google would enhance this API and launch something like this for videos as well. But as of now even the simple Google Cloud Vision API for images is doing wonders. Let me know what you guys are planning to build with these awesome set of APIs. Connect with us on Facebook, Google+ and Twitter for more updates.

Mohit Gupt

Born in New Delhi, India. A software engineer by profession, an android enthusiast and mobile development evangelist. My motive here is to create a group of skilled engineers, who can build better software. Reason being programming is my passion, and also it feels good to make a device do something you want. Professionally I have worked with many software engineering and product development firms. As of now too, I am employed as a senior engineer in a leading tech company. In total I may have worked on more than 20 projects professionally, but whenever I get spare time I share my thoughts here at Truiton.

www.truiton.com

The post Image Recognition on Andorid with Google Cloud Vision API appeared first on Truiton.