Multiview Image and Decoding Pipeline

Learn about the Multiview Image and decoding different file types to generate multiple views.

In the previous sections, you learnt how to leverage the SDKs to display different image in 4V using a Quad, as well as convert other supported image formats to a Quad using the built in Image decoders and synthesizers.

In this section, you will learn about the decoding process, and how to leverage the SDK to perform operations on the different image formats in your android applications.

What is a Multiview Image?

The MultiviewImage is a collection of ViewPoints, and global metadata that affects all views like gain, convergence and depth of field. It is a decoded representation of an image that can be rendered on a Leia Lightfield display.

A ViewPoint represents the image and disparity map from a single camera view point.

Decoding Pipeline

The Multiview Image Decoder can be used to decode the different image formats supported by the SDK and obtain a Multiview Image instance.

The Multivew Image provides a unified API for loading the different image formats.

The Android Media SDK provides a set of operators that work with the Multiview image, allowing you to perform operations like View synthesis, disparity estimation and depth of field.

The Multiview Image can be displayed on the screen using the QuadView as we seen in the previous sections.

Multiview Image Decoder

The MultiviewImage decoder decodes different image types to return a MultiviewImage instance.

Using the Multiview Image Decoder

You can obtain an instance of the MultiviewImageDecoder by calling MultiviewImageDecoder.getDefault()

Kotlin

val multiviewImageDecoder = MultiviewImageDecoder.getDefault()

Java

MultiviewImageDecoder decoder = MultiviewImageDecoder.getDefault();

Decoding Images

The MultiviewImageDecoder provides different methods to decode images. In the previous sections, we decoded a LIF image using raw byte[] as well as decoded a SBS image by using the File Uri.

1. Decoding with Uri

This is the most recommended method for decoding the different image formats due to its compatibility with all image formats.

You can call the following function to decode using the file Uri.

Kotlin

val multiviewImage = multiviewImageDecoder.decode(context, fileUri, 1280 * 720)

Java

MultiviewImage multiviewImage = multiviewImageDecoder.decode(context, fileUri, 1280 * 720);

The function takes in 3 parameters-

• The Application context.

• The File Uri or contentUri for the image on the device's internal storage.

• The output resolution for all the decoded viewpoints in the Multiview image. This rescales all views to the set number of pixels, if necessary.

Optionally, if you know the encoded image file contains ViewPoints with the same resolution, you can decode without specifying the output pixels.

Kotlin

val multiviewImage = multiviewImageDecoder.decode(context, fileUri)

Java

MultiviewImage multiviewImage = multiviewImageDecoder.decode(context, fileUri);

Note: Calling decode() without the output image resolution will throw an Exception if all decoded viewpoints are not the same resolution.

2. Decoding with raw bytes

You can also decode images using raw bytes. However, this method is only supported for Leia Image file types. (H4VBinary & H4V_XMP)

Kotlin

val multiviewImage = multiviewImageDecoder.decode(imageBytes, 1280 * 720)

Java

MultiviewImage multiviewImage = multiviewImageDecoder.decode(imageBytes, 1280 * 720);

Optionally, if you know the encoded image file contains ViewPoints of the same resolution, you can decode without specifying the output pixels.

Kotlin

val multiviewImage = multiviewImageDecoder.decode(imageBytes)

Java

MultiviewImage multiviewImage = multiviewImageDecoder.decode(imageBytes);

Note: Calling decode() without output image resolution will throw an Exception if all decoded viewpoints are not the same resolution.

Detect File Types

You can also use the MultiviewImageDecoder to detect the image file type.

Kotlin

val decoder = MultiviewImageDecoder.getDefault()
val fileType = decoder.getFileType(context, fileUri)

Java

MultiviewImageDecoder decoder = MultiviewImageDecoder.getDefault();
MultiviewFileType fileType = decoder.getFileType(context, fileUri);

This function takes in the File Uri and returns a MultiviewFileType. The returned Multiview File types are categorized below

Multiview File Type	Image File Type
MultiviewFileType.H4V_BINARY	Leia Image Format
MultiviewFileType.H4V_XMP	Leia Image Format
MultiviewFileType.TWO_BY_ONE	SBS (2x1)
MultiviewFileType.TWO_BY_TWO	Quad (2x2)

The decoder returns null if the Uri specified is not a supported file type.

Decoder Compatibility

To ensure decoding compatibility for SBS (2x1) and Quad (2x2) images, you need to make sure the files for these image formats follow proper naming conventions.

• SBS image file names must have a suffix of _2x1

• Quad image file names must have a suffix of _2x2

File Type	Valid File name
SBS (2x1)	cards_2x1.jpg
Quad (2x2)	cards_2x2.jpg

SBS and Quad images that don't follow this convention will not be detected as a valid file type, and will also fail decoding.

Multiview Operations

We can now perform a several operations on the Multiview Image obtained from the decoder. This includes disparity estimation and View Synthesis.

You can perform these operations on the Multiview Image using the MultiviewSynthesizer2. You can obtain an instance of this class as shown below.

Kotlin

val synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context)

Java

MultiviewSynthesizer2 synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context);

Disparity Estimation

Images decoded using the Multiview Image Decoder may or may not have depth information, depending on how they were encoded. In order to generate multiple views for a Multiview Image, we need to estimate disparity.

You can populate the disparity maps in a Multiview Image using the MultiviewSynthesizer

Kotlin

val synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context)
synthesizer.populateDisparityMaps(multiviewImage)

Java

MultiviewSynthesizer2 synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context);
synthesizer.populateDisparityMaps(multiviewImage);

View Synthesis

The final steps to display a Multiview image on a Leia Lightfield device is to generate multiple views. This step performs View Synthesis to generate 4 viewpoints and returns a Bitmap representing a Quad.

Kotlin

val synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context)
val quadBitmap = synthesizer.toQuadBitmap(multiviewImage)

Java

MultiviewSynthesizer2 synthesizer = MultiviewSynthesizer2.createMultiviewSynthesizer(context);
Bitmap quadBitmap = synthesizer.toQuadBitmap(multiviewImage);

The Quad bitmap can be used to display the image on the device using the QuadView

Kotlin

quadView.setQuadBitmap(quadBitmap)

Java

quadView.setQuadBitmap(quadBitmap);