Evaluation of 4D Light Field Compression Methods

TL;DR

This paper evaluates how state-of-the-art image and video compression methods, including AV1 and XVC, perform on 4D light field data, demonstrating significant compression potential while maintaining visual quality.

Contribution

It extends existing compression methods into 3D and 4D for light fields and compares their effectiveness, identifying the most suitable approach for light field data.

Findings

4D light field data can be compressed more than independent images.

Certain video compression standards outperform traditional image codecs.

Extended compression methods maintain visual quality at high compression ratios.

Abstract

Light field data records the amount of light at multiple points in space, captured e.g. by an array of cameras or by a light-field camera that uses microlenses. Since the storage and transmission requirements for such data are tremendous, compression techniques for light fields are gaining momentum in recent years. Although plenty of efficient compression formats do exist for still and moving images, only a little research on the impact of these methods on light field imagery is performed. In this paper, we evaluate the impact of state-of-the-art image and video compression methods on quality of images rendered from light field data. The methods include recent video compression standards, especially AV1 and XVC finalised in 2018. To fully exploit the potential of common image compression methods on four-dimensional light field imagery, we have extended these methods into three and four dimensions. In this paper, we show that the four-dimensional light field data can be compressed much more than independent still images while maintaining the same visual quality of a perceived picture. We gradually compare the compression performance of all image and video compression methods, and eventually answer the question, "What is the best compression method for light field data?".

Tables

Table 1. Table 1 : Dataset used in this paper. The first and last light field are taken using a plenoptic camera; the Chessboard is captured using a camera array; the Lego Bulldozer is captured using a motorized gantry holding a camera. The adjacent image disparity range (last column) is given in pixels.

o l|X[l]|c|l   description	source	resolution	disparity
Black Fence	EPFL Light-field data set	$15\times 15$$\times$$625\times 434$	$-1$ to $1$
Chessboard	Saarland University	$8\times 8$$\times$$1920\times 1080$	$40$ to $90$
Lego Bulldozer	Stanford Computer Graphics Laboratory	$17\times 17$$\times$$1536\times 1152$	$-1$ to $7$
Palais du Luxembourg	EPFL Light-field data set	$15\times 15$$\times$$625\times 434$	$-1$ to $1$

Equations

$$E_d(m,n)=\frac{1}{N}\sum _{k,l}L(k,l,m+\alpha k,n+\alpha l),$$

Full text

Keywords

Light field, Plenoptic imaging, Lossy compression, Image refocusing

1 Introduction

To describe a three-dimensional scene from any possible viewing position at any viewing angle, one could define a plenoptic function $P(x,y,z,\phi,\psi)$, where the $(x,y,z)$ is the position and $(\phi,\psi)$ is a viewing angle (in spherical coordinates) of a camera. Figure 1 shows the situation. The value of the $P$ is color. The definition can be further extended with $t$ (time) to describe a dynamic scene.

Our interest here is to describe the scene by capturing either via an array of cameras or by a single compact sensor preceded by microlenses. In this case, the aperture is modeled by a grid of views (cameras) located on a two-dimensional plane. This situation is shown in Figure 2, where the baseline between individual views from the grid is described by the distance $d$. This representation is often referred to as 4D light field (LF) since we deal with the light field function, $L$, sampled in four dimensions, $(k,l,m,n)$, where the $(m,n)$ are pixel coordinates, and $(k,l)$ are indices of a sub-aperture image.

Light fields acquired by the single compact sensor have limited support for the viewing angle. Light fields based on the array of cameras offer larger viewing angles at the cost of missing information in between the cameras. In practice, the number of views located on the two-dimensional plane ranges from a couple of units to several hundred. Considering increasing resolution sensors, it is no surprise that the light field data reach huge sizes. As an example, consider "Lego Bulldozer" light field (Figure 3) taken from the Stanford Light Field Archive. The light field is captured using a $17\times 17$ grid of cameras having image resolution $1536\times 1152$ (rectified and cropped). The uncompressed size easily exceeds a gigabyte. For light field videos, storage and transmission requirements are enormous.

Several methods to compress 4D light fields have been recently proposed. Some of them attempt to compress directly the data from sensors preceded by microlenses (lenslet image). Other compresses the resulting 4D light field. In this paper, we focus only on the latter ones. We compare various state-of-the-art compression methods applicable to 4D light field data. These methods include recent video compression standards, especially AV1 (validated in June 2018), and XVC (version released in July 2018). In order to evaluate the comparison, we refocus the original and decompressed light field. The evaluation is then carried out using the PSNR as a full-reference quality assessment metric.

The remainder of the paper is organized as follows. Section 2 reviews related work and compression methods. Section 3 presents our experiments in detail, and discusses the results. Section 4 concludes the paper.

2 Related Work

The individual views from a light field are usually never displayed. Therefore, it is not very meaningful to compare the original and decompressed light field directly, even though such methodology is usual to asses a single view compression performance. For this reason, we adopt the compression performance assessment methodology for multi-focus rendering from [2]. This methodology basically lies in assessing the quality of the rendered views for multiple focal points. The rendered views are obtained by combining pixels from different 4D light field views for various focal planes. The average distortion is computed as the mean of the PSNR for multiple rendered focal plane views. This situation is shown in Figure 4. Note that the PSNR is computed from the MSE over all three color components.

The 4D light field comprises a two-dimensional grid of two-dimensional views. The baseline between individual views ranges from a few millimeters (microlenses) to several centimeters (camera array). It is, therefore, natural to expect a high similarity of views adjacent in any of two grid directions. This similarity opens the door to understanding the 4D light field data as a video sequence navigating between the viewpoints. Another possible point of view is to see the 4D light field as the three- or directly four-dimensional body. The above approaches can also be reflected in light field compression by using either an image, video, volumetric, or four-dimensional coding system. Although other approaches (like 3D video) are also possible, we are not aware of generally available coding systems for such cases.

In recent years, several papers compared and evaluated the compression performance of various approaches on light field imagery. The authors of [2] evaluated the performance of the main image coding standards, JPEG, JPEG 2000, H.264/AVC intra profile, and H.265/HEVC intra profile. The "intra" suffix refers to the fact the individual views were compressed independently (intra profile). The video coding approaches were not evaluated. As could be expected, the H.265/HEVC intra profile proved to be the most efficient compression method. In [17], the authors compared the compression performance of three strategies using the H.265/HEVC. Their first strategy performs compression directly on the lenslet image. Another strategy arranges 4D LF views a pseudo-temporal sequence in spiral order and subsequently compressed it. The last strategy compresses a subset of lenslet images through the transformation to 4D LF. Their results show that coding 4D LF leads to better performance when compared to coding lenslet images directly. The authors of [6] compared the performance of JPEG, JPEG 2000, and SPIHT directly on lenslet images. The comparison was performed using the same methodology as in this paper. As could be expected, the JPEG 2000 exhibits the best compression performance. In [16], the authors proposed to rearrange 4D LF views into tiles of a big rectangular image. This image is then compressed using the JPEG 2000 coder. The proposed scheme was compared against standard image coding algorithms, namely the JPEG 2000 and JPEG XR. It is, however, unclear how these standard coding algorithms were exactly applied to the 4D light field data. In [1], the author rearranges the 4D light field into a three-dimensional body. The three-dimensional volume is then encoded using the 3D DCT scheme on $8\times 8\times 8$ blocks, similarly as in the JPEG coding system.

Besides conventional coding methods, also an alternative approach [3] exists that uses deep learning to estimate the 2D view from the sparse sets of 4D views. Another approach [4] proposes own sparse coding scheme for the entire 4D LF based on several optimized key views. The method in [9] decomposes the 4D light field into homography parameters and residual matrix. The matrix is then factored as the product of a matrix containing $k$ basis vectors and a smaller matrix of coefficients. The basis vectors are then encoded using the H.265/HEVC intra profile. In [11, 12], the authors propose a hierarchical coding structure for 4D light fields. The 4D LF is decomposed into multiple views and then organized them into a coding structure according to the spatial coordinates. All of the views are encoded hierarchically. The scheme is implemented in the reference H.265/HEVC software. In [5], the authors propose a coding scheme that splits the 4D light field into several central views and remaining adjacent views. The adjacent views are subtracted from the central views, and both groups are then encoded using H.265/HEVC coder. The authors of [13, 14] feed the 4D LF into the H.265/HEVC exploiting the inter prediction mode for individual LF views. Finally, tremendous attentions have also been focused on convolutional neural network based compression approaches [7, 8].

From the above, it can be seen that the JPEG 2000 and especially the H.265/HEVC coding schemes are quite popular. In this paper, we compare the compression performance of the main state-of-the-art lossy compression methods. These methods can be divided into four groups according to the way they process the 4D LF data. The first group covers the following image coding methods—the JPEG and JPEG 2000. In the literature [10], this kind of methods is sometimes referred to as the self-similarity based methods. The second group comprises video coding methods: H.265/HEVC, AV1, VP9, and XVC. In the literature, these methods are referred to as the pseudo sequence based methods. The third group extends the image coding methods into three dimensions. This group consists of JPEG 3D (our own implementation) and JPEG 2000 3D (Part 10, JP3D). Notice that the JPEG 3D refers to a volume image rather than a pair of stereoscopic images. The fourth group extends the image coding methods into four dimensions. However, only one method in this group exists, JPEG 4D (our own implementation). To evaluate the above methods, we use the following list of encoders: OpenJPEG, x265, libaom (AV1 Codec Library), libvpx (VP8/VP9 Codec SDK), xvc codec, and our own implementation of the JPEG method.

Since our comparison also deals with the latest video compression standards, we consider it appropriate to present their short description here. The H.265/HEVC (High Efficiency Video Coding, MPEG-H Part 2) is a video compression standard designed as a successor to the widely used H.264/AVC (MPEG-4 Part 10). The standard was published in June 2013. The AV1 (AOMedia Video 1) is an open video coding format standardized in June 2018. It succeeds the VP9 video coding format developed by Google. According to [18], the AV1 outperforms the H.265/HEVC by 17 %, and VP9 by 13 % over a wide range of bitrate/resolutions. The XVC is a video coding format with a strong focus on low bitrate streaming applications. The official website claims that the codec outperforms the AV1, H.265/HEVC, and VP9.

3 Evaluation