GAN-Based Multi-View Video Coding with Spatio-Temporal EPI Reconstruction

TL;DR

This paper introduces a GAN-based multi-view video coding method that enhances side information reconstruction accuracy and reduces bitrate by leveraging spatio-temporal EPIs and a joint encoder constraint, outperforming existing techniques.

Contribution

It proposes a novel GAN-driven approach utilizing spatio-temporal EPIs and joint encoding constraints for improved multi-view video compression.

Findings

Significant improvement in Rate-Distortion performance.

Effective reduction of side information redundancy.

Enhanced reconstruction quality compared to state-of-the-art methods.

Abstract

The introduction of multiple viewpoints in video scenes inevitably increases the bitrates required for storage and transmission. To reduce bitrates, researchers have developed methods to skip intermediate viewpoints during compression and delivery, and ultimately reconstruct them using Side Information (SI). Typically, depth maps are used to construct SI. However, their methods suffer from inaccuracies in reconstruction and inherently high bitrates. In this paper, we propose a novel multi-view video coding method that leverages the image generation capabilities of Generative Adversarial Network (GAN) to improve the reconstruction accuracy of SI. Additionally, we consider incorporating information from adjacent temporal and spatial viewpoints to further reduce SI redundancy. At the encoder, we construct a spatio-temporal Epipolar Plane Image (EPI) and further utilize a convolutional network to extract the latent code of a GAN as SI. At the decoder side, we combine the SI and adjacent viewpoints to reconstruct intermediate views using the GAN generator. Specifically, we establish a joint encoder constraint for reconstruction cost and SI entropy to achieve an optimal trade-off between reconstruction quality and bitrates overhead. Experiments demonstrate significantly improved Rate-Distortion (RD) performance compared with state-of-the-art methods.