Disentangling Light Fields for Super-Resolution and Disparity Estimation

TL;DR

This paper introduces a novel disentangling mechanism for light field images that improves the performance of neural networks on super-resolution and disparity estimation tasks by effectively separating spatial and angular information.

Contribution

The paper proposes a generic disentangling mechanism using domain-specific convolutions for light field processing, enabling improved performance across multiple tasks.

Findings

Achieves state-of-the-art results on super-resolution and disparity estimation

Effectively handles 4D light field data with improved efficiency

Demonstrates the generality of the disentangling approach

Abstract

Light field (LF) cameras record both intensity and directions of light rays, and encode 3D scenes into 4D LF images. Recently, many convolutional neural networks (CNNs) have been proposed for various LF image processing tasks. However, it is challenging for CNNs to effectively process LF images since the spatial and angular information are highly inter-twined with varying disparities. In this paper, we propose a generic mechanism to disentangle these coupled information for LF image processing. Specifically, we first design a class of domain-specific convolutions to disentangle LFs from different dimensions, and then leverage these disentangled features by designing task-specific modules. Our disentangling mechanism can well incorporate the LF structure prior and effectively handle 4D LF data. Based on the proposed mechanism, we develop three networks (i.e., DistgSSR, DistgASR and DistgDisp) for spatial super-resolution, angular super-resolution and disparity estimation. Experimental results show that our networks achieve state-of-the-art performance on all these three tasks, which demonstrates the effectiveness, efficiency, and generality of our disentangling mechanism. Project page: https://yingqianwang.github.io/DistgLF/.