Breaking the Spatio-Angular Trade-off for Light Field Super-Resolution via LSTM Modelling on Epipolar Plane Images

TL;DR

This paper introduces a novel light field super-resolution method that leverages LSTM modeling on EPIs to surpass traditional spatio-angular resolution limits, demonstrating significant improvements in synthetic and real data scenarios.

Contribution

It presents a theoretical analysis showing the potential to break the resolution trade-off and proposes a CNN-LSTM network focusing on geometric continuity for enhanced super-resolution.

Findings

Outperforms state-of-the-art methods in synthetic and real light fields

Effective in large disparity regions

Theoretically demonstrates surpassing the resolution trade-off

Abstract

Light-field cameras (LFC) have received increasing attention due to their wide-spread applications. However, current LFCs suffer from the well-known spatio-angular trade-off, which is considered as an inherent and fundamental limit for LFC designs. In this paper, by doing a detailed geometrical optical analysis of the sampling process in an LFC, we show that the effective sampling resolution is generally higher than the number of micro-lenses. This contribution makes it theoretically possible to break the resolution trade-off. Our second contribution is an epipolar plane image (EPI) based super-resolution method, which can super-resolve the spatial and angular dimensions simultaneously. We prove that the light field is a 2D series, thus, a specifically designed CNN-LSTM network is proposed to capture the continuity property of the EPI. Rather than leveraging semantic information, our network focuses on extracting geometric continuity in the EPI. This gives our method an improved generalization ability and makes it applicable to a wide range of previously unseen scenes. Experiments on both synthetic and real light fields demonstrate the improvements over state-of-the-art, especially in large disparity areas.