I See-Through You: A Framework for Removing Foreground Occlusion in Both Sparse and Dense Light Field Images

TL;DR

This paper introduces ISTY, a novel framework for removing foreground occlusion in both sparse and dense light field images, enabling better occlusion-free view reconstruction and analysis.

Contribution

The paper proposes a unified framework, ISTY, that effectively handles occlusion removal in both sparse and dense light field datasets with an explainable occlusion mask.

Findings

Outperforms state-of-the-art methods in both sparse and dense LF datasets.

Provides an explainable occlusion mask for analysis and manipulation.

Demonstrates effectiveness through qualitative and quantitative results.

Abstract

Light field (LF) camera captures rich information from a scene. Using the information, the LF de-occlusion (LF-DeOcc) task aims to reconstruct the occlusion-free center view image. Existing LF-DeOcc studies mainly focus on the sparsely sampled (sparse) LF images where most of the occluded regions are visible in other views due to the large disparity. In this paper, we expand LF-DeOcc in more challenging datasets, densely sampled (dense) LF images, which are taken by a micro-lens-based portable LF camera. Due to the small disparity ranges of dense LF images, most of the background regions are invisible in any view. To apply LF-DeOcc in both LF datasets, we propose a framework, ISTY, which is defined and divided into three roles: (1) extract LF features, (2) define the occlusion, and (3) inpaint occluded regions. By dividing the framework into three specialized components according to the roles, the development and analysis can be easier. Furthermore, an explainable intermediate representation, an occlusion mask, can be obtained in the proposed framework. The occlusion mask is useful for comprehensive analysis of the model and other applications by manipulating the mask. In experiments, qualitative and quantitative results show that the proposed framework outperforms state-of-the-art LF-DeOcc methods in both sparse and dense LF datasets.