DeepDR: Deep Structure-Aware RGB-D Inpainting for Diminished Reality

TL;DR

DeepDR is a real-time RGB-D inpainting framework that generates coherent color and depth images for diminished reality, effectively removing objects while maintaining scene structure and geometry.

Contribution

It introduces a structure-aware generative network that explicitly conditions on scene semantics to improve inpainting quality in RGB-D data for diminished reality.

Findings

Outperforms related methods qualitatively and quantitatively

Operates at real-time frame rates with minimal temporal artifacts

Successfully reconstructs sharp, consistent boundaries in complex backgrounds

Abstract

Diminished reality (DR) refers to the removal of real objects from the environment by virtually replacing them with their background. Modern DR frameworks use inpainting to hallucinate unobserved regions. While recent deep learning-based inpainting is promising, the DR use case is complicated by the need to generate coherent structure and 3D geometry (i.e., depth), in particular for advanced applications, such as 3D scene editing. In this paper, we propose DeepDR, a first RGB-D inpainting framework fulfilling all requirements of DR: Plausible image and geometry inpainting with coherent structure, running at real-time frame rates, with minimal temporal artifacts. Our structure-aware generative network allows us to explicitly condition color and depth outputs on the scene semantics, overcoming the difficulty of reconstructing sharp and consistent boundaries in regions with complex backgrounds. Experimental results show that the proposed framework can outperform related work qualitatively and quantitatively.