3D Photography using Context-aware Layered Depth Inpainting

TL;DR

This paper introduces a learning-based method to generate 3D photos from a single RGB-D image by inpainting occluded regions with context-aware color and depth, enabling realistic view synthesis with fewer artifacts.

Contribution

It presents a novel layered depth inpainting approach that effectively hallucinates occluded regions for 3D photo creation from a single image.

Findings

Fewer artifacts compared to state-of-the-art methods.

Effective in diverse everyday scenes.

Enables efficient rendering with motion parallax.

Abstract

We propose a method for converting a single RGB-D input image into a 3D photo - a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show fewer artifacts compared with the state of the arts.