Automatic Scene Inference for 3D Object Compositing

TL;DR

This paper introduces an automatic system for 3D scene inference from a single image, enabling realistic object insertion and editing with minimal user effort, by recovering scene geometry, lighting, and camera parameters.

Contribution

The paper presents novel algorithms for automatic illumination inference and depth estimation, enabling comprehensive 3D scene modeling from a single photograph.

Findings

System produces perceptually convincing results

Achieves realism comparable to interactive methods

Enables drag-and-drop object insertion and editing

Abstract

We present a user-friendly image editing system that supports a drag-and-drop object insertion (where the user merely drags objects into the image, and the system automatically places them in 3D and relights them appropriately), post-process illumination editing, and depth-of-field manipulation. Underlying our system is a fully automatic technique for recovering a comprehensive 3D scene model (geometry, illumination, diffuse albedo and camera parameters) from a single, low dynamic range photograph. This is made possible by two novel contributions: an illumination inference algorithm that recovers a full lighting model of the scene (including light sources that are not directly visible in the photograph), and a depth estimation algorithm that combines data-driven depth transfer with geometric reasoning about the scene layout. A user study shows that our system produces perceptually convincing results, and achieves the same level of realism as techniques that require significant user interaction.