3D-Aware Scene Manipulation via Inverse Graphics

TL;DR

This paper introduces 3D-SDN, a deep generative model that creates interpretable, disentangled 3D scene representations enabling realistic scene manipulation like object rotation, movement, and appearance changes.

Contribution

The work presents a novel 3D scene de-rendering network that integrates disentangled semantics, geometry, and appearance into a structured, interpretable scene representation.

Findings

3D-SDN outperforms 2D-based editing methods in scene manipulation tasks.

The model achieves disentangled control over object shape, texture, and position.

Experiments validate the effectiveness of inverse graphics in 3D scene editing.

Abstract

We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart.