gCoRF: Generative Compositional Radiance Fields

TL;DR

gCoRF introduces a novel 3D generative model that decomposes objects into semantic parts learned from 2D data, enabling flexible editing and compositional reasoning for photorealistic scene synthesis.

Contribution

It presents a new compositional generative approach that models semantic parts independently, allowing for scene editing and generalizable 3D reasoning from in-the-wild 2D data.

Findings

Effective decomposition of objects into semantic parts

Enables independent sampling and editing of parts

Demonstrates high-quality 3D scene synthesis

Abstract

3D generative models of objects enable photorealistic image synthesis with 3D control. Existing methods model the scene as a global scene representation, ignoring the compositional aspect of the scene. Compositional reasoning can enable a wide variety of editing applications, in addition to enabling generalizable 3D reasoning. In this paper, we present a compositional generative model, where each semantic part of the object is represented as an independent 3D representation learned from only in-the-wild 2D data. We start with a global generative model (GAN) and learn to decompose it into different semantic parts using supervision from 2D segmentation masks. We then learn to composite independently sampled parts in order to create coherent global scenes. Different parts can be independently sampled while keeping the rest of the object fixed. We evaluate our method on a wide variety of objects and parts and demonstrate editing applications.