Blocks2World: Controlling Realistic Scenes with Editable Primitives

TL;DR

Blocks2World introduces a novel 3D scene editing method that decomposes scenes into convex primitives, enabling flexible and controllable scene synthesis through primitive manipulation.

Contribution

The paper presents a new approach combining convex decomposition and conditioned synthesis for editable 3D scene rendering and editing.

Findings

Effective primitive-based scene control at test time

High-quality scene synthesis from primitive manipulation

Enables scene editing and data augmentation

Abstract

We present Blocks2World, a novel method for 3D scene rendering and editing that leverages a two-step process: convex decomposition of images and conditioned synthesis. Our technique begins by extracting 3D parallelepipeds from various objects in a given scene using convex decomposition, thus obtaining a primitive representation of the scene. These primitives are then utilized to generate paired data through simple ray-traced depth maps. The next stage involves training a conditioned model that learns to generate images from the 2D-rendered convex primitives. This step establishes a direct mapping between the 3D model and its 2D representation, effectively learning the transition from a 3D model to an image. Once the model is fully trained, it offers remarkable control over the synthesis of novel and edited scenes. This is achieved by manipulating the primitives at test time, including translating or adding them, thereby enabling a highly customizable scene rendering process. Our method provides a fresh perspective on 3D scene rendering and editing, offering control and flexibility. It opens up new avenues for research and applications in the field, including authoring and data augmentation.