DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancement

TL;DR

DECOLLAGE introduces a machine learning-based 3D modeling technique that allows users to selectively add detailed styles to coarse shapes, preserving structure and enabling interactive creative workflows.

Contribution

The paper presents a novel masking-aware Pyramid GAN approach with structural losses for controllable, localized 3D detailization from diverse style inputs.

Findings

Produces high-resolution, style-transferred geometries with preserved structures.

Enables interactive, localized editing of 3D shapes.

Outperforms prior global detailization methods in coherence and style transition quality.

Abstract

We present a 3D modeling method which enables end-users to refine or detailize 3D shapes using machine learning, expanding the capabilities of AI-assisted 3D content creation. Given a coarse voxel shape (e.g., one produced with a simple box extrusion tool or via generative modeling), a user can directly "paint" desired target styles representing compelling geometric details, from input exemplar shapes, over different regions of the coarse shape. These regions are then up-sampled into high-resolution geometries which adhere with the painted styles. To achieve such controllable and localized 3D detailization, we build on top of a Pyramid GAN by making it masking-aware. We devise novel structural losses and priors to ensure that our method preserves both desired coarse structures and fine-grained features even if the painted styles are borrowed from diverse sources, e.g., different semantic parts and even different shape categories. Through extensive experiments, we show that our ability to localize details enables novel interactive creative workflows and applications. Our experiments further demonstrate that in comparison to prior techniques built on global detailization, our method generates structure-preserving, high-resolution stylized geometries with more coherent shape details and style transitions.