3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation

TL;DR

3D Paintbrush introduces a cascaded score distillation method for local texturing of 3D meshes, combining semantic localization and multi-resolution generative priors to enhance detail and control in stylization.

Contribution

The paper presents a novel cascaded score distillation approach that enables high-quality, local texturing of 3D shapes directly on meshes using text descriptions.

Findings

Effective local texturing on diverse 3D shapes

Improved detail and resolution through cascaded diffusion supervision

Seamless integration into standard graphics pipelines

Abstract

In this work we develop 3D Paintbrush, a technique for automatically texturing local semantic regions on meshes via text descriptions. Our method is designed to operate directly on meshes, producing texture maps which seamlessly integrate into standard graphics pipelines. We opt to simultaneously produce a localization map (to specify the edit region) and a texture map which conforms to it. This synergistic approach improves the quality of both the localization and the stylization. To enhance the details and resolution of the textured area, we leverage multiple stages of a cascaded diffusion model to supervise our local editing technique with generative priors learned from images at different resolutions. Our technique, referred to as Cascaded Score Distillation (CSD), simultaneously distills scores at multiple resolutions in a cascaded fashion, enabling control over both the granularity and global understanding of the supervision. We demonstrate the effectiveness of 3D Paintbrush to locally texture a variety of shapes within different semantic regions. Project page: https://threedle.github.io/3d-paintbrush