Text2VDM: Text to Vector Displacement Maps for Expressive and Interactive 3D Sculpting

TL;DR

Text2VDM introduces a text-guided framework for generating vector displacement maps, enabling artists to create diverse and high-quality 3D sculpting brushes that integrate seamlessly into existing workflows.

Contribution

The paper proposes a novel text-to-VDM generation method using score distillation sampling with weighted prompt blending, addressing semantic coupling issues.

Findings

Generates diverse, high-quality VDM brushes for 3D sculpting.

Enables seamless integration into modeling software.

Supports applications like mesh stylization and real-time editing.

Abstract

Professional 3D asset creation often requires diverse sculpting brushes to add surface details and geometric structures. Despite recent progress in 3D generation, producing reusable sculpting brushes compatible with artists' workflows remains an open and challenging problem. These sculpting brushes are typically represented as vector displacement maps (VDMs), which existing models cannot easily generate compared to natural images. This paper presents Text2VDM, a novel framework for text-to-VDM brush generation through the deformation of a dense planar mesh guided by score distillation sampling (SDS). The original SDS loss is designed for generating full objects and struggles with generating desirable sub-object structures from scratch in brush generation. We refer to this issue as semantic coupling, which we address by introducing weighted blending of prompt tokens to SDS, resulting in a more accurate target distribution and semantic guidance. Experiments demonstrate that Text2VDM can generate diverse, high-quality VDM brushes for sculpting surface details and geometric structures. Our generated brushes can be seamlessly integrated into mainstream modeling software, enabling various applications such as mesh stylization and real-time interactive modeling.