Spice-E : Structural Priors in 3D Diffusion using Cross-Entity Attention

TL;DR

Spice-E introduces a novel cross-entity attention mechanism to incorporate structural priors into 3D diffusion models, enabling versatile 3D shape editing and stylization with improved speed and performance.

Contribution

The paper proposes Spice-E, a framework that adds structural guidance to 3D diffusion models via cross-entity attention, extending their capabilities beyond text-conditional generation.

Findings

Achieves state-of-the-art results in 3D stylization and editing tasks.

Supports various applications without task-specific tailoring.

Operates faster than existing methods while maintaining high quality.

Abstract

We are witnessing rapid progress in automatically generating and manipulating 3D assets due to the availability of pretrained text-image diffusion models. However, time-consuming optimization procedures are required for synthesizing each sample, hindering their potential for democratizing 3D content creation. Conversely, 3D diffusion models now train on million-scale 3D datasets, yielding high-quality text-conditional 3D samples within seconds. In this work, we present Spice-E - a neural network that adds structural guidance to 3D diffusion models, extending their usage beyond text-conditional generation. At its core, our framework introduces a cross-entity attention mechanism that allows for multiple entities (in particular, paired input and guidance 3D shapes) to interact via their internal representations within the denoising network. We utilize this mechanism for learning task-specific structural priors in 3D diffusion models from auxiliary guidance shapes. We show that our approach supports a variety of applications, including 3D stylization, semantic shape editing and text-conditional abstraction-to-3D, which transforms primitive-based abstractions into highly-expressive shapes. Extensive experiments demonstrate that Spice-E achieves SOTA performance over these tasks while often being considerably faster than alternative methods. Importantly, this is accomplished without tailoring our approach for any specific task.