Debiasing Diffusion Priors via 3D Attention for Consistent Gaussian Splatting

TL;DR

This paper introduces TD-Attn, a novel framework that improves multi-view consistency in 3D tasks derived from Text-to-Image diffusion models by addressing prior view bias through 3D-aware attention guidance and hierarchical modulation.

Contribution

The paper presents a new framework, TD-Attn, with two modules that mitigate prior view bias and enhance multi-view consistency in 3D generation and editing tasks.

Findings

Significantly improves multi-view consistency in 3D tasks

Enables controllable and precise 3D editing

Serves as a universal plugin for 3D applications

Abstract

Versatile 3D tasks (e.g., generation or editing) that distill from Text-to-Image (T2I) diffusion models have attracted significant research interest for not relying on extensive 3D training data. However, T2I models exhibit limitations resulting from prior view bias, which produces conflicting appearances between different views of an object. This bias causes subject-words to preferentially activate prior view features during cross-attention (CA) computation, regardless of the target view condition. To overcome this limitation, we conduct a comprehensive mathematical analysis to reveal the root cause of the prior view bias in T2I models. Moreover, we find different UNet layers show different effects of prior view in CA. Therefore, we propose a novel framework, TD-Attn, which addresses multi-view inconsistency via two key components: (1) the 3D-Aware Attention Guidance Module (3D-AAG) constructs a view-consistent 3D attention Gaussian for subject-words to enforce spatial consistency across attention-focused regions, thereby compensating for the limited spatial information in 2D individual view CA maps; (2) the Hierarchical Attention Modulation Module (HAM) utilizes a Semantic Guidance Tree (SGT) to direct the Semantic Response Profiler (SRP) in localizing and modulating CA layers that are highly responsive to view conditions, where the enhanced CA maps further support the construction of more consistent 3D attention Gaussians. Notably, HAM facilitates semantic-specific interventions, enabling controllable and precise 3D editing. Extensive experiments firmly establish that TD-Attn has the potential to serve as a universal plugin, significantly enhancing multi-view consistency across 3D tasks.