HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generation

TL;DR

HOTS3D introduces a novel spherical optimal transport approach to improve semantic alignment between text and 3D shapes in CLIP-guided generation, achieving more faithful and semantically consistent 3D outputs.

Contribution

This paper presents the first application of hyper-spherical optimal transport for aligning text and image features in 3D generation, addressing high-dimensional challenges with a new mathematical formulation and neural network implementation.

Findings

Outperforms state-of-the-art methods in text-to-3D generation

Achieves higher semantic consistency in generated shapes

Demonstrates effectiveness through extensive experiments

Abstract

Recent CLIP-guided 3D generation methods have achieved promising results but struggle with generating faithful 3D shapes that conform with input text due to the gap between text and image embeddings. To this end, this paper proposes HOTS3D which makes the first attempt to effectively bridge this gap by aligning text features to the image features with spherical optimal transport(SOT). However, in high-dimensional situations, solving the SOT remains a challenge. To obtain the SOT map for high-dimensional features obtained from CLIP encoding of two modalities, we mathematically formulate and derive the solution based on Villani's theorem, which can directly align two hyper-sphere distributions without manifold exponential maps. Furthermore, we implement it by leveraging input convex neural networks (ICNNs) for the optimal Kantorovich potential. With the optimally mapped features, a diffusion-based generator is utilized to decode them into 3D shapes. Extensive quantitative and qualitative comparisons with state-of-the-art methods demonstrate the superiority of HOTS3D for text-to-3D generation, especially in the consistency with text semantics.