Align 3D Representation and Text Embedding for 3D Content Personalization

TL;DR

Invert3D introduces a novel framework that aligns 3D representations with text embeddings, enabling efficient and natural language-based personalization of 3D content without retraining.

Contribution

The paper proposes a new method to align 3D content with text embeddings, facilitating personalized 3D content manipulation via language prompts, bypassing costly retraining.

Findings

Effective 3D content personalization demonstrated

Alignment between 3D representations and text embeddings achieved

No retraining required for personalization

Abstract

Recent advances in NeRF and 3DGS have significantly enhanced the efficiency and quality of 3D content synthesis. However, efficient personalization of generated 3D content remains a critical challenge. Current 3D personalization approaches predominantly rely on knowledge distillation-based methods, which require computationally expensive retraining procedures. To address this challenge, we propose \textbf{Invert3D}, a novel framework for convenient 3D content personalization. Nowadays, vision-language models such as CLIP enable direct image personalization through aligned vision-text embedding spaces. However, the inherent structural differences between 3D content and 2D images preclude direct application of these techniques to 3D personalization. Our approach bridges this gap by establishing alignment between 3D representations and text embedding spaces. Specifically, we develop a camera-conditioned 3D-to-text inverse mechanism that projects 3D contents into a 3D embedding aligned with text embeddings. This alignment enables efficient manipulation and personalization of 3D content through natural language prompts, eliminating the need for computationally retraining procedures. Extensive experiments demonstrate that Invert3D achieves effective personalization of 3D content. Our work is available at: https://github.com/qsong2001/Invert3D.