TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition

TL;DR

TANGO is a novel method that enables photorealistic, text-driven 3D style transfer by disentangling appearance, geometry, and lighting, working effectively on arbitrary meshes without task-specific training.

Contribution

It introduces a disentangled approach using CLIP supervision and a differentiable renderer to achieve robust, photorealistic 3D stylization from text prompts without specialized datasets.

Findings

Outperforms existing methods in photorealism and consistency

Works effectively on low-quality meshes

Does not require task-specific training data

Abstract

Creation of 3D content by stylization is a promising yet challenging problem in computer vision and graphics research. In this work, we focus on stylizing photorealistic appearance renderings of a given surface mesh of arbitrary topology. Motivated by the recent surge of cross-modal supervision of the Contrastive Language-Image Pre-training (CLIP) model, we propose TANGO, which transfers the appearance style of a given 3D shape according to a text prompt in a photorealistic manner. Technically, we propose to disentangle the appearance style as the spatially varying bidirectional reflectance distribution function, the local geometric variation, and the lighting condition, which are jointly optimized, via supervision of the CLIP loss, by a spherical Gaussians based differentiable renderer. As such, TANGO enables photorealistic 3D style transfer by automatically predicting reflectance effects even for bare, low-quality meshes, without training on a task-specific dataset. Extensive experiments show that TANGO outperforms existing methods of text-driven 3D style transfer in terms of photorealistic quality, consistency of 3D geometry, and robustness when stylizing low-quality meshes. Our codes and results are available at our project webpage https://cyw-3d.github.io/tango/.