End-to-End Fine-Tuning of 3D Texture Generation using Differentiable Rewards

TL;DR

This paper introduces an end-to-end differentiable framework for 3D texture generation that incorporates human preferences via reward functions, improving quality and control over existing methods.

Contribution

It presents a novel, reinforcement-learning-free approach that embeds human feedback into 3D texture synthesis, enabling better geometry-aware and controllable texture generation.

Findings

Outperforms state-of-the-art methods in quality and preference alignment.

Introduces three novel geometry-aware reward functions.

Demonstrates versatility across natural language and geometric criteria.

Abstract

While recent 3D generative models can produce high-quality texture images, they often fail to capture human preferences or meet task-specific requirements. Moreover, a core challenge in the 3D texture generation domain is that most existing approaches rely on repeated calls to 2D text-to-image generative models, which lack an inherent understanding of the 3D structure of the input 3D mesh object. To alleviate these issues, we propose an end-to-end differentiable, reinforcement-learning-free framework that embeds human feedback, expressed as differentiable reward functions, directly into the 3D texture synthesis pipeline. By back-propagating preference signals through both geometric and appearance modules of the proposed framework, our method generates textures that respect the 3D geometry structure and align with desired criteria. To demonstrate its versatility, we introduce three novel geometry-aware reward functions, which offer a more controllable and interpretable pathway for creating high-quality 3D content from natural language. By conducting qualitative, quantitative, and user-preference evaluations against state-of-the-art methods, we demonstrate that our proposed strategy consistently outperforms existing approaches. Our implementation code is publicly available at: https://github.com/AHHHZ975/Differentiable-Texture-Learning