TexTailor: Customized Text-aligned Texturing via Effective Resampling

TL;DR

TexTailor is a new method that improves the consistency of object textures generated from text descriptions by integrating previous textures, fine-tuning a depth-aware diffusion model, and adaptively selecting camera viewpoints based on object geometry.

Contribution

It introduces a resampling scheme for texture integration, a performance preservation loss for limited training data, and adaptive camera positioning to enhance view-consistent text-to-texture synthesis.

Findings

Outperforms state-of-the-art in view-consistent texture synthesis

Effective in generating high-fidelity, texture-aligned images

Demonstrates robustness on Objaverse and ShapeNet datasets

Abstract

We present TexTailor, a novel method for generating consistent object textures from textual descriptions. Existing text-to-texture synthesis approaches utilize depth-aware diffusion models to progressively generate images and synthesize textures across predefined multiple viewpoints. However, these approaches lead to a gradual shift in texture properties across viewpoints due to (1) insufficient integration of previously synthesized textures at each viewpoint during the diffusion process and (2) the autoregressive nature of the texture synthesis process. Moreover, the predefined selection of camera positions, which does not account for the object's geometry, limits the effective use of texture information synthesized from different viewpoints, ultimately degrading overall texture consistency. In TexTailor, we address these issues by (1) applying a resampling scheme that repeatedly integrates information from previously synthesized textures within the diffusion process, and (2) fine-tuning a depth-aware diffusion model on these resampled textures. During this process, we observed that using only a few training images restricts the model's original ability to generate high-fidelity images aligned with the conditioning, and therefore propose an performance preservation loss to mitigate this issue. Additionally, we improve the synthesis of view-consistent textures by adaptively adjusting camera positions based on the object's geometry. Experiments on a subset of the Objaverse dataset and the ShapeNet car dataset demonstrate that TexTailor outperforms state-of-the-art methods in synthesizing view-consistent textures. The source code for TexTailor is available at https://github.com/Adios42/Textailor