Im2SurfTex: Surface Texture Generation via Neural Backprojection of Multi-View Images

TL;DR

Im2SurfTex introduces a neural method for generating seamless, high-quality textures for 3D shapes by aggregating multi-view images with attention mechanisms and surface-aware positional encodings.

Contribution

The paper proposes a novel neural backprojection approach that enhances texture coherency and continuity in 3D shape texturing, surpassing traditional blending methods.

Findings

Improved texture quality with fewer seams and artifacts.

Superior high-resolution texture generation performance.

Effective use of neural attention and surface-aware encodings.

Abstract

We present Im2SurfTex, a method that generates textures for input 3D shapes by learning to aggregate multi-view image outputs produced by 2D image diffusion models onto the shapes' texture space. Unlike existing texture generation techniques that use ad hoc backprojection and averaging schemes to blend multiview images into textures, often resulting in texture seams and artifacts, our approach employs a trained neural module to boost texture coherency. The key ingredient of our module is to leverage neural attention and appropriate positional encodings of image pixels based on their corresponding 3D point positions, normals, and surface-aware coordinates as encoded in geodesic distances within surface patches. These encodings capture texture correlations between neighboring surface points, ensuring better texture continuity. Experimental results show that our module improves texture quality, achieving superior performance in high-resolution texture generation.