TUVF: Learning Generalizable Texture UV Radiance Fields
An-Chieh Cheng, Xueting Li, Sifei Liu, Xiaolong Wang
TL;DR
This paper introduces TUVF, a novel method for generating high-fidelity, controllable textures for 3D models by learning in UV space, enabling transferability and improved texture editing capabilities.
Contribution
We propose Texture UV Radiance Fields (TUVF), a new approach that learns textures in UV space, allowing for disentangled, transferable, and controllable texture generation for 3D assets.
Findings
Achieves realistic texture synthesis on synthetic and real-world datasets.
Outperforms state-of-the-art methods in texture control and editing.
Enables transfer of textures across shapes within the same category.
Abstract
Textures are a vital aspect of creating visually appealing and realistic 3D models. In this paper, we study the problem of generating high-fidelity texture given shapes of 3D assets, which has been relatively less explored compared with generic 3D shape modeling. Our goal is to facilitate a controllable texture generation process, such that one texture code can correspond to a particular appearance style independent of any input shapes from a category. We introduce Texture UV Radiance Fields (TUVF) that generate textures in a learnable UV sphere space rather than directly on the 3D shape. This allows the texture to be disentangled from the underlying shape and transferable to other shapes that share the same UV space, i.e., from the same category. We integrate the UV sphere space with the radiance field, which provides a more efficient and accurate representation of textures than…
Peer Reviews
Decision·ICLR 2024 poster
The paper is well written and easy to understand. The method sections constraints useful figures and clear structure. The research problem is important since a general formulation of UV mapping is an open topic. The experimental sections contain many insights and support claims, e.g.Table 4 the ablation on the texture mapping network.
Even though the method requires a GT shape as input, the rendered shapes appear to have over-smoothed regions, e.g. the mirrors of the cars. It is unclear how well the learned UV correspondence preserves surface areas in the UV space.
The paper is overall clear with a good structure. Readers can follow the text easily. The paper shows a lot of results and comparisons, which makes the pipeline more convincing. Details of the network architecture are given in the supp, making reproduction easier.
The biggest issue is that the paper is not novel. Most part in the paper has been explored in previous papers. Though well combined, it only produces OK results instead of exciting results. For example, recent PointUVDiffusion (Texture Generation on 3D Meshes with Point-UV Diffusion) can generate very realistic textures for a 3D shape. Though this paper can support more things like texture transfer via its UV mapper. But actually, correspondence between shapes can also be obtained by postpro
The paper does a good job presenting what is done with helpful visuals and is easy to follow. It also tackles an interesting problem where one needs to learn a canonical space for a category shapes and simultaneously put textures onto the shapes, without paired 3D-2D data. By leveraging autoencoding and adversarial learning, the model learns meaningful patterns/correlations without explicit, direct supervision. Dense correspondence emerging from autoencoding is also interesting and makes sense
I have two significant concerns that need addressing before I can consider raising my ratings. While I understand how canonical surface autoencoding eventually leads to a mapping between a given shape and the canonical sphere, for rigid shapes like cars and airplanes, one can project the shape onto an enclosing sphere, e.g., via raycasting from the sphere to the shape, to obtain similar mappings -- "similar" as in all cars' front bumpers mostly map to the same location on the canonical sphere.
Videos
Taxonomy
Topics3D Shape Modeling and Analysis · Computer Graphics and Visualization Techniques · Generative Adversarial Networks and Image Synthesis