GeLaTO: Generative Latent Textured Objects

TL;DR

This paper introduces GeLaTO, a novel method for modeling complex 3D objects with transparency and reflections by combining shape proxies with neural textures, enabling photorealistic rendering from sparse views.

Contribution

The paper proposes a compact representation combining shape proxies and neural textures, along with a joint latent space for category-level appearance and geometry interpolation.

Findings

Effective reconstruction of complex objects from sparse views

Successful application to challenging datasets like eyeglasses

Proxies can be handcrafted or generated for different categories

Abstract

Accurate modeling of 3D objects exhibiting transparency, reflections and thin structures is an extremely challenging problem. Inspired by billboards and geometric proxies used in computer graphics, this paper proposes Generative Latent Textured Objects (GeLaTO), a compact representation that combines a set of coarse shape proxies defining low frequency geometry with learned neural textures, to encode both medium and fine scale geometry as well as view-dependent appearance. To generate the proxies' textures, we learn a joint latent space allowing category-level appearance and geometry interpolation. The proxies are independently rasterized with their corresponding neural texture and composited using a U-Net, which generates an output photorealistic image including an alpha map. We demonstrate the effectiveness of our approach by reconstructing complex objects from a sparse set of views. We show results on a dataset of real images of eyeglasses frames, which are particularly challenging to reconstruct using classical methods. We also demonstrate that these coarse proxies can be handcrafted when the underlying object geometry is easy to model, like eyeglasses, or generated using a neural network for more complex categories, such as cars.