GET3D--: Learning GET3D from Unconstrained Image Collections

TL;DR

GET3D-- is a novel method that generates textured 3D shapes directly from 2D images with unknown pose and scale, using a learnable camera sampler and a unified training schedule.

Contribution

It introduces the first approach to generate textured 3D shapes from unconstrained 2D images with unknown camera parameters, integrating shape generation and camera sampling.

Findings

Generates high-quality textured 3D shapes from diverse datasets.

Accurately models 6D camera pose distribution.

Outperforms existing methods in shape quality and realism.

Abstract

The demand for efficient 3D model generation techniques has grown exponentially, as manual creation of 3D models is time-consuming and requires specialized expertise. While generative models have shown potential in creating 3D textured shapes from 2D images, their applicability in 3D industries is limited due to the lack of a well-defined camera distribution in real-world scenarios, resulting in low-quality shapes. To overcome this limitation, we propose GET3D--, the first method that directly generates textured 3D shapes from 2D images with unknown pose and scale. GET3D-- comprises a 3D shape generator and a learnable camera sampler that captures the 6D external changes on the camera. In addition, We propose a novel training schedule to stably optimize both the shape generator and camera sampler in a unified framework. By controlling external variations using the learnable camera sampler, our method can generate aligned shapes with clear textures. Extensive experiments demonstrate the efficacy of GET3D--, which precisely fits the 6D camera pose distribution and generates high-quality shapes on both synthetic and realistic unconstrained datasets.