Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data

TL;DR

Viewset Diffusion introduces a diffusion-based method that generates 3D objects from multi-view 2D data, enabling efficient, multi-solution 3D reconstructions conditioned on limited views.

Contribution

It presents a novel diffusion model that learns from viewsets to generate 3D models using only 2D supervision, supporting flexible conditioning and multiple solutions.

Findings

Effective 3D reconstruction from limited views

Supports conditioning on multiple input views

Achieves efficient, feed-forward generation

Abstract

We present Viewset Diffusion, a diffusion-based generator that outputs 3D objects while only using multi-view 2D data for supervision. We note that there exists a one-to-one mapping between viewsets, i.e., collections of several 2D views of an object, and 3D models. Hence, we train a diffusion model to generate viewsets, but design the neural network generator to reconstruct internally corresponding 3D models, thus generating those too. We fit a diffusion model to a large number of viewsets for a given category of objects. The resulting generator can be conditioned on zero, one or more input views. Conditioned on a single view, it performs 3D reconstruction accounting for the ambiguity of the task and allowing to sample multiple solutions compatible with the input. The model performs reconstruction efficiently, in a feed-forward manner, and is trained using only rendering losses using as few as three views per viewset. Project page: szymanowiczs.github.io/viewset-diffusion.