Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis

TL;DR

This paper introduces MVCGAN, a novel 3D-aware image synthesis method that enforces geometry constraints to produce multi-view consistent images with high quality, advancing the state-of-the-art in 3D-aware generation.

Contribution

The paper proposes a geometry-constrained GAN framework that explicitly models 3D geometry and stereo correspondence, improving multi-view consistency in 3D-aware image synthesis.

Findings

Achieves state-of-the-art results on three datasets.

Enforces photometric consistency between views.

Incorporates stereo mixup for better 3D reasoning.

Abstract

3D-aware image synthesis aims to generate images of objects from multiple views by learning a 3D representation. However, one key challenge remains: existing approaches lack geometry constraints, hence usually fail to generate multi-view consistent images. To address this challenge, we propose Multi-View Consistent Generative Adversarial Networks (MVCGAN) for high-quality 3D-aware image synthesis with geometry constraints. By leveraging the underlying 3D geometry information of generated images, i.e., depth and camera transformation matrix, we explicitly establish stereo correspondence between views to perform multi-view joint optimization. In particular, we enforce the photometric consistency between pairs of views and integrate a stereo mixup mechanism into the training process, encouraging the model to reason about the correct 3D shape. Besides, we design a two-stage training strategy with feature-level multi-view joint optimization to improve the image quality. Extensive experiments on three datasets demonstrate that MVCGAN achieves the state-of-the-art performance for 3D-aware image synthesis.