Deep View Synthesis via Self-Consistent Generative Network

TL;DR

This paper introduces SCGN, a deep generative model that synthesizes novel views from multiple input images without relying on explicit geometric information, improving realism and handling large camera baselines.

Contribution

The paper proposes a self-consistent generative network with view synthesis and decomposition components that bypass geometric matching, enhancing view synthesis quality especially with large camera baselines.

Findings

Outperforms state-of-the-art methods on benchmark tasks.

Produces more photo-realistic synthesized views.

Handles large baseline camera setups effectively.

Abstract

View synthesis aims to produce unseen views from a set of views captured by two or more cameras at different positions. This task is non-trivial since it is hard to conduct pixel-level matching among different views. To address this issue, most existing methods seek to exploit the geometric information to match pixels. However, when the distinct cameras have a large baseline (i.e., far away from each other), severe geometry distortion issues would occur and the geometric information may fail to provide useful guidance, resulting in very blurry synthesized images. To address the above issues, in this paper, we propose a novel deep generative model, called Self-Consistent Generative Network (SCGN), which synthesizes novel views from the given input views without explicitly exploiting the geometric information. The proposed SCGN model consists of two main components, i.e., a View Synthesis Network (VSN) and a View Decomposition Network (VDN), both employing an Encoder-Decoder structure. Here, the VDN seeks to reconstruct input views from the synthesized novel view to preserve the consistency of view synthesis. Thanks to VDN, SCGN is able to synthesize novel views without using any geometric rectification before encoding, making it easier for both training and applications. Finally, adversarial loss is introduced to improve the photo-realism of novel views. Both qualitative and quantitative comparisons against several state-of-the-art methods on two benchmark tasks demonstrated the superiority of our approach.