Novel View Synthesis from only a 6-DoF Camera Pose by Two-stage Networks

TL;DR

This paper introduces a novel approach for synthesizing new views from only a 6-DoF camera pose using a two-stage CNN framework, bypassing the need for reference images or 3D models.

Contribution

It presents a new paradigm for view synthesis directly from camera pose and proposes a two-stage CNN model, including a coarse generator and a refinement GAN.

Findings

Effective view synthesis from 6-DoF pose demonstrated on public datasets.

Two-stage CNN approach outperforms existing methods that require reference images.

Decoupling geometry and texture improves synthesis quality.

Abstract

Novel view synthesis is a challenging problem in computer vision and robotics. Different from the existing works, which need the reference images or 3D models of the scene to generate images under novel views, we propose a novel paradigm to this problem. That is, we synthesize the novel view from only a 6-DoF camera pose directly. Although this setting is the most straightforward way, there are few works addressing it. While, our experiments demonstrate that, with a concise CNN, we could get a meaningful parametric model that could reconstruct the correct scenery images only from the 6-DoF pose. To this end, we propose a two-stage learning strategy, which consists of two consecutive CNNs: GenNet and RefineNet. GenNet generates a coarse image from a camera pose. RefineNet is a generative adversarial network that refines the coarse image. In this way, we decouple the geometric relationship between mapping and texture detail rendering. Extensive experiments conducted on the public datasets prove the effectiveness of our method. We believe this paradigm is of high research and application value and could be an important direction in novel view synthesis.