DeepStereo: Learning to Predict New Views from the World's Imagery

TL;DR

This paper introduces DeepStereo, a deep learning system that synthesizes new views from posed image sets, offering a general, end-to-end approach that produces high-quality images for complex scenes.

Contribution

It presents the first deep learning method for new view synthesis from real-world imagery, trained end-to-end to generate unseen views directly from pixels.

Findings

Successfully reproduces known test views from nearby images.

Generates high-quality images from novel viewpoints.

Applicable to various domains with minimal tuning.

Abstract

Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision, but their use in graphics problems has been limited. In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to traditional approaches which consist of multiple complex stages of processing, each of which require careful tuning and can fail in unexpected ways, our system is trained end-to-end. The pixels from neighboring views of a scene are presented to the network which then directly produces the pixels of the unseen view. The benefits of our approach include generality (we only require posed image sets and can easily apply our method to different domains), and high quality results on traditionally difficult scenes. We believe this is due to the end-to-end nature of our system which is able to plausibly generate pixels according to color, depth, and texture priors learnt automatically from the training data. To verify our method we show that it can convincingly reproduce known test views from nearby imagery. Additionally we show images rendered from novel viewpoints. To our knowledge, our work is the first to apply deep learning to the problem of new view synthesis from sets of real-world, natural imagery.