Stable View Synthesis

TL;DR

Stable View Synthesis (SVS) is a novel end-to-end method that synthesizes realistic new views of a scene from multiple source images by leveraging a geometric scaffold and view-dependent feature aggregation, outperforming existing techniques.

Contribution

SVS introduces a differentiable, end-to-end approach that combines structure-from-motion with view-dependent feature processing for high-quality view synthesis.

Findings

Outperforms state-of-the-art methods quantitatively.

Achieves high realism in free-viewpoint videos.

Supports synthesis of view-dependent effects like specular reflection.

Abstract

We present Stable View Synthesis (SVS). Given a set of source images depicting a scene from freely distributed viewpoints, SVS synthesizes new views of the scene. The method operates on a geometric scaffold computed via structure-from-motion and multi-view stereo. Each point on this 3D scaffold is associated with view rays and corresponding feature vectors that encode the appearance of this point in the input images. The core of SVS is view-dependent on-surface feature aggregation, in which directional feature vectors at each 3D point are processed to produce a new feature vector for a ray that maps this point into the new target view. The target view is then rendered by a convolutional network from a tensor of features synthesized in this way for all pixels. The method is composed of differentiable modules and is trained end-to-end. It supports spatially-varying view-dependent importance weighting and feature transformation of source images at each point; spatial and temporal stability due to the smooth dependence of on-surface feature aggregation on the target view; and synthesis of view-dependent effects such as specular reflection. Experimental results demonstrate that SVS outperforms state-of-the-art view synthesis methods both quantitatively and qualitatively on three diverse real-world datasets, achieving unprecedented levels of realism in free-viewpoint video of challenging large-scale scenes. Code is available at https://github.com/intel-isl/StableViewSynthesis