Free View Synthesis

TL;DR

This paper introduces a novel view synthesis method that generates new scene views from arbitrarily distributed input images without scene-specific optimization, leveraging SfM, MVS, and a recurrent neural network.

Contribution

It presents a scene-agnostic view synthesis approach that does not require scene-specific training or calibration, enabling free camera movement in unconstrained environments.

Findings

Successfully synthesizes novel views in real-world datasets.

Outperforms prior methods on challenging datasets.

Works without scene-specific fine-tuning.

Abstract

We present a method for novel view synthesis from input images that are freely distributed around a scene. Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts. We calibrate the input images via SfM and erect a coarse geometric scaffold via MVS. This scaffold is used to create a proxy depth map for a novel view of the scene. Based on this depth map, a recurrent encoder-decoder network processes reprojected features from nearby views and synthesizes the new view. Our network does not need to be optimized for a given scene. After training on a dataset, it works in previously unseen environments with no fine-tuning or per-scene optimization. We evaluate the presented approach on challenging real-world datasets, including Tanks and Temples, where we demonstrate successful view synthesis for the first time and substantially outperform prior and concurrent work.