Generalizable Novel-View Synthesis using a Stereo Camera

TL;DR

This paper introduces StereoNeRF, a novel framework that integrates stereo matching into NeRF-based view synthesis, enabling high-quality, generalizable novel-view synthesis from multi-view stereo-camera images.

Contribution

The paper presents StereoNeRF, the first approach to incorporate stereo matching into NeRF for improved generalizable view synthesis from stereo-camera images.

Findings

StereoNeRF outperforms previous methods in view synthesis quality.

Introduction of the StereoNVS dataset with diverse real and synthetic scenes.

Effective use of stereo features and depth-guided plane-sweeping enhances synthesis results.

Abstract

In this paper, we propose the first generalizable view synthesis approach that specifically targets multi-view stereo-camera images. Since recent stereo matching has demonstrated accurate geometry prediction, we introduce stereo matching into novel-view synthesis for high-quality geometry reconstruction. To this end, this paper proposes a novel framework, dubbed StereoNeRF, which integrates stereo matching into a NeRF-based generalizable view synthesis approach. StereoNeRF is equipped with three key components to effectively exploit stereo matching in novel-view synthesis: a stereo feature extractor, a depth-guided plane-sweeping, and a stereo depth loss. Moreover, we propose the StereoNVS dataset, the first multi-view dataset of stereo-camera images, encompassing a wide variety of both real and synthetic scenes. Our experimental results demonstrate that StereoNeRF surpasses previous approaches in generalizable view synthesis.