SVS: Adversarial refinement for sparse novel view synthesis

TL;DR

This paper introduces a novel approach combining radiance fields with adversarial learning and perceptual losses to improve sparse view synthesis, especially when reference views are limited and the baseline is large.

Contribution

It proposes a new method that unifies radiance fields with adversarial refinement to hallucinate plausible scene content in challenging sparse view scenarios.

Findings

Achieves up to 60% improvement in perceptual accuracy

Effectively handles large baseline and limited reference views

Reduces artifacts like floating blobs and structural duplication

Abstract

This paper proposes Sparse View Synthesis. This is a view synthesis problem where the number of reference views is limited, and the baseline between target and reference view is significant. Under these conditions, current radiance field methods fail catastrophically due to inescapable artifacts such 3D floating blobs, blurring and structural duplication, whenever the number of reference views is limited, or the target view diverges significantly from the reference views. Advances in network architecture and loss regularisation are unable to satisfactorily remove these artifacts. The occlusions within the scene ensure that the true contents of these regions is simply not available to the model. In this work, we instead focus on hallucinating plausible scene contents within such regions. To this end we unify radiance field models with adversarial learning and perceptual losses. The resulting system provides up to 60% improvement in perceptual accuracy compared to current state-of-the-art radiance field models on this problem.