VMRF: View Matching Neural Radiance Fields

TL;DR

VMRF is a novel neural radiance field approach that trains without prior camera pose knowledge by matching rendered images to real images using optimal transport and pose calibration, outperforming existing methods.

Contribution

The paper introduces VMRF, a view matching NeRF that eliminates the need for camera pose initialization through a novel feature transport and pose calibration technique.

Findings

VMRF outperforms state-of-the-art methods on synthetic datasets.

VMRF achieves significant improvements on real-world datasets.

The approach effectively calibrates camera poses without prior information.

Abstract

Neural Radiance Fields (NeRF) have demonstrated very impressive performance in novel view synthesis via implicitly modelling 3D representations from multi-view 2D images. However, most existing studies train NeRF models with either reasonable camera pose initialization or manually-crafted camera pose distributions which are often unavailable or hard to acquire in various real-world data. We design VMRF, an innovative view matching NeRF that enables effective NeRF training without requiring prior knowledge in camera poses or camera pose distributions. VMRF introduces a view matching scheme, which exploits unbalanced optimal transport to produce a feature transport plan for mapping a rendered image with randomly initialized camera pose to the corresponding real image. With the feature transport plan as the guidance, a novel pose calibration technique is designed which rectifies the initially randomized camera poses by predicting relative pose transformations between the pair of rendered and real images. Extensive experiments over a number of synthetic and real datasets show that the proposed VMRF outperforms the state-of-the-art qualitatively and quantitatively by large margins.