Entangled View-Epipolar Information Aggregation for Generalizable Neural Radiance Fields

TL;DR

EVE-NeRF introduces an entangled view-epipolar feature aggregation method that enhances the generalization of neural radiance fields by effectively integrating appearance and geometry priors, leading to state-of-the-art results.

Contribution

The paper proposes a novel entangled aggregation approach for view and epipolar features, improving 3D representation generalization in neural radiance fields.

Findings

Achieves state-of-the-art performance in view synthesis tasks.

Outperforms existing methods in 3D geometry accuracy.

Demonstrates robustness across various evaluation scenarios.

Abstract

Generalizable NeRF can directly synthesize novel views across new scenes, eliminating the need for scene-specific retraining in vanilla NeRF. A critical enabling factor in these approaches is the extraction of a generalizable 3D representation by aggregating source-view features. In this paper, we propose an Entangled View-Epipolar Information Aggregation method dubbed EVE-NeRF. Different from existing methods that consider cross-view and along-epipolar information independently, EVE-NeRF conducts the view-epipolar feature aggregation in an entangled manner by injecting the scene-invariant appearance continuity and geometry consistency priors to the aggregation process. Our approach effectively mitigates the potential lack of inherent geometric and appearance constraint resulting from one-dimensional interactions, thus further boosting the 3D representation generalizablity. EVE-NeRF attains state-of-the-art performance across various evaluation scenarios. Extensive experiments demonstate that, compared to prevailing single-dimensional aggregation, the entangled network excels in the accuracy of 3D scene geometry and appearance reconstruction. Our code is publicly available at https://github.com/tatakai1/EVENeRF.