Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations

TL;DR

Neural Feature Fusion Fields (N3F) leverage self-supervised 2D image features to train a 3D neural network, enhancing scene understanding and rendering without manual labels across various tasks and datasets.

Contribution

N3F introduces a novel 3D distillation approach that improves dense 2D features for 3D scene analysis, compatible with neural rendering methods.

Findings

Improves 2D feature-based scene understanding tasks

Enhances neural rendering with semantic information

Effective across diverse datasets and dynamic scenes

Abstract

We present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene. Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a teacher to learn a student network defined in 3D space. The 3D student network is similar to a neural radiance field that distills said features and can be trained with the usual differentiable rendering machinery. As a consequence, N3F is readily applicable to most neural rendering formulations, including vanilla NeRF and its extensions to complex dynamic scenes. We show that our method not only enables semantic understanding in the context of scene-specific neural fields without the use of manual labels, but also consistently improves over the self-supervised 2D baselines. This is demonstrated by considering various tasks, such as 2D object retrieval, 3D segmentation, and scene editing, in diverse sequences, including long egocentric videos in the EPIC-KITCHENS benchmark.