JacobiNeRF: NeRF Shaping with Mutual Information Gradients

TL;DR

JacobiNeRF introduces a novel training approach for neural radiance fields that encodes semantic correlations by aligning Jacobians to maximize mutual information, improving label propagation and scene understanding.

Contribution

The paper presents a method to regularize NeRF training by aligning Jacobians to capture mutual information, enabling semantic and instance segmentation with fewer annotations.

Findings

Enhanced label propagation in sparse annotation settings.

More efficient semantic segmentation compared to traditional NeRFs.

Ability to perform scene modifications using mutual information shaping.

Abstract

We propose a method that trains a neural radiance field (NeRF) to encode not only the appearance of the scene but also semantic correlations between scene points, regions, or entities -- aiming to capture their mutual co-variation patterns. In contrast to the traditional first-order photometric reconstruction objective, our method explicitly regularizes the learning dynamics to align the Jacobians of highly-correlated entities, which proves to maximize the mutual information between them under random scene perturbations. By paying attention to this second-order information, we can shape a NeRF to express semantically meaningful synergies when the network weights are changed by a delta along the gradient of a single entity, region, or even a point. To demonstrate the merit of this mutual information modeling, we leverage the coordinated behavior of scene entities that emerges from our shaping to perform label propagation for semantic and instance segmentation. Our experiments show that a JacobiNeRF is more efficient in propagating annotations among 2D pixels and 3D points compared to NeRFs without mutual information shaping, especially in extremely sparse label regimes -- thus reducing annotation burden. The same machinery can further be used for entity selection or scene modifications.