Neural Radiance Field Codebooks
Matthew Wallingford, Aditya Kusupati, Alex Fang, Vivek Ramanujan,, Aniruddha Kembhavi, Roozbeh Mottaghi, Ali Farhadi

TL;DR
Neural Radiance Field Codebooks (NRC) introduce a scalable, object-centric representation learning method for complex scenes, enabling better transfer to downstream tasks like navigation, segmentation, and depth ordering.
Contribution
NRC is a novel approach that learns object codes for scene reconstruction, discovering transferable visual and geometric patterns across scenes.
Findings
Outperforms existing methods in object navigation success rate by 3.1%.
Achieves 29% relative improvement in unsupervised segmentation.
Improves depth ordering accuracy by 5.5% in THOR.
Abstract
Compositional representations of the world are a promising step towards enabling high-level scene understanding and efficient transfer to downstream tasks. Learning such representations for complex scenes and tasks remains an open challenge. Towards this goal, we introduce Neural Radiance Field Codebooks (NRC), a scalable method for learning object-centric representations through novel view reconstruction. NRC learns to reconstruct scenes from novel views using a dictionary of object codes which are decoded through a volumetric renderer. This enables the discovery of reoccurring visual and geometric patterns across scenes which are transferable to downstream tasks. We show that NRC representations transfer well to object navigation in THOR, outperforming 2D and 3D representation learning methods by 3.1% success rate. We demonstrate that our approach is able to perform unsupervised…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsRobotics and Sensor-Based Localization · Advanced Neural Network Applications · Domain Adaptation and Few-Shot Learning
