AutoNeRF: Training Implicit Scene Representations with Autonomous Agents

TL;DR

AutoNeRF enables autonomous agents to collect data and train neural radiance fields in unseen environments, facilitating efficient scene understanding and downstream robotic tasks without manual data collection.

Contribution

This work introduces AutoNeRF, a novel autonomous data collection method for training NeRFs, reducing manual effort and enabling scene-specific adaptation for robotic applications.

Findings

NeRFs can be trained with a single exploration episode in unseen environments.

Modular exploration models outperform classical and end-to-end baselines.

AutoNeRF effectively reconstructs large-scale scenes for scene-specific adaptation.

Abstract

Implicit representations such as Neural Radiance Fields (NeRF) have been shown to be very effective at novel view synthesis. However, these models typically require manual and careful human data collection for training. In this paper, we present AutoNeRF, a method to collect data required to train NeRFs using autonomous embodied agents. Our method allows an agent to explore an unseen environment efficiently and use the experience to build an implicit map representation autonomously. We compare the impact of different exploration strategies including handcrafted frontier-based exploration, end-to-end and modular approaches composed of trained high-level planners and classical low-level path followers. We train these models with different reward functions tailored to this problem and evaluate the quality of the learned representations on four different downstream tasks: classical viewpoint rendering, map reconstruction, planning, and pose refinement. Empirical results show that NeRFs can be trained on actively collected data using just a single episode of experience in an unseen environment, and can be used for several downstream robotic tasks, and that modular trained exploration models outperform other classical and end-to-end baselines. Finally, we show that AutoNeRF can reconstruct large-scale scenes, and is thus a useful tool to perform scene-specific adaptation as the produced 3D environment models can be loaded into a simulator to fine-tune a policy of interest.