Instant Continual Learning of Neural Radiance Fields

TL;DR

This paper introduces a novel continual learning framework for Neural Radiance Fields (NeRFs) that mitigates catastrophic forgetting, improves reconstruction quality, and significantly increases training speed for sequential scene data updates.

Contribution

It proposes a replay-based continual learning method with a hybrid explicit-implicit scene representation, outperforming prior approaches in quality and efficiency.

Findings

Outperforms previous methods in reconstruction quality.

Reduces training latency by an order of magnitude.

Effectively mitigates catastrophic forgetting in NeRFs.

Abstract

Neural radiance fields (NeRFs) have emerged as an effective method for novel-view synthesis and 3D scene reconstruction. However, conventional training methods require access to all training views during scene optimization. This assumption may be prohibitive in continual learning scenarios, where new data is acquired in a sequential manner and a continuous update of the NeRF is desired, as in automotive or remote sensing applications. When naively trained in such a continual setting, traditional scene representation frameworks suffer from catastrophic forgetting, where previously learned knowledge is corrupted after training on new data. Prior works in alleviating forgetting with NeRFs suffer from low reconstruction quality and high latency, making them impractical for real-world application. We propose a continual learning framework for training NeRFs that leverages replay-based methods combined with a hybrid explicit--implicit scene representation. Our method outperforms previous methods in reconstruction quality when trained in a continual setting, while having the additional benefit of being an order of magnitude faster.