CLNeRF: Continual Learning Meets NeRF

TL;DR

CLNeRF introduces a continual learning approach for Neural Radiance Fields, enabling efficient adaptation to scenes that change over time without catastrophic forgetting, demonstrated on a new dataset and benchmarks.

Contribution

The paper presents CLNeRF, a novel continual learning method for NeRFs that handles scene changes over time without storing all past data, using generative replay and scene embeddings.

Findings

CLNeRF performs on par with models trained on all data simultaneously.

It outperforms other continual learning baselines on standard benchmarks.

The WAT dataset captures realistic scene changes in appearance and geometry.

Abstract

Novel view synthesis aims to render unseen views given a set of calibrated images. In practical applications, the coverage, appearance or geometry of the scene may change over time, with new images continuously being captured. Efficiently incorporating such continuous change is an open challenge. Standard NeRF benchmarks only involve scene coverage expansion. To study other practical scene changes, we propose a new dataset, World Across Time (WAT), consisting of scenes that change in appearance and geometry over time. We also propose a simple yet effective method, CLNeRF, which introduces continual learning (CL) to Neural Radiance Fields (NeRFs). CLNeRF combines generative replay and the Instant Neural Graphics Primitives (NGP) architecture to effectively prevent catastrophic forgetting and efficiently update the model when new data arrives. We also add trainable appearance and geometry embeddings to NGP, allowing a single compact model to handle complex scene changes. Without the need to store historical images, CLNeRF trained sequentially over multiple scans of a changing scene performs on-par with the upper bound model trained on all scans at once. Compared to other CL baselines CLNeRF performs much better across standard benchmarks and WAT. The source code, and the WAT dataset are available at https://github.com/IntelLabs/CLNeRF. Video presentation is available at: https://youtu.be/nLRt6OoDGq0?si=8yD6k-8MMBJInQPs