UnCLe: Benchmarking Unsupervised Continual Learning for Depth Completion

TL;DR

UnCLe introduces a standardized benchmark for evaluating unsupervised continual learning in depth completion, highlighting the challenges of non-stationary data distributions and catastrophic forgetting across diverse environments.

Contribution

This paper presents the first benchmark for unsupervised continual learning in depth completion, adapting existing methods to non-stationary data streams and evaluating their performance.

Findings

Unsupervised continual learning in depth completion remains an open challenge.

Models experience significant catastrophic forgetting on non-stationary data.

UnCLe provides a platform for future research in this area.

Abstract

We propose UnCLe, the first standardized benchmark for Unsupervised Continual Learning of a multimodal 3D reconstruction task: Depth completion aims to infer a dense depth map from a pair of synchronized RGB image and sparse depth map. We benchmark depth completion models under the practical scenario of unsupervised learning over continuous streams of data. While unsupervised learning of depth boasts the possibility continual learning of novel data distributions over time, existing methods are typically trained on a static, or stationary, dataset. However, when adapting to novel nonstationary distributions, they ``catastrophically forget'' previously learned information. UnCLe simulates these non-stationary distributions by adapting depth completion models to sequences of datasets containing diverse scenes captured from distinct domains using different visual and range sensors. We adopt representative methods from continual learning paradigms and translate them to enable unsupervised continual learning of depth completion. We benchmark these models across indoor and outdoor environments, and investigate the degree of catastrophic forgetting through standard quantitative metrics. We find that unsupervised continual learning of depth completion is an open problem, and we invite researchers to leverage UnCLe as a development platform.