Is Continual Learning Ready for Real-world Challenges?

TL;DR

This paper highlights the gap between current continual learning research and real-world applications, demonstrating that existing methods perform poorly under realistic protocols using a new 3D semantic segmentation benchmark, and advocates for more practical evaluation standards.

Contribution

The paper introduces a new benchmark, OCL-3DSS, and emphasizes the need for realistic evaluation protocols to better assess continual learning methods in real-world scenarios.

Findings

Existing methods perform poorly under realistic protocols

Current evaluation protocols do not reflect real-world challenges

All methods deviate significantly from offline training upper bound

Abstract

Despite continual learning's long and well-established academic history, its application in real-world scenarios remains rather limited. This paper contends that this gap is attributable to a misalignment between the actual challenges of continual learning and the evaluation protocols in use, rendering proposed solutions ineffective for addressing the complexities of real-world setups. We validate our hypothesis and assess progress to date, using a new 3D semantic segmentation benchmark, OCL-3DSS. We investigate various continual learning schemes from the literature by utilizing more realistic protocols that necessitate online and continual learning for dynamic, real-world scenarios (eg., in robotics and 3D vision applications). The outcomes are sobering: all considered methods perform poorly, significantly deviating from the upper bound of joint offline training. This raises questions about the applicability of existing methods in realistic settings. Our paper aims to initiate a paradigm shift, advocating for the adoption of continual learning methods through new experimental protocols that better emulate real-world conditions to facilitate breakthroughs in the field.