Autonomous Deep Learning: Continual Learning Approach for Dynamic Environments

TL;DR

This paper introduces Autonomous Deep Learning (ADL), a continual learning algorithm that dynamically constructs neural networks from scratch, effectively manages catastrophic forgetting, and adapts to data stream changes in lifelong environments.

Contribution

The paper presents a novel self-constructing deep learning framework with a dynamic structure, including a new network significance formula and drift detection, to improve continual learning performance.

Findings

ADL outperforms recent continual learning methods on nine data stream problems.

The network structure adapts automatically to data distribution changes.

ADL effectively balances plasticity and stability in dynamic environments.

Abstract

The feasibility of deep neural networks (DNNs) to address data stream problems still requires intensive study because of the static and offline nature of conventional deep learning approaches. A deep continual learning algorithm, namely autonomous deep learning (ADL), is proposed in this paper. Unlike traditional deep learning methods, ADL features a flexible structure where its network structure can be constructed from scratch with the absence of an initial network structure via the self-constructing network structure. ADL specifically addresses catastrophic forgetting by having a different-depth structure which is capable of achieving a trade-off between plasticity and stability. Network significance (NS) formula is proposed to drive the hidden nodes growing and pruning mechanism. Drift detection scenario (DDS) is put forward to signal distributional changes in data streams which induce the creation of a new hidden layer. The maximum information compression index (MICI) method plays an important role as a complexity reduction module eliminating redundant layers. The efficacy of ADL is numerically validated under the prequential test-then-train procedure in lifelong environments using nine popular data stream problems. The numerical results demonstrate that ADL consistently outperforms recent continual learning methods while characterizing the automatic construction of network structures.