Hierarchically Gated Experts for Efficient Online Continual Learning

TL;DR

This paper introduces Hierarchically Gated Experts (HGE), a novel method for online continual learning that efficiently identifies tasks and adapts to new data streams without forgetting previous knowledge.

Contribution

The paper proposes the Hierarchically Gated Experts (HGE) framework, extending existing Gated Experts to hierarchically organize experts for improved efficiency in online continual learning.

Findings

HGE achieves comparable performance to current methods on benchmarks.

HGE demonstrates more efficient task selection and learning.

The hierarchical structure reduces computational complexity.

Abstract

Continual Learning models aim to learn a set of tasks under the constraint that the tasks arrive sequentially with no way to access data from previous tasks. The Online Continual Learning framework poses a further challenge where the tasks are unknown and instead the data arrives as a single stream. Building on existing work, we propose a method for identifying these underlying tasks: the Gated Experts (GE) algorithm, where a dynamically growing set of experts allows for new knowledge to be acquired without catastrophic forgetting. Furthermore, we extend GE to Hierarchically Gated Experts (HGE), a method which is able to efficiently select the best expert for each data sample by organising the experts into a hierarchical structure. On standard Continual Learning benchmarks, GE and HGE are able to achieve results comparable with current methods, with HGE doing so more efficiently.