An Investigation of the Combination of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understanding

TL;DR

This paper explores combining rehearsal and knowledge distillation techniques to improve continual learning in spoken language understanding, addressing catastrophic forgetting in non-stationary data streams.

Contribution

It introduces a novel approach combining feature-level and predictions-level knowledge distillation for speech tasks, with comprehensive analysis and low-resource device considerations.

Findings

Combining feature-level and predictions-level KDs yields the best performance.

Rehearsal memory size impacts the effectiveness of continual learning.

Approach is effective for low-resource devices.

Abstract

Continual learning refers to a dynamical framework in which a model receives a stream of non-stationary data over time and must adapt to new data while preserving previously acquired knowledge. Unluckily, neural networks fail to meet these two desiderata, incurring the so-called catastrophic forgetting phenomenon. Whereas a vast array of strategies have been proposed to attenuate forgetting in the computer vision domain, for speech-related tasks, on the other hand, there is a dearth of works. In this paper, we consider the joint use of rehearsal and knowledge distillation (KD) approaches for spoken language understanding under a class-incremental learning scenario. We report on multiple KD combinations at different levels in the network, showing that combining feature-level and predictions-level KDs leads to the best results. Finally, we provide an ablation study on the effect of the size of the rehearsal memory that corroborates the efficacy of our approach for low-resource devices.