Continual Contrastive Spoken Language Understanding

TL;DR

This paper introduces COCONUT, a continual learning method for spoken language understanding that combines experience replay and contrastive learning to effectively retain knowledge and improve discrimination in sequence-to-sequence models.

Contribution

The paper proposes COCONUT, a novel continual learning approach that integrates contrastive learning with experience replay for improved spoken language understanding.

Findings

COCONUT outperforms baseline methods on SLU datasets.

Contrastive loss enhances representation discrimination.

Combining COCONUT with decoder-side methods yields further improvements.

Abstract

Recently, neural networks have shown impressive progress across diverse fields, with speech processing being no exception. However, recent breakthroughs in this area require extensive offline training using large datasets and tremendous computing resources. Unfortunately, these models struggle to retain their previously acquired knowledge when learning new tasks continually, and retraining from scratch is almost always impractical. In this paper, we investigate the problem of learning sequence-to-sequence models for spoken language understanding in a class-incremental learning (CIL) setting and we propose COCONUT, a CIL method that relies on the combination of experience replay and contrastive learning. Through a modified version of the standard supervised contrastive loss applied only to the rehearsal samples, COCONUT preserves the learned representations by pulling closer samples from the same class and pushing away the others. Moreover, we leverage a multimodal contrastive loss that helps the model learn more discriminative representations of the new data by aligning audio and text features. We also investigate different contrastive designs to combine the strengths of the contrastive loss with teacher-student architectures used for distillation. Experiments on two established SLU datasets reveal the effectiveness of our proposed approach and significant improvements over the baselines. We also show that COCONUT can be combined with methods that operate on the decoder side of the model, resulting in further metrics improvements.