Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneck

TL;DR

This paper presents a discrete key-value bottleneck (DKVB) approach for encoder-only language models that enables efficient continual learning by reducing catastrophic forgetting and lowering computational costs.

Contribution

The paper introduces a novel DKVB architecture for NLP, including a task-independent initialization technique, and demonstrates its effectiveness across multiple continual learning scenarios.

Findings

DKVB alleviates catastrophic forgetting in NLP models.

DKVB achieves competitive performance with lower computational costs.

DKVB remains effective without task IDs in challenging scenarios.

Abstract

Continual learning remains a challenge across various natural language processing (NLP) tasks, as models updated with new training data often risk catastrophic forgetting of previously acquired knowledge. We introduce a discrete key-value bottleneck (DKVB) for encoder-only language models, enabling efficient continual learning through localized updates. Inspired by a discrete key-value bottleneck in vision, we consider new and NLP-specific challenges. We compare different bottleneck architectures for NLP and introduce a new, task-independent initialization technique for the discrete keys. We evaluate our DKVB for NLP in four continual learning scenarios and show that it alleviates catastrophic forgetting. Our experiments demonstrate that the proposed approach achieves competitive performance compared to popular continual learning methods while incurring lower computational costs. Furthermore, we show that DKVB remains effective even in challenging single-head continual learning scenarios where no task ID is provided.