Towards continually learning new languages

TL;DR

This paper proposes a method combining weight factorization and elastic weight consolidation to enable neural networks to learn new languages sequentially without catastrophic forgetting, achieving performance comparable to training on all languages simultaneously.

Contribution

It introduces a novel approach that effectively prevents catastrophic forgetting in multilingual speech recognition, allowing incremental learning of new languages.

Findings

Successfully expanded from 10 to 26 languages without forgetting

Achieved performance close to joint training on all languages

Demonstrated efficient incremental learning process

Abstract

Multilingual speech recognition with neural networks is often implemented with batch-learning, when all of the languages are available before training. An ability to add new languages after the prior training sessions can be economically beneficial, but the main challenge is catastrophic forgetting. In this work, we combine the qualities of weight factorization and elastic weight consolidation in order to counter catastrophic forgetting and facilitate learning new languages quickly. Such combination allowed us to eliminate catastrophic forgetting while still achieving performance for the new languages comparable with having all languages at once, in experiments of learning from an initial 10 languages to achieve 26 languages without catastrophic forgetting and a reasonable performance compared to training all languages from scratch.