On Robust Incremental Learning over Many Multilingual Steps

TL;DR

This paper presents a robust incremental learning method for multilingual models that effectively handles dozens of fine-tuning steps without needing access to previous data, suitable for privacy-sensitive scenarios.

Contribution

It introduces a novel combination of data augmentation and optimized training for incremental learning over many steps without data retention.

Findings

Effective over up to fifty training steps

Does not require access to previous training data

Suitable for privacy-constrained environments

Abstract

Recent work in incremental learning has introduced diverse approaches to tackle catastrophic forgetting from data augmentation to optimized training regimes. However, most of them focus on very few training steps. We propose a method for robust incremental learning over dozens of fine-tuning steps using data from a variety of languages. We show that a combination of data-augmentation and an optimized training regime allows us to continue improving the model even for as many as fifty training steps. Crucially, our augmentation strategy does not require retaining access to previous training data and is suitable in scenarios with privacy constraints.