Cross-lingual Lifelong Learning

TL;DR

This paper introduces a new evaluation framework for cross-lingual continual learning, benchmarking algorithms on their ability to adapt to new languages sequentially while preserving knowledge, addressing challenges beyond traditional transfer learning.

Contribution

It proposes a principled evaluation paradigm for cross-lingual continual learning and benchmarks various algorithms to analyze their knowledge retention and generalization capabilities.

Findings

Certain algorithms excel at knowledge preservation.

Sequential learning poses unique challenges in multilingual settings.

Insights into balancing transfer and retention in cross-lingual models.

Abstract

The longstanding goal of multi-lingual learning has been to develop a universal cross-lingual model that can withstand the changes in multi-lingual data distributions. There has been a large amount of work to adapt such multi-lingual models to unseen target languages. However, the majority of work in this direction focuses on the standard one-hop transfer learning pipeline from source to target languages, whereas in realistic scenarios, new languages can be incorporated at any time in a sequential manner. In this paper, we present a principled Cross-lingual Continual Learning (CCL) evaluation paradigm, where we analyze different categories of approaches used to continually adapt to emerging data from different languages. We provide insights into what makes multilingual sequential learning particularly challenging. To surmount such challenges, we benchmark a representative set of cross-lingual continual learning algorithms and analyze their knowledge preservation, accumulation, and generalization capabilities compared to baselines on carefully curated datastreams. The implications of this analysis include a recipe for how to measure and balance different cross-lingual continual learning desiderata, which go beyond conventional transfer learning.