Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?

TL;DR

This paper empirically evaluates how pre-trained multilingual models like mBART perform in translating low-resource languages, revealing their limitations and emphasizing the importance of data quality and quantity over model complexity.

Contribution

The study provides a comprehensive framework for assessing data sensitivities in multilingual translation models and offers heuristics for improving low-resource language translation.

Findings

mBART is robust to domain mismatch

Translations for unseen languages remain below 3 BLEU

Model performance heavily depends on data quality and quantity

Abstract

What can pre-trained multilingual sequence-to-sequence models like mBART contribute to translating low-resource languages? We conduct a thorough empirical experiment in 10 languages to ascertain this, considering five factors: (1) the amount of fine-tuning data, (2) the noise in the fine-tuning data, (3) the amount of pre-training data in the model, (4) the impact of domain mismatch, and (5) language typology. In addition to yielding several heuristics, the experiments form a framework for evaluating the data sensitivities of machine translation systems. While mBART is robust to domain differences, its translations for unseen and typologically distant languages remain below 3.0 BLEU. In answer to our title's question, mBART is not a low-resource panacea; we therefore encourage shifting the emphasis from new models to new data.