Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs

TL;DR

This paper presents a method combining multilingual multitask training, data augmentation, and model ensembling to improve translation quality for low-resource language pairs, achieving significant BLEU score gains.

Contribution

It introduces an ensembling approach of distilled models from multi-task teachers specifically designed for low-resource language translation.

Findings

70% BLEU improvement for English-Hausa translation

25% BLEU improvement for Bengali-Hindi and Xhosa-Zulu translations

Effective use of multitask learning and data augmentation

Abstract

This paper describes our submission to the constrained track of WMT21 shared news translation task. We focus on the three relatively low resource language pairs Bengali to and from Hindi, English to and from Hausa, and Xhosa to and from Zulu. To overcome the limitation of relatively low parallel data we train a multilingual model using a multitask objective employing both parallel and monolingual data. In addition, we augment the data using back translation. We also train a bilingual model incorporating back translation and knowledge distillation then combine the two models using sequence-to-sequence mapping. We see around 70% relative gain in BLEU point for English to and from Hausa, and around 25% relative improvements for both Bengali to and from Hindi, and Xhosa to and from Zulu compared to bilingual baselines.