Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation

TL;DR

This paper introduces a simple knowledge distillation method to create smaller, efficient single-language transformers from multilingual models, improving performance in low-resource languages like Tagalog.

Contribution

It presents a novel distillation approach that enhances low-resource language models by leveraging multilingual transformers, with detailed analyses and ablations demonstrating its effectiveness.

Findings

Smaller models match strong baselines in benchmark tasks

Distillation improves soft-supervision for target languages

Method enhances efficiency for low-resource language processing

Abstract

In this paper, we propose the use of simple knowledge distillation to produce smaller and more efficient single-language transformers from Massively Multilingual Transformers (MMTs) to alleviate tradeoffs associated with the use of such in low-resource settings. Using Tagalog as a case study, we show that these smaller single-language models perform on-par with strong baselines in a variety of benchmark tasks in a much more efficient manner. Furthermore, we investigate additional steps during the distillation process that improves the soft-supervision of the target language, and provide a number of analyses and ablations to show the efficacy of the proposed method.