MergeDistill: Merging Pre-trained Language Models using Distillation

TL;DR

MergeDistill is a framework that merges pre-trained language models through task-agnostic knowledge distillation, enabling the creation of smaller, efficient models that can outperform larger teachers by leveraging their strengths.

Contribution

The paper introduces MergeDistill, a novel method for merging pre-trained language models via distillation, addressing limitations of multilingual models and model capacity.

Findings

Merged models perform competitively or better than larger teacher models.

Teacher selection significantly impacts student model performance.

Framework enables leveraging multiple models with minimal dependencies.

Abstract

Pre-trained multilingual language models (LMs) have achieved state-of-the-art results in cross-lingual transfer, but they often lead to an inequitable representation of languages due to limited capacity, skewed pre-training data, and sub-optimal vocabularies. This has prompted the creation of an ever-growing pre-trained model universe, where each model is trained on large amounts of language or domain specific data with a carefully curated, linguistically informed vocabulary. However, doing so brings us back full circle and prevents one from leveraging the benefits of multilinguality. To address the gaps at both ends of the spectrum, we propose MergeDistill, a framework to merge pre-trained LMs in a way that can best leverage their assets with minimal dependencies, using task-agnostic knowledge distillation. We demonstrate the applicability of our framework in a practical setting by leveraging pre-existing teacher LMs and training student LMs that perform competitively with or even outperform teacher LMs trained on several orders of magnitude more data and with a fixed model capacity. We also highlight the importance of teacher selection and its impact on student model performance.