Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency

TL;DR

This paper evaluates structured pruning on multilingual pre-trained models, revealing surprising findings about settings, algorithms, and efficiency, and introduces Dynamic Sparsification for flexible model size adaptation.

Contribution

It provides the first comprehensive analysis of structured pruning on multilingual models, highlighting counter-intuitive phenomena and proposing a novel dynamic sparsification method.

Findings

Individually pruning per language does not improve results.

The simplest pruning algorithm outperforms more complex ones.

A faster model is not necessarily smaller in size.

Abstract

Structured pruning has been extensively studied on monolingual pre-trained language models and is yet to be fully evaluated on their multilingual counterparts. This work investigates three aspects of structured pruning on multilingual pre-trained language models: settings, algorithms, and efficiency. Experiments on nine downstream tasks show several counter-intuitive phenomena: for settings, individually pruning for each language does not induce a better result; for algorithms, the simplest method performs the best; for efficiency, a fast model does not imply that it is also small. To facilitate the comparison on all sparsity levels, we present Dynamic Sparsification, a simple approach that allows training the model once and adapting to different model sizes at inference. We hope this work fills the gap in the study of structured pruning on multilingual pre-trained models and sheds light on future research.