What Do Compressed Multilingual Machine Translation Models Forget?

TL;DR

This paper investigates how compression techniques affect multilingual neural machine translation models, revealing significant performance drops for under-represented languages and increased biases, despite minimal changes in average metrics.

Contribution

It provides a comprehensive analysis of the impact of compression on multilingual translation models, highlighting bias amplification and performance degradation for low-resource languages.

Findings

Performance drops significantly for under-represented languages after compression.

Compression amplifies intrinsic gender and semantic biases.

Noisy memorization removal can improve medium-resource language performance.

Abstract

Recently, very large pre-trained models achieve state-of-the-art results in various natural language processing (NLP) tasks, but their size makes it more challenging to apply them in resource-constrained environments. Compression techniques allow to drastically reduce the size of the models and therefore their inference time with negligible impact on top-tier metrics. However, the general performance averaged across multiple tasks and/or languages may hide a drastic performance drop on under-represented features, which could result in the amplification of biases encoded by the models. In this work, we assess the impact of compression methods on Multilingual Neural Machine Translation models (MNMT) for various language groups, gender, and semantic biases by extensive analysis of compressed models on different machine translation benchmarks, i.e. FLORES-101, MT-Gender, and DiBiMT. We show that the performance of under-represented languages drops significantly, while the average BLEU metric only slightly decreases. Interestingly, the removal of noisy memorization with compression leads to a significant improvement for some medium-resource languages. Finally, we demonstrate that compression amplifies intrinsic gender and semantic biases, even in high-resource languages. Code: https://github.com/alirezamshi/bias-compressedMT