On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning

TL;DR

This paper investigates how language-specific calibration impacts the effectiveness of pruning multilingual large language models, revealing limitations in preserving nuanced, language-agnostic knowledge crucial for diverse tasks.

Contribution

It provides the first comprehensive empirical analysis of calibration language effects in multilingual LLM pruning, highlighting limitations in current approaches.

Findings

Calibration on target language improves perplexity but not downstream tasks.

Pruning preserves language-specific features but loses nuanced, language-agnostic knowledge.

Current pruning methods have broader limitations in retaining essential internal representations.

Abstract

Recent advances in large language model (LLM) pruning have shown state-of-the-art (SotA) compression results in post-training and retraining-free settings while maintaining high predictive performance. However, previous research mainly considered calibrating based on English text, despite the multilingual nature of modern LLMs and their frequent use in non-English languages. This analysis paper conducts an in-depth investigation of the performance and internal representation changes associated with pruning multilingual language models for monolingual applications. We present the first comprehensive empirical study, comparing different calibration languages for pruning multilingual models across diverse languages, tasks, models, and SotA pruning techniques. We further analyze the latent subspaces, pruning masks, and individual neurons within pruned models. Our results reveal that while calibration on the target language effectively retains perplexity and yields high signal-to-noise ratios, it does not consistently improve downstream task performance. Further analysis of internal representations at three different levels highlights broader limitations of current pruning approaches: While they effectively preserve dominant information like language-specific features, this is insufficient to counteract the loss of nuanced, language-agnostic features that are crucial for knowledge retention and reasoning.