Do Multilingual LLMs have specialized language heads?

TL;DR

This paper investigates whether multilingual LLMs have language-specific attention heads and explores the potential for removing unnecessary heads to improve deployment efficiency without sacrificing performance.

Contribution

It is the first study to analyze language head specialization in multilingual LLMs and assess the impact of removing language-specific heads on model performance.

Findings

Multilingual LLMs exhibit some degree of language-specific attention heads.

Removing language-specific heads for certain languages can reduce model complexity.

Performance on targeted languages remains stable after removing irrelevant language heads.

Abstract

Multilingual large language models (LLMs) have gained significant popularity for their ability to process and generate text across multiple languages. However, deploying these models in production can be inefficient when only a subset of the supported languages is of interest. There has been some research conducted on identifying whether machine translation models have language-specific or language-agnostic heads, however no research has been conducted for multilingual LLMs, to the best of our knowledge, that as we know are capable of performing diverse tasks beyond just translation. This paper explores whether multilingual LLMs have specialized language attention heads for each language, and investigates the possibility of removing language-specific heads for unwanted languages without degrading performance in the targeted languages. Our findings could inform more efficient deployment strategies for multilingual LLMs, enabling reduced model complexity while maintaining high accuracy for targeted languages.