Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models

TL;DR

This paper investigates the role of multi-head self-attention in multilingual large language models, introduces a method to identify important attention heads, and demonstrates how this understanding can improve multilingual performance and interpretability.

Contribution

It proposes LAHIS, a novel method to identify important attention heads for multilingual tasks, and introduces a lightweight adaptation to enhance multilingual capabilities of LLMs.

Findings

Identified language-specific and language-general attention heads.

Using head importance scores improves cross-lingual transfer.

A soft head mask improves XQuAD accuracy with minimal parameters.

Abstract

Large language models (LLMs) increasingly support multilingual understanding and generation. Meanwhile, efforts to interpret their internal mechanisms have emerged, offering insights to enhance multilingual performance. While multi-head self-attention (MHA) has proven critical in many areas, its role in multilingual capabilities remains underexplored. In this work, we study the contribution of MHA in supporting multilingual processing in LLMs. We propose Language Attention Head Importance Scores (LAHIS), an effective and efficient method that identifies attention head importance for multilingual capabilities via a single forward and backward pass through the LLM. Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads. Language-specific heads enable cross-lingual attention transfer to guide the model toward target language contexts and mitigate off-target language generation issue, contributing to addressing challenges in multilingual LLMs. We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy. Overall, our work enhances both the interpretability and multilingual capabilities of LLMs from the perspective of MHA.