Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages

TL;DR

This paper demonstrates that sparse autoencoders can identify language-specific features in large language models, revealing their influence on multilingual performance and enabling interpretable language identification.

Contribution

Introduces SAE-LAPE, a novel method to detect language-specific features in LLMs using feature activation probability, enhancing interpretability and understanding of multilingual mechanisms.

Findings

Many language-specific features appear in middle to final layers.

Features are interpretable and influence multilingual performance.

Language identification accuracy is comparable to fastText.

Abstract

Understanding the multilingual mechanisms of large language models (LLMs) provides insight into how they process different languages, yet this remains challenging. Existing studies often focus on individual neurons, but their polysemantic nature makes it difficult to isolate language-specific units from cross-lingual representations. To address this, we explore sparse autoencoders (SAEs) for their ability to learn monosemantic features that represent concrete and abstract concepts across languages in LLMs. While some of these features are language-independent, the presence of language-specific features remains underexplored. In this work, we introduce SAE-LAPE, a method based on feature activation probability, to identify language-specific features within the feed-forward network. We find that many such features predominantly appear in the middle to final layers of the model and are interpretable. These features influence the model's multilingual performance and language output and can be used for language identification with performance comparable to fastText along with more interpretability. Our code and complete figures are available at https://github.com/LyzanderAndrylie/language-specific-features