Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders

TL;DR

This paper uses Sparse Autoencoders to analyze and identify language-specific features in Large Language Models, revealing their role in multilingual capabilities and enabling improved language control.

Contribution

It introduces a novel SAE-based method and metric to identify language-specific features, demonstrating their impact on multilingual abilities and steering control in LLMs.

Findings

Some SAE features are strongly language-specific.

Ablating these features affects only certain languages.

Combining features enhances language control.

Abstract

The mechanisms behind multilingual capabilities in Large Language Models (LLMs) have been examined using neuron-based or internal-activation-based methods. However, these methods often face challenges such as superposition and layer-wise activation variance, which limit their reliability. Sparse Autoencoders (SAEs) offer a more nuanced analysis by decomposing the activations of LLMs into a sparse linear combination of SAE features. We introduce a novel metric to assess the monolinguality of features obtained from SAEs, discovering that some features are strongly related to specific languages. Additionally, we show that ablating these SAE features only significantly reduces abilities in one language of LLMs, leaving others almost unaffected. Interestingly, we find some languages have multiple synergistic SAE features, and ablating them together yields greater improvement than ablating individually. Moreover, we leverage these SAE-derived language-specific features to enhance steering vectors, achieving control over the language generated by LLMs. The code is publicly available at https://github.com/Aatrox103/multilingual-llm-features.