What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations

TL;DR

This paper investigates how the multilingual large language model Aya-23 internally represents and processes multiple languages, revealing language activation patterns, neuron specialization, and the influence of typological relations on internal representations.

Contribution

It provides the first detailed analysis of Aya-23's internal language representations, highlighting differences from monolingual models and the effects of multilingual training on neuron specialization.

Findings

Aya-23 activates typologically related languages during translation

Neuron activation patterns in code-mixed inputs depend on base language

Language-specific neurons are concentrated in final layers of Aya-23

Abstract

Large language models (LLMs) excel at multilingual tasks, yet their internal language processing remains poorly understood. We analyze how Aya-23-8B, a decoder-only LLM trained on balanced multilingual data, handles code-mixed, cloze, and translation tasks compared to predominantly monolingual models like Llama 3 and Chinese-LLaMA-2. Using logit lens and neuron specialization analyses, we find: (1) Aya-23 activates typologically related language representations during translation, unlike English-centric models that rely on a single pivot language; (2) code-mixed neuron activation patterns vary with mixing rates and are shaped more by the base language than the mixed-in one; and (3) Aya-23's languagespecific neurons for code-mixed inputs concentrate in final layers, diverging from prior findings on decoder-only models. Neuron overlap analysis further shows that script similarity and typological relations impact processing across model types. These findings reveal how multilingual training shapes LLM internals and inform future cross-lingual transfer research.