Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?

TL;DR

This paper investigates how large language models acquire new languages during training, identifying functional specializations and proposing a layer-wise fine-tuning heuristic that improves efficiency and performance.

Contribution

It introduces CogSym, a layer-wise heuristic for effective language adaptation by fine-tuning only specific model layers, reducing computational costs.

Findings

Perceptual and productive specialization emerge in different model regions.

Fine-tuning 25% of outer layers achieves near full fine-tuning performance.

CogSym performs comparably to adapter methods like LoRA.

Abstract

Adapting large language models (LLMs) to new languages is an expensive and opaque process. Understanding how language models acquire new languages and multilingual abilities is key to achieve efficient adaptation. Prior work on multilingual interpretability research focuses primarily on how trained models process multilingual instructions, leaving unexplored the mechanisms through which they acquire new languages during training. We investigate these training dynamics on decoder-only transformers through the lens of two functional cognitive specializations: language perception (input comprehension) and production (output generation). Through experiments on low-resource languages, we demonstrate how perceptual and productive specialization emerges in different regions of a language model by running layer ablation sweeps from the model's input and output directions. Based on the observed specialization patterns, we propose CogSym, a layer-wise heuristic that enables effective adaptation by exclusively fine-tuning a few early and late layers. We show that tuning only the 25% outermost layers achieves downstream task performance within 2-3% deviation from the full fine-tuning baseline. CogSym yields consistent performance with adapter methods such as LoRA, showcasing generalization beyond full fine-tuning. These findings provide insights to better understand how LLMs learn new languages and push toward accessible and inclusive language modeling.