Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms

TL;DR

This paper investigates how large language models internally handle arithmetic reasoning, revealing a layered division of labor and differences in processing between proficient and less proficient models.

Contribution

It provides a detailed analysis of internal mechanisms in LLMs during arithmetic tasks, highlighting the roles of attention and MLP modules in reasoning.

Findings

Proficient models show a division of labor between attention and MLP modules.

Correct arithmetic results are generated only in the final layers.

Proficient models process challenging tasks in a reasoning-like manner.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities, yet their internal mechanisms for handling reasoning-intensive tasks remain underexplored. To advance the understanding of model-internal processing mechanisms, we present an investigation of how LLMs perform arithmetic operations by examining internal mechanisms during task execution. Using early decoding, we trace how next-token predictions are constructed across layers. Our experiments reveal that while the models recognize arithmetic tasks early, correct result generation occurs only in the final layers. Notably, models proficient in arithmetic exhibit a clear division of labor between attention and MLP modules, where attention propagates input information and MLP modules aggregate it. This division is absent in less proficient models. Furthermore, successful models appear to process more challenging arithmetic tasks functionally, suggesting reasoning capabilities beyond factual recall.