Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models

TL;DR

This survey explores the internal mechanisms enabling multi-step reasoning in Large Language Models, focusing on understanding how they perform implicit and explicit reasoning processes and proposing future research directions.

Contribution

It provides a comprehensive framework analyzing the internal reasoning mechanisms of LLMs, which is a novel focus compared to existing engineering-centric surveys.

Findings

Organized around seven key research questions.

Highlights five future research directions.

Provides insights into implicit and explicit reasoning processes.

Abstract

Large Language Models (LLMs) have demonstrated remarkable abilities to solve problems requiring multiple reasoning steps, yet the internal mechanisms enabling such capabilities remain elusive. Unlike existing surveys that primarily focus on engineering methods to enhance performance, this survey provides a comprehensive overview of the mechanisms underlying LLM multi-step reasoning. We organize the survey around a conceptual framework comprising seven interconnected research questions, from how LLMs execute implicit multi-hop reasoning within hidden activations to how verbalized explicit reasoning remodels the internal computation. Finally, we highlight five research directions for future mechanistic studies.