Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models

TL;DR

This paper investigates the internal mechanisms of large language models in look-ahead planning, revealing how they encode future decisions and the flow of information through their components, advancing understanding of their planning capabilities.

Contribution

It provides a detailed analysis of the internal representations and information flow in LLMs during planning, highlighting how future decisions are encoded and decoded internally.

Findings

MHSA output in last token can decode decisions

MHSA mainly uses goal spans and recent steps

Short-term future decisions are encoded in middle and upper layers

Abstract

Planning, as the core module of agents, is crucial in various fields such as embodied agents, web navigation, and tool using. With the development of large language models (LLMs), some researchers treat large language models as intelligent agents to stimulate and evaluate their planning capabilities. However, the planning mechanism is still unclear. In this work, we focus on exploring the look-ahead planning mechanism in large language models from the perspectives of information flow and internal representations. First, we study how planning is done internally by analyzing the multi-layer perception (MLP) and multi-head self-attention (MHSA) components at the last token. We find that the output of MHSA in the middle layers at the last token can directly decode the decision to some extent. Based on this discovery, we further trace the source of MHSA by information flow, and we reveal that MHSA mainly extracts information from spans of the goal states and recent steps. According to information flow, we continue to study what information is encoded within it. Specifically, we explore whether future decisions have been encoded in advance in the representation of flow. We demonstrate that the middle and upper layers encode a few short-term future decisions to some extent when planning is successful. Overall, our research analyzes the look-ahead planning mechanisms of LLMs, facilitating future research on LLMs performing planning tasks.