How Do LLMs Perform Two-Hop Reasoning in Context?

TL;DR

This paper investigates how large language models perform two-hop reasoning, revealing their initial failures, subsequent improvements through fine-tuning, and the internal mechanisms that emerge during training, including a structured attention process.

Contribution

It demonstrates the failure modes of LLMs on two-hop reasoning tasks, shows how fine-tuning improves performance, and uncovers the internal attention dynamics through training and minimal models.

Findings

Pre-trained LLMs often guess randomly on two-hop reasoning with distractors.

Fine-tuning leads to near-perfect accuracy and length generalization.

Structured attention mechanisms emerge during training, enabling reasoning.

Abstract

``Socrates is human. All humans are mortal. Therefore, Socrates is mortal.'' This form of argument illustrates a typical pattern of two-hop reasoning. Formally, two-hop reasoning refers to the process of inferring a conclusion by making two logical steps, each connecting adjacent concepts, such that the final conclusion depends on the integration of both steps. It is one of the most fundamental components of human reasoning and plays a crucial role in both formal logic and everyday decision-making. Despite recent progress in large language models (LLMs), we surprisingly find that they can fail at solving simple two-hop reasoning problems when distractors are present. We observe on a synthetic dataset that pre-trained LLMs often resort to random guessing among all plausible conclusions. However, after few steps of fine-tuning, models achieve near-perfect accuracy and exhibit strong length generalization. To understand the underlying mechanisms, we train a 3-layer Transformer from scratch on a synthetic two-hop reasoning task and reverse-engineer its internal information flow. We observe a clear progression in the attention logits throughout training. This pictures a sharp phase transition from an initial stage of random guessing to the emergence of a structured sequential query mechanism, where the model first retrieves the preceding and the bridge concepts in the early layers and then uses them to infer the final answer. Finally, we show that these dynamics can be captured by a minimal three-parameter attention-only network.