Causal Interventions on Causal Paths: Mapping GPT-2's Reasoning From Syntax to Semantics

TL;DR

This paper investigates how GPT-2 models causal reasoning by analyzing internal circuits, finding that syntax is processed early and semantic reasoning involves specific later-layer heads, advancing interpretability of language models.

Contribution

It introduces a method to analyze causal reasoning in GPT-2, revealing the localization of causal syntax and semantic inference within specific layers and heads.

Findings

Causal syntax is localized within the first 2-3 layers.

Certain heads in later layers are sensitive to nonsensical causal variations.

Models infer reasoning by detecting syntactic cues and focusing on semantic relationships.

Abstract

While interpretability research has shed light on some internal algorithms utilized by transformer-based LLMs, reasoning in natural language, with its deep contextuality and ambiguity, defies easy categorization. As a result, formulating clear and motivating questions for circuit analysis that rely on well-defined in-domain and out-of-domain examples required for causal interventions is challenging. Although significant work has investigated circuits for specific tasks, such as indirect object identification (IOI), deciphering natural language reasoning through circuits remains difficult due to its inherent complexity. In this work, we take initial steps to characterize causal reasoning in LLMs by analyzing clear-cut cause-and-effect sentences like "I opened an umbrella because it started raining," where causal interventions may be possible through carefully crafted scenarios using GPT-2 small. Our findings indicate that causal syntax is localized within the first 2-3 layers, while certain heads in later layers exhibit heightened sensitivity to nonsensical variations of causal sentences. This suggests that models may infer reasoning by (1) detecting syntactic cues and (2) isolating distinct heads in the final layers that focus on semantic relationships.