Interpreting and Controlling LLM Reasoning through Integrated Policy Gradient

TL;DR

This paper introduces Integrated Policy Gradient (IPG), a novel method for interpreting and controlling LLM reasoning by attributing reasoning behaviors to internal components through outcome-based signals, improving localization and modulation of reasoning processes.

Contribution

The paper presents IPG, a new framework that enhances interpretability and control of LLM reasoning by propagating outcome signals backward through model trajectories.

Findings

IPG achieves more precise localization of reasoning components.

IPG enables reliable modulation of reasoning capabilities.

Empirical results show improved interpretability and control across models.

Abstract

Large language models (LLMs) demonstrate strong reasoning abilities in solving complex real-world problems. Yet, the internal mechanisms driving these complex reasoning behaviors remain opaque. Existing interpretability approaches targeting reasoning either identify components (e.g., neurons) correlated with special textual patterns, or rely on human-annotated contrastive pairs to derive control vectors. Consequently, current methods struggle to precisely localize complex reasoning mechanisms or capture sequential influence from model internal workings to the reasoning outputs. In this paper, built on outcome-oriented and sequential-influence-aware principles, we focus on identifying components that have sequential contribution to reasoning behavior where outcomes are cumulated by long-range effects. We propose Integrated Policy Gradient (IPG), a novel framework that attributes reasoning behaviors to model's inner components by propagating compound outcome-based signals such as post reasoning accuracy backward through model inference trajectories. Empirical evaluations demonstrate that our approach achieves more precise localization and enables reliable modulation of reasoning behaviors (e.g., reasoning capability, reasoning strength) across diverse reasoning models.