Improving Chain-of-Thought for Logical Reasoning via Attention-Aware Intervention

TL;DR

This paper introduces Attention-Aware Intervention (AAI), a novel method that improves logical reasoning in large language models by modulating attention scores without external resources, enhancing performance and interpretability.

Contribution

The work presents a non-interactive, end-to-end framework that activates logical reasoning patterns within attention heads and introduces AAI for inference-time attention reweighting, improving reasoning accuracy.

Findings

AAI enhances logical reasoning across multiple benchmarks.

Attention heads aligned with logical operators can be identified and modulated.

The method incurs negligible additional computational cost.

Abstract

Modern logical reasoning with LLMs primarily relies on employing complex interactive frameworks that decompose the reasoning process into subtasks solved through carefully designed prompts or requiring external resources (e.g., symbolic solvers) to exploit their strong logical structures. While interactive approaches introduce additional overhead or depend on external components, which limit their scalability. In this work, we introduce a non-interactive, end-to-end framework for reasoning tasks, enabling reasoning to emerge within the model itself-improving generalization while preserving analyzability without any external resources. We show that introducing structural information into the few-shot prompt activates a subset of attention heads that patterns aligned with logical reasoning operators. Building on this insight, we propose Attention-Aware Intervention (AAI), an inference-time intervention method that reweights attention scores across selected heads identified by their logical patterns. AAI offers an efficient way to steer the model's reasoning toward leveraging prior knowledge through attention modulation. Extensive experiments show that AAI enhances logical reasoning performance across diverse benchmarks, and model architectures, while incurring negligible additional computational overhead. Code is available at https://github.com/phuongnm94/aai_for_logical_reasoning.