Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models

TL;DR

This paper introduces an abstraction-guided reasoning framework that separates structural inference from semantics in large language models, reducing content bias and improving deductive reasoning accuracy.

Contribution

It proposes a novel activation-space abstraction method with learned Abstractors to enhance formal reasoning robustness in LLMs by mitigating semantic interference.

Findings

Reduces content-driven errors in reasoning tasks

Improves validity-sensitive performance in cross-lingual transfer

Positions activation-level abstraction as a scalable robustness mechanism

Abstract

Large Language Models (LLMs) often struggle with deductive judgment in syllogistic reasoning, systematically conflating semantic plausibility with formal validity a phenomenon known as content effect. This bias persists even when models generate step-wise explanations, indicating that intermediate rationales may inherit the same semantic shortcuts that affect answers. Recent approaches propose mitigating this issue by increasing inference-time structural constraints, either by encouraging abstract intermediate representations or by intervening directly in the model's internal computations; however, reliably suppressing semantic interference remains an open challenge. To make formal deduction less sensitive to semantic content, we introduce a framework for abstraction-guided reasoning that explicitly separates structural inference from lexical semantics. We construct paired content-laden and abstract syllogisms and use the model's activations on abstract inputs to define an abstract reasoning space. We then learn lightweight Abstractors that, from content-conditioned residual-stream states, predict representations aligned with this space and integrate these predictions via multi-layer interventions during the forward pass. Using cross-lingual transfer as a test bed, we show that abstraction-aligned steering reduces content-driven errors and improves validity-sensitive performance. Our results position activation-level abstraction as a scalable mechanism for enhancing the robustness of formal reasoning in LLMs against semantic interference.