Think Clearly: Improving Reasoning via Redundant Token Pruning

TL;DR

This paper introduces a method to improve reasoning in large language models by identifying and removing redundant tokens based on attention scores, leading to better accuracy without additional training.

Contribution

The paper proposes a structure-aware token pruning technique that enhances reasoning performance by eliminating redundancy in the reasoning process of language models.

Findings

Significant accuracy improvements on reasoning benchmarks

Effective reduction of reasoning redundancy through token pruning

Enhanced performance on mathematical competition datasets

Abstract

Recent large language models have shown promising capabilities in long-form reasoning, following structured chains of thought before arriving at a final answer. However, we observe that these reasoning paths tend to include substantial redundancy; analyzing attention patterns reveals that attention scores are widely scattered, particularly incorrect answers exhibit greater attention sparsity. In this paper, we demonstrate that deliberately removing this redundancy in the reasoning process significantly improves performance through clear thinking, i.e., removing distraction. Specifically, we systematically identify reasoning redundancy by measuring token-level attention scores to a special end-of-thinking token, which is appended to an explicit instruction inserted to conclude each intermediate reasoning step. Furthermore, we propose structure-aware pruning that prioritizes removing tokens in low-contributing reasoning chunks over individual tokens. After evicting redundant tokens, we remove the injected end-of-thinking instruction, then resume the reasoning generation. We demonstrate that our method significantly improves overall accuracy across reasoning-intensive benchmarks without any training involved. In particular, our method shows strong performance on challenging mathematical competition benchmarks such as AIME and AMC, where reasoning redundancy is more prevalent.