Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoning

TL;DR

Accordion-Thinking enables LLMs to self-regulate reasoning granularity through dynamic summarization, reducing memory use and maintaining accuracy, thus improving efficiency and interpretability in complex reasoning tasks.

Contribution

The paper introduces a novel self-regulation framework for reasoning steps in LLMs, combining dynamic summarization with reinforcement learning to enhance efficiency without sacrificing accuracy.

Findings

Fold inference mode reduces token dependency by summarizing reasoning steps.

Reinforcement learning narrows the accuracy gap between Fold and Unfold modes.

Achieves three times throughput on a 48GB GPU while maintaining solution quality.

Abstract

Scaling test-time compute via long Chain-of-Thought unlocks remarkable gains in reasoning capabilities, yet it faces practical limits due to the linear growth of KV cache and quadratic attention complexity. In this paper, we introduce Accordion-Thinking, an end-to-end framework where LLMs learn to self-regulate the granularity of the reasoning steps through dynamic summarization. This mechanism enables a Fold inference mode, where the model periodically summarizes its thought process and discards former thoughts to reduce dependency on historical tokens. We apply reinforcement learning to incentivize this capability further, uncovering a critical insight: the accuracy gap between the highly efficient Fold mode and the exhaustive Unfold mode progressively narrows and eventually vanishes over the course of training. This phenomenon demonstrates that the model learns to encode essential reasoning information into compact summaries, achieving effective compression of the reasoning context. Our Accordion-Thinking demonstrates that with learned self-compression, LLMs can tackle complex reasoning tasks with minimal dependency token overhead without compromising solution quality, and it achieves a three times throughput while maintaining accuracy on a 48GB GPU memory configuration, while the structured step summaries provide a human-readable account of the reasoning process.