Chain Of Thought Compression: A Theoritical Analysis

TL;DR

This paper provides a theoretical analysis of Chain-of-Thought compression in large language models, revealing the challenges of learning to internalize reasoning steps and proposing a novel framework that achieves significant speedup without performance loss.

Contribution

It introduces the first theoretical analysis of CoT compression, identifies the exponential decay of learning signals for high-order dependencies, and proposes ALiCoT to overcome these challenges.

Findings

ALiCoT achieves 54.4x speedup over explicit CoT.

Theoretical proof of exponential decay in learning signals for high-order interactions.

Empirical validation on NatBool-DAG benchmark shows improved reasoning efficiency.

Abstract

Chain-of-Thought (CoT) has unlocked advanced reasoning abilities of Large Language Models (LLMs) with intermediate steps, yet incurs prohibitive computational costs due to generation of extra tokens. Recent studies empirically show that compressing reasoning steps into latent states, or implicit CoT compression, offers a token-efficient alternative. However, the mechanism behind CoT compression remains unclear. In this paper, we provide the first theoretical analysis of the difficulty of learning to internalize intermediate reasoning steps. By introducing Order-r Interaction, we prove that the learning signal for high-order logical dependencies exponentially decays to solve irreducible problem, where skipping intermediate steps inevitably leads to high-order interaction barriers. To empirically validate this, we introduce NatBool-DAG, a challenging benchmark designed to enforce irreducible logical reasoning and eliminate semantic shortcuts. Guided by our theoretical findings, we propose ALiCoT (Aligned Implicit CoT), a novel framework that overcomes the signal decay by aligning latent token distributions with intermediate reasoning states. Experimental results demonstrate that ALiCoT successfully unlocks efficient reasoning: it achieves a 54.4x speedup while maintaining performance comparable to explicit CoT.