CtrlCoT: Dual-Granularity Chain-of-Thought Compression for Controllable Reasoning

TL;DR

CtrlCoT introduces a dual-granularity compression framework for chain-of-thought reasoning that reduces token usage by 30.7% while improving reasoning accuracy, balancing semantic abstraction and token-level pruning.

Contribution

It proposes a novel dual-granularity approach combining hierarchical reasoning abstraction and logic-preserving pruning with distribution alignment for efficient CoT compression.

Findings

Achieves 30.7% token reduction on MATH-500 dataset.

Outperforms baselines with 7.6 percentage points higher accuracy.

Demonstrates more reliable reasoning with compressed traces.

Abstract

Chain-of-thought (CoT) prompting improves LLM reasoning but incurs high latency and memory cost due to verbose traces, motivating CoT compression with preserved correctness. Existing methods either shorten CoTs at the semantic level, which is often conservative, or prune tokens aggressively, which can miss task-critical cues and degrade accuracy. Moreover, combining the two is non-trivial due to sequential dependency, task-agnostic pruning, and distribution mismatch. We propose \textbf{CtrlCoT}, a dual-granularity CoT compression framework that harmonizes semantic abstraction and token-level pruning through three components: Hierarchical Reasoning Abstraction produces CoTs at multiple semantic granularities; Logic-Preserving Distillation trains a logic-aware pruner to retain indispensable reasoning cues (e.g., numbers and operators) across pruning ratios; and Distribution-Alignment Generation aligns compressed traces with fluent inference-time reasoning styles to avoid fragmentation. On MATH-500 with Qwen2.5-7B-Instruct, CtrlCoT uses 30.7\% fewer tokens while achieving 7.6 percentage points higher than the strongest baseline, demonstrating more efficient and reliable reasoning. Our code will be publicly available at https://github.com/fanzhenxuan/Ctrl-CoT.