Compressed Chain of Thought: Efficient Reasoning Through Dense Representations

TL;DR

This paper introduces Compressed Chain-of-Thought (CCoT), a method for generating dense, variable-length contemplation tokens that improve reasoning accuracy in language models while reducing decoding latency.

Contribution

The paper proposes a novel framework for continuous, contentful contemplation tokens that are compressed representations of reasoning chains, applicable to existing decoder models.

Findings

CCoT improves reasoning accuracy with dense representations.

The method allows adaptive control over reasoning depth.

Experiments show enhanced performance over fixed-length tokens.

Abstract

Chain-of-thought (CoT) decoding enables language models to improve reasoning performance at the cost of high generation latency in decoding. Recent proposals have explored variants of contemplation tokens, a term we introduce that refers to special tokens used during inference to allow for extra computation. Prior work has considered fixed-length sequences drawn from a discrete set of embeddings as contemplation tokens. Here we propose Compressed Chain-of-Thought (CCoT), a framework to generate contentful and continuous contemplation tokens of variable sequence length. The generated contemplation tokens are compressed representations of explicit reasoning chains, and our method can be applied to off-the-shelf decoder language models. Through experiments, we illustrate how CCoT enables additional reasoning over dense contentful representations to achieve corresponding improvements in accuracy. Moreover, the reasoning improvements can be adaptively modified on demand by controlling the number of contemplation tokens generated.