SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning

TL;DR

SoftCoT++ enhances reasoning in large language models by enabling diverse exploration of continuous latent thoughts through perturbation and contrastive learning, significantly improving performance across multiple benchmarks.

Contribution

It introduces SoftCoT++ which extends SoftCoT by enabling diverse reasoning paths via latent thought perturbation and contrastive learning, improving test-time scaling methods.

Findings

SoftCoT++ outperforms SoftCoT with self-consistency scaling.

It significantly boosts reasoning performance across five benchmarks.

It is compatible with conventional scaling techniques.

Abstract

Test-Time Scaling (TTS) refers to approaches that improve reasoning performance by allocating extra computation during inference, without altering the model's parameters. While existing TTS methods operate in a discrete token space by generating more intermediate steps, recent studies in Coconut and SoftCoT have demonstrated that thinking in the continuous latent space can further enhance the reasoning performance. Such latent thoughts encode informative thinking without the information loss associated with autoregressive token generation, sparking increased interest in continuous-space reasoning. Unlike discrete decoding, where repeated sampling enables exploring diverse reasoning paths, latent representations in continuous space are fixed for a given input, which limits diverse exploration, as all decoded paths originate from the same latent thought. To overcome this limitation, we introduce SoftCoT++ to extend SoftCoT to the Test-Time Scaling paradigm by enabling diverse exploration of thinking paths. Specifically, we perturb latent thoughts via multiple specialized initial tokens and apply contrastive learning to promote diversity among soft thought representations. Experiments across five reasoning benchmarks and two distinct LLM architectures demonstrate that SoftCoT++ significantly boosts SoftCoT and also outperforms SoftCoT with self-consistency scaling. Moreover, it shows strong compatibility with conventional scaling techniques such as self-consistency. Source code is available at https://github.com/xuyige/SoftCoT.