Fast Quiet-STaR: Thinking Without Thought Tokens

TL;DR

This paper introduces Fast Quiet STaR, a more efficient reasoning framework for large language models that reduces inference overhead while maintaining or improving reasoning accuracy through curriculum learning and reinforcement fine-tuning.

Contribution

It proposes a novel training and inference approach that internalizes token-level reasoning, significantly improving efficiency and accuracy over previous Quiet STaR methods.

Findings

Fast Quiet STaR outperforms Quiet STaR in accuracy under the same inference time.

Fast Quiet-STaR NTP achieves 9% and 5.7% accuracy improvements on two benchmarks.

The method maintains inference latency while enhancing reasoning performance.

Abstract

Large Language Models (LLMs) have achieved impressive performance across a range of natural language processing tasks. However, recent advances demonstrate that further gains particularly in complex reasoning tasks require more than merely scaling up model sizes or training data. One promising direction is to enable models to think during the reasoning process. Recently, Quiet STaR significantly improves reasoning by generating token-level thought traces, but incurs substantial inference overhead. In this work, we propose Fast Quiet STaR, a more efficient reasoning framework that preserves the benefits of token-level reasoning while reducing computational cost. Our method introduces a curriculum learning based training strategy that gradually reduces the number of thought tokens, enabling the model to internalize more abstract and concise reasoning processes. We further extend this approach to the standard Next Token Prediction (NTP) setting through reinforcement learning-based fine-tuning, resulting in Fast Quiet-STaR NTP, which eliminates the need for explicit thought token generation during inference. Experiments on four benchmark datasets with Mistral 7B and Qwen2.5 7B demonstrate that Fast Quiet-STaR consistently outperforms Quiet-STaR in terms of average accuracy under the same inference time budget. Notably, Fast Quiet-STaR NTP achieves an average accuracy improvement of 9\% on Mistral 7B and 5.7\% on Qwen2.5 7B, while maintaining the same inference latency. Our code will be available at https://github.com/huangwei200012/Fast-Quiet-STaR.