Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners

TL;DR

This paper introduces distilled Mamba models that, despite reduced training data, outperform larger Transformer models in mathematical reasoning tasks under fixed inference time, enabling faster and more efficient reasoning at scale.

Contribution

The paper presents a novel distillation approach for creating faster, scalable reasoning models that outperform their Transformer teachers under fixed computational budgets.

Findings

Distilled Mamba models outperform Transformer teachers in reasoning accuracy.

Models achieve faster inference speeds on large batches and long sequences.

Scaling coverage and accuracy is possible despite initial performance drops.

Abstract

Recent advancements have demonstrated that the performance of large language models (LLMs) can be significantly enhanced by scaling computational resources at test time. A common strategy involves generating multiple Chain-of-Thought (CoT) trajectories and aggregating their outputs through various selection mechanisms. This raises a fundamental question: can models with lower complexity leverage their superior generation throughput to outperform similarly sized Transformers for a fixed computational budget? To address this question and overcome the lack of strong subquadratic reasoners, we distill pure and hybrid Mamba models from pretrained Transformers. Trained on only 8 billion tokens, our distilled models show strong performance and scaling on mathematical reasoning datasets while being much faster at inference for large batches and long sequences. Despite the zero-shot performance hit due to distillation, both pure and hybrid Mamba models can scale their coverage and accuracy performance past their Transformer teacher models under fixed time budgets, opening a new direction for scaling inference compute.