Distilling System 2 into System 1

TL;DR

This paper explores methods to distill the reasoning capabilities of System 2 techniques in large language models into more efficient System 1 responses, improving performance while reducing inference costs.

Contribution

It introduces self-supervised distillation methods to embed System 2 reasoning into System 1 outputs, enhancing efficiency and performance.

Findings

Distillation improves System 1 performance over baseline.

Reduced inference cost compared to System 2 techniques.

Effective self-supervised methods for reasoning distillation.

Abstract

Large language models (LLMs) can spend extra compute during inference to generate intermediate thoughts, which helps to produce better final responses. Since Chain-of-Thought (Wei et al., 2022), many such System 2 techniques have been proposed such as Rephrase and Respond (Deng et al., 2023a), System 2 Attention (Weston and Sukhbaatar, 2023) and Branch-Solve-Merge (Saha et al., 2023). In this work we investigate self-supervised methods to ``compile'' (distill) higher quality outputs from System 2 techniques back into LLM generations without intermediate reasoning token sequences, as this reasoning has been distilled into System 1. We show that several such techniques can be successfully distilled, resulting in improved results compared to the original System 1 performance, and with less inference cost than System 2. We posit that such System 2 distillation will be an important feature of future continually learning AI systems, enabling them to focus System 2 capabilities on the reasoning tasks that they cannot yet do well.