Didactic to Constructive: Turning Expert Solutions into Learnable Reasoning

TL;DR

This paper introduces DAIL, a method that transforms expert solutions into detailed reasoning traces to improve large language models' reasoning abilities efficiently, achieving significant performance gains with limited expert data.

Contribution

The paper proposes DAIL, a novel two-step approach that converts expert solutions into in-distribution reasoning traces and applies contrastive learning, enabling better reasoning with fewer expert samples.

Findings

Achieves 10-25% pass@k improvements on Qwen models.

Improves reasoning efficiency by 2x to 4x.

Enables out-of-domain generalization.

Abstract

Improving the reasoning capabilities of large language models (LLMs) typically relies either on the model's ability to sample a correct solution to be reinforced or on the existence of a stronger model able to solve the problem. However, many difficult problems remain intractable for even current frontier models, preventing the extraction of valid training signals. A promising alternative is to leverage high-quality expert human solutions, yet naive imitation of this data fails because it is fundamentally out of distribution: expert solutions are typically didactic, containing implicit reasoning gaps intended for human readers rather than computational models. Furthermore, high-quality expert solutions are expensive, necessitating generalizable sample-efficient training methods. We propose Distribution Aligned Imitation Learning (DAIL), a two-step method that bridges the distributional gap by first transforming expert solutions into detailed, in-distribution reasoning traces and then applying a contrastive objective to focus learning on expert insights and methodologies. We find that DAIL can leverage fewer than 1000 high-quality expert solutions to achieve 10-25% pass@k gains on Qwen2.5-Instruct and Qwen3 models, improve reasoning efficiency by 2x to 4x, and enable out-of-domain generalization.