Hindsight Hint Distillation: Scaffolded Reasoning for SWE Agents from CoT-free Answers

TL;DR

Hindsight Hint Distillation (HHD) enables training reasoning agents using only question-answer pairs, synthesizing guidance from model failures to improve long-horizon task performance without requiring explicit chain-of-thought annotations.

Contribution

HHD introduces a novel method to learn reasoning strategies from CoT-free data by using model-generated hindsight hints, reducing annotation costs and improving out-of-distribution generalization.

Findings

HHD outperforms RFT and trajectory-synthesis baselines by 8% on SWE-bench Verified.

HHD's reasoning strategies generalize well to out-of-distribution tasks.

HHD achieves significant improvements without explicit CoT annotations.

Abstract

Solving complex long-horizon tasks requires strong planning and reasoning capabilities. Although datasets with explicit chain-of-thought (CoT) rationales can substantially benefit learning, they are costly to obtain. To address this challenge, we propose Hindsight Hint Distillation (HHD), which only requires easy-to-obtain question-answer pairs without CoT annotations. Inspired by how human teachers use student mistakes to provide targeted guidance, HHD synthesizes hindsight hints from the model's own failed self-rollouts and uses them to scaffold on-policy rollouts that successfully complete the tasks. The model then self-distills these scaffolded trajectories and generalizes to new problems without hint guidance. Experiments show that HHD significantly outperforms iterative RFT and trajectory-synthesis baselines, achieving an absolute improvement of 8\% on SWE-bench Verified, while all baselines improve by only around 2\%. Notably, the reasoning strategies induced by HHD generalize effectively to out-of-distribution tasks, yielding the largest gains on SWE-bench Multilingual despite no training on multilingual data. These results demonstrate that HHD can effectively synthesize expert-like reasoning from CoT-free data and substantially improve long-horizon performance.