Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning

TL;DR

This paper introduces On-Policy Harness Self-Distillation (OPHSD), a method that internalizes harness-augmented capabilities into language models to improve complex reasoning without relying on external workflows during inference.

Contribution

The paper proposes OPHSD, a novel self-distillation approach that enhances model reasoning by internalizing harness capabilities, outperforming existing methods on reasoning tasks.

Findings

OPHSD outperforms strong baselines, e.g., +10.83% over OPSD on HMMT25.

Reattaching the harness during inference offers no benefits and can degrade performance.

Harnesses can serve as temporary training scaffolds, with benefits fed back into the base model.

Abstract

Inference-time harnesses substantially improve large language models on complex reasoning tasks. However, the intrinsic capabilities of the underlying model remain unchanged by the addition of these external workflows. To bridge this gap, we introduce \emph{On-Policy Harness Self-Distillation} (OPHSD), which employs the harness-augmented current model as a teacher for self-distillation, thereby introducing extra supervisory signals from the harness beyond training data. OPHSD internalizes task-specific harness capabilities into the student model, yielding robust generalizability and strong standalone performance across diverse reasoning tasks. Evaluated across draft--verify harness for text classification and plan--solve for mathematical reasoning tasks, OPHSD consistently outperforms strong baselines (e.g., +10.83\% over OPSD on HMMT25). Our analysis further indicates that reattaching the harness during inference yields no additional benefits and can even degrade performance, suggesting that complex harnesses need not always be permanent fixtures; instead, they can serve as temporary training scaffolds whose benefits are permanently fed back into the base model. Our code and training data are available at https://github.com/zzy1127/OPHSD-On-Policy-Harness-Self-Distillation.