Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation

TL;DR

This paper introduces RESD, a novel self-distillation framework that leverages failure feedback through reflection to improve large language models, especially in rare-success scenarios.

Contribution

It proposes a reflection-based method to transform failure feedback into corrective supervision, enhancing learning efficiency and performance in low-success regimes.

Findings

RESD outperforms standard self-distillation baselines in multiple tasks.

It achieves faster early-stage improvement with fewer samples.

RESD enables effective learning even with rare successes.

Abstract

Enabling Large Language Models (LLMs) to continuously improve from environmental interactions is a central challenge in post-training. While on-policy self-distillation offers a promising paradigm, existing methods predominantly treat environmental feedback as a passive conditioning signal. Consequently, they heavily rely on successful demonstrations and struggle to learn in rare-success regimes. To bridge this gap, we introduce Reflection-Enhanced Self-Distillation (RESD), a framework that transforms raw failure feedback into an active source of corrective supervision. Instead of passively appending feedback, RESD interprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook to preserve reusable lessons across training steps. The enriched context enables the self-teacher to provide actionable token-level supervision even in the absence of successful rollouts. Empirical evaluations on multiple continual learning tasks demonstrate that RESD substantially outperforms standard self-distillation baselines. Furthermore, RESD achieves significantly faster early-stage improvement than GRPO with $8\times$ samples using only a single rollout per prompt, highlighting its superior interaction efficiency.