Self-Distillation Enables Continual Learning

TL;DR

This paper introduces Self-Distillation Fine-Tuning (SDFT), a novel on-policy learning method from demonstrations that improves continual learning by reducing forgetting and enhancing skill acquisition in models.

Contribution

The paper proposes SDFT, a simple yet effective on-policy distillation technique that enables continual learning from demonstrations, outperforming supervised fine-tuning in retaining and acquiring skills.

Findings

SDFT outperforms supervised fine-tuning in skill learning tasks.

SDFT significantly reduces catastrophic forgetting in sequential learning.

SDFT enables models to accumulate multiple skills over time without performance loss.

Abstract

Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations.