Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models

TL;DR

This paper introduces DynSDPB, a model-agnostic self-distillation method for fine-tuning small language models that dynamically adjusts to improve learning without requiring complex teacher models.

Contribution

The paper proposes DynSDPB, a novel, adaptable self-distillation approach for small language models that does not depend on architectural modifications or external teachers.

Findings

Outperforms existing self-distillation methods on various benchmarks

Effective for both encoder-only and decoder-only language models

Enhances fine-tuning adaptability and accuracy

Abstract

Knowledge distillation (KD) has become a widely adopted approach for compressing large language models (LLMs) to reduce computational costs and memory footprints. However, the availability of complex teacher models is a prerequisite for running most KD pipelines. Thus, the traditional KD procedure can be unachievable or budget-unfriendly, particularly when relying on commercial LLMs like GPT4. In this regard, Self-distillation (SelfD) emerges as an advisable alternative, enabling student models to learn without teachers' guidance. Nonetheless, existing SelfD approaches for LMs often involve architectural modifications, assuming the models are open-source, which may not always be practical. In this work, we introduce a model-agnostic and task-agnostic method named dynamic SelfD from the previous minibatch (DynSDPB), which realizes current iterations' distillation from the last ones' generated logits. Additionally, to address prediction inaccuracies during the early iterations, we dynamically adjust the distillation influence and temperature values to enhance the adaptability of fine-tuning. Furthermore, DynSDPB is a novel fine-tuning policy that facilitates the seamless integration of existing self-correction and self-training techniques for small language models (SLMs) because they all require updating SLMs' parameters. We demonstrate the superior performance of DynSDPB on both encoder-only LMs (e.g., BERT model families) and decoder-only LMs (e.g., LLaMA model families), validating its effectiveness across natural language understanding (NLU) and natural language generation (NLG) benchmarks.