Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method

TL;DR

SAGE introduces an adaptive knowledge distillation method that dynamically generates synthetic training data in high-loss regions, improving efficiency and performance of compact models in NLP tasks.

Contribution

The paper presents a novel adaptive distillation framework using UMAP-based augmentation and a lightweight interface for efficient knowledge transfer.

Findings

Achieves state-of-the-art results with fewer training epochs.

Matches or surpasses baseline performance on NLP benchmarks.

Reduces computational overhead in model distillation.

Abstract

Model distillation enables the transfer of knowledge from large-scale models to compact student models, facilitating deployment in resource-constrained environments. However, conventional distillation approaches often suffer from computational overhead and limited generalization. We propose a novel adaptive distillation framework that dynamically augments training data in regions of high student model loss. Using UMAP-based dimensionality reduction and nearest neighbor sampling, our method identifies underperforming regions in the embedding space and generates targeted synthetic examples to guide student learning. To further improve efficiency, we introduce a lightweight teacher-student interface that bypasses the teacher's input layer, enabling direct distillation on vectorized representations. Experiments across standard NLP benchmarks demonstrate that our 66M-parameter student model consistently matches or surpasses established baselines, achieving 91.2% on QNLI and 92.3% on SST-2, while training with fewer epochs. These results highlight the promise of loss-aware data augmentation and vectorized distillation for efficient and effective model compression.