AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation

TL;DR

AuG-KD introduces an anchor-based mixup technique guided by uncertainty to improve out-of-domain knowledge distillation, effectively transferring relevant teacher knowledge to student models without access to original training data.

Contribution

It proposes a novel anchor-based mixup method that selectively transfers teacher knowledge aligned with the student domain in data-free settings.

Findings

Outperforms existing DFKD methods across multiple datasets

Demonstrates stability and robustness in various settings

Effectively balances OOD knowledge transfer and domain-specific learning

Abstract

Due to privacy or patent concerns, a growing number of large models are released without granting access to their training data, making transferring their knowledge inefficient and problematic. In response, Data-Free Knowledge Distillation (DFKD) methods have emerged as direct solutions. However, simply adopting models derived from DFKD for real-world applications suffers significant performance degradation, due to the discrepancy between teachers' training data and real-world scenarios (student domain). The degradation stems from the portions of teachers' knowledge that are not applicable to the student domain. They are specific to the teacher domain and would undermine students' performance. Hence, selectively transferring teachers' appropriate knowledge becomes the primary challenge in DFKD. In this work, we propose a simple but effective method AuG-KD. It utilizes an uncertainty-guided and sample-specific anchor to align student-domain data with the teacher domain and leverages a generative method to progressively trade off the learning process between OOD knowledge distillation and domain-specific information learning via mixup learning. Extensive experiments in 3 datasets and 8 settings demonstrate the stability and superiority of our approach. Code available at https://github.com/IshiKura-a/AuG-KD .