Knowledge Distillation with Adapted Weight

TL;DR

This paper introduces KD-AIF, a knowledge distillation framework that uses influence functions to weight training data, enhancing model transparency, robustness, and performance in semi-supervised learning across multiple benchmarks.

Contribution

It proposes a novel influence-based weighting method for knowledge distillation that improves transparency, robustness, and generalization of student models.

Findings

KD-AIF outperforms existing methods on multiple benchmarks.

The influence weighting improves learning efficiency and model interpretability.

Enhanced semi-supervised learning performance with better data utilization.

Abstract

Although large models have shown a strong capacity to solve large-scale problems in many areas including natural language and computer vision, their voluminous parameters are hard to deploy in a real-time system due to computational and energy constraints. Addressing this, knowledge distillation through Teacher-Student architecture offers a sustainable pathway to compress the knowledge of large models into more manageable sizes without significantly compromising performance. To enhance the robustness and interpretability of this framework, it is critical to understand how individual training data impact model performance, which is an area that remains underexplored. We propose the \textbf{Knowledge Distillation with Adaptive Influence Weight (KD-AIF)} framework which leverages influence functions from robust statistics to assign weights to training data, grounded in the four key SAFE principles: Sustainability, Accuracy, Fairness, and Explainability. This novel approach not only optimizes distillation but also increases transparency by revealing the significance of different data. The exploration of various update mechanisms within the KD-AIF framework further elucidates its potential to significantly improve learning efficiency and generalization in student models, marking a step toward more explainable and deployable Large Models. KD-AIF is effective in knowledge distillation while also showing exceptional performance in semi-supervised learning with outperforms existing baselines and methods in multiple benchmarks (CIFAR-100, CIFAR-10-4k, SVHN-1k, and GLUE).