Learning Interpretation with Explainable Knowledge Distillation

TL;DR

This paper introduces XDistillation, a novel explainable knowledge distillation method that transfers both predictive performance and explanation fidelity from teacher to student models, improving accuracy and interpretability.

Contribution

The paper presents a new explainable knowledge distillation approach using autoencoders to transfer explanation information alongside predictions.

Findings

XDistillation outperforms conventional KD in accuracy.

XDistillation produces more faithful explanations.

Models trained with XDistillation are more interpretable.

Abstract

Knowledge Distillation (KD) has been considered as a key solution in model compression and acceleration in recent years. In KD, a small student model is generally trained from a large teacher model by minimizing the divergence between the probabilistic outputs of the two. However, as demonstrated in our experiments, existing KD methods might not transfer critical explainable knowledge of the teacher to the student, i.e. the explanations of predictions made by the two models are not consistent. In this paper, we propose a novel explainable knowledge distillation model, called XDistillation, through which both the performance the explanations' information are transferred from the teacher model to the student model. The XDistillation model leverages the idea of convolutional autoencoders to approximate the teacher explanations. Our experiments shows that models trained by XDistillation outperform those trained by conventional KD methods not only in term of predictive accuracy but also faithfulness to the teacher models.