Introspective Learning by Distilling Knowledge from Online Self-explanation
Jindong Gu, Zhiliang Wu, Volker Tresp

TL;DR
This paper introduces a novel introspective learning method that distills knowledge from a model's own online self-explanations, enhancing learning without external teachers or peer networks.
Contribution
It proposes a new approach to improve neural network training by leveraging self-explanations as privileged information, inspired by knowledge distillation techniques.
Findings
Models trained with introspective learning outperform standard training.
The method achieves competitive results without external teachers.
Incorporating class-similarity information enhances knowledge transfer.
Abstract
In recent years, many explanation methods have been proposed to explain individual classifications of deep neural networks. However, how to leverage the created explanations to improve the learning process has been less explored. As the privileged information, the explanations of a model can be used to guide the learning process of the model itself. In the community, another intensively investigated privileged information used to guide the training of a model is the knowledge from a powerful teacher model. The goal of this work is to leverage the self-explanation to improve the learning process by borrowing ideas from knowledge distillation. We start by investigating the effective components of the knowledge transferred from the teacher network to the student network. Our investigation reveals that both the responses in non-ground-truth classes and class-similarity information in…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsExplainable Artificial Intelligence (XAI) · Machine Learning and Data Classification · Adversarial Robustness in Machine Learning
