Improved Knowledge Distillation via Adversarial Collaboration

TL;DR

This paper introduces ACKD, a novel knowledge distillation method that uses adversarial collaboration and auxiliary learners to enhance student model capacity, effectively reducing the capacity gap issue.

Contribution

It proposes a new adversarial collaborative learning framework with auxiliary learners to improve knowledge distillation performance.

Findings

ACKD outperforms existing methods on four classification tasks.

The use of adversarial collaborative module enhances student capacity.

Auxiliary learners contribute to better knowledge transfer.

Abstract

Knowledge distillation has become an important approach to obtain a compact yet effective model. To achieve this goal, a small student model is trained to exploit the knowledge of a large well-trained teacher model. However, due to the capacity gap between the teacher and the student, the student's performance is hard to reach the level of the teacher. Regarding this issue, existing methods propose to reduce the difficulty of the teacher's knowledge via a proxy way. We argue that these proxy-based methods overlook the knowledge loss of the teacher, which may cause the student to encounter capacity bottlenecks. In this paper, we alleviate the capacity gap problem from a new perspective with the purpose of averting knowledge loss. Instead of sacrificing part of the teacher's knowledge, we propose to build a more powerful student via adversarial collaborative learning. To this end, we further propose an Adversarial Collaborative Knowledge Distillation (ACKD) method that effectively improves the performance of knowledge distillation. Specifically, we construct the student model with multiple auxiliary learners. Meanwhile, we devise an adversarial collaborative module (ACM) that introduces attention mechanism and adversarial learning to enhance the capacity of the student. Extensive experiments on four classification tasks show the superiority of the proposed ACKD.