Relational Knowledge Distillation

TL;DR

Relational Knowledge Distillation (RKD) transfers the mutual relations between data examples from teacher to student models, leading to improved performance and state-of-the-art results in metric learning tasks.

Contribution

The paper introduces RKD, a novel knowledge distillation method focusing on relational information, with new distance-wise and angle-wise loss functions.

Findings

RKD improves student model performance significantly.

In metric learning, students outperform teachers using RKD.

Achieves state-of-the-art results on benchmark datasets.

Abstract

Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.