CoupleFace: Relation Matters for Face Recognition Distillation

TL;DR

This paper introduces CoupleFace, a face recognition distillation method that incorporates mutual relation knowledge transfer to enhance student model performance, outperforming existing methods and winning a challenge.

Contribution

It proposes Mutual Relation Distillation (MRD) and Relation-Aware Distillation (RAD) to transfer relation knowledge, improving face recognition distillation beyond feature consistency methods.

Findings

Outperforms existing face recognition distillation methods.

Achieved first place in ICCV21 Masked Face Recognition Challenge.

Demonstrates significant improvements on multiple benchmark datasets.

Abstract

Knowledge distillation is an effective method to improve the performance of a lightweight neural network (i.e., student model) by transferring the knowledge of a well-performed neural network (i.e., teacher model), which has been widely applied in many computer vision tasks, including face recognition. Nevertheless, the current face recognition distillation methods usually utilize the Feature Consistency Distillation (FCD) (e.g., L2 distance) on the learned embeddings extracted by the teacher and student models for each sample, which is not able to fully transfer the knowledge from the teacher to the student for face recognition. In this work, we observe that mutual relation knowledge between samples is also important to improve the discriminative ability of the learned representation of the student model, and propose an effective face recognition distillation method called CoupleFace by additionally introducing the Mutual Relation Distillation (MRD) into existing distillation framework. Specifically, in MRD, we first propose to mine the informative mutual relations, and then introduce the Relation-Aware Distillation (RAD) loss to transfer the mutual relation knowledge of the teacher model to the student model. Extensive experimental results on multiple benchmark datasets demonstrate the effectiveness of our proposed CoupleFace for face recognition. Moreover, based on our proposed CoupleFace, we have won the first place in the ICCV21 Masked Face Recognition Challenge (MS1M track).