Cross Architecture Distillation for Face Recognition

TL;DR

This paper proposes a novel cross-architecture knowledge distillation method for face recognition, enabling effective transfer from Transformer teachers to CNN students by aligning spatial features and enhancing distillation-specific learning.

Contribution

It introduces URFM for spatial feature alignment and APT for distillation-specific knowledge management, addressing key challenges in cross-architecture face recognition distillation.

Findings

Outperforms existing methods on face benchmarks

Effective knowledge transfer from Transformers to CNNs

Improved face verification accuracy

Abstract

Transformers have emerged as the superior choice for face recognition tasks, but their insufficient platform acceleration hinders their application on mobile devices. In contrast, Convolutional Neural Networks (CNNs) capitalize on hardware-compatible acceleration libraries. Consequently, it has become indispensable to preserve the distillation efficacy when transferring knowledge from a Transformer-based teacher model to a CNN-based student model, known as Cross-Architecture Knowledge Distillation (CAKD). Despite its potential, the deployment of CAKD in face recognition encounters two challenges: 1) the teacher and student share disparate spatial information for each pixel, obstructing the alignment of feature space, and 2) the teacher network is not trained in the role of a teacher, lacking proficiency in handling distillation-specific knowledge. To surmount these two constraints, 1) we first introduce a Unified Receptive Fields Mapping module (URFM) that maps pixel features of the teacher and student into local features with unified receptive fields, thereby synchronizing the pixel-wise spatial information of teacher and student. Subsequently, 2) we develop an Adaptable Prompting Teacher network (APT) that integrates prompts into the teacher, enabling it to manage distillation-specific knowledge while preserving the model's discriminative capacity. Extensive experiments on popular face benchmarks and two large-scale verification sets demonstrate the superiority of our method.