PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillation

TL;DR

PocketNet introduces a highly lightweight face recognition network optimized through neural architecture search and multi-step knowledge distillation, achieving state-of-the-art accuracy on multiple benchmarks with significantly fewer parameters.

Contribution

The paper proposes a novel lightweight face recognition architecture developed via neural architecture search and a multi-step knowledge distillation training paradigm, improving efficiency and accuracy.

Findings

PocketNet outperforms existing compact models on nine benchmarks.

The smallest model, PocketNetS-128, achieves competitive results with only 0.92M parameters.

Multi-step knowledge distillation enhances training effectiveness.

Abstract

Deep neural networks have rapidly become the mainstream method for face recognition (FR). However, this limits the deployment of such models that contain an extremely large number of parameters to embedded and low-end devices. In this work, we present an extremely lightweight and accurate FR solution, namely PocketNet. We utilize neural architecture search to develop a new family of lightweight face-specific architectures. We additionally propose a novel training paradigm based on knowledge distillation (KD), the multi-step KD, where the knowledge is distilled from the teacher model to the student model at different stages of the training maturity. We conduct a detailed ablation study proving both, the sanity of using NAS for the specific task of FR rather than general object classification, and the benefits of our proposed multi-step KD. We present an extensive experimental evaluation and comparisons with the state-of-the-art (SOTA) compact FR models on nine different benchmarks including large-scale evaluation benchmarks such as IJB-B, IJB-C, and MegaFace. PocketNets have consistently advanced the SOTA FR performance on nine mainstream benchmarks when considering the same level of model compactness. With 0.92M parameters, our smallest network PocketNetS-128 achieved very competitive results to recent SOTA compacted models that contain up to 4M parameters.