Robust Student Network Learning

TL;DR

This paper proposes a teacher-student learning framework that enhances the robustness of lightweight student neural networks against perturbations while maintaining high accuracy, addressing a gap in existing methods focused mainly on accuracy and compression.

Contribution

It introduces a novel approach that improves student network robustness by optimizing prediction confidence and gradient-based objectives during training.

Findings

Student networks become more robust to perturbations.

The method achieves a good balance of accuracy and compactness.

Experimental results validate the effectiveness of the approach.

Abstract

Deep neural networks bring in impressive accuracy in various applications, but the success often relies on the heavy network architecture. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable student network with significantly fewer parameters can achieve a considerable accuracy which is comparable to that of teacher network. However, beyond accuracy, robustness of the learned student network against perturbation is also essential for practical uses. Existing teacher-student learning frameworks mainly focus on accuracy and compression ratios, but ignore the robustness. In this paper, we make the student network produce more confident predictions with the help of the teacher network, and analyze the lower bound of the perturbation that will destroy the confidence of the student network. Two important objectives regarding prediction scores and gradients of examples are developed to maximize this lower bound, so as to enhance the robustness of the student network without sacrificing the performance. Experiments on benchmark datasets demonstrate the efficiency of the proposed approach to learn robust student networks which have satisfying accuracy and compact sizes.