RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation

TL;DR

This paper introduces RNAS-CL, a neural architecture search method that enhances robustness against adversarial attacks by utilizing cross-layer knowledge distillation from a robust teacher, leading to small, resilient neural networks.

Contribution

It proposes a novel NAS algorithm that automatically searches for optimal teacher-student layer supervision to improve robustness through cross-layer knowledge distillation.

Findings

RNAS-CL produces neural architectures that are both small and robust.

Experimental results demonstrate the effectiveness of RNAS-CL against adversarial attacks.

RNAS-CL outperforms baseline methods in robustness and efficiency.

Abstract

Deep Neural Networks are vulnerable to adversarial attacks. Neural Architecture Search (NAS), one of the driving tools of deep neural networks, demonstrates superior performance in prediction accuracy in various machine learning applications. However, it is unclear how it performs against adversarial attacks. Given the presence of a robust teacher, it would be interesting to investigate if NAS would produce robust neural architecture by inheriting robustness from the teacher. In this paper, we propose Robust Neural Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL), a novel NAS algorithm that improves the robustness of NAS by learning from a robust teacher through cross-layer knowledge distillation. Unlike previous knowledge distillation methods that encourage close student/teacher output only in the last layer, RNAS-CL automatically searches for the best teacher layer to supervise each student layer. Experimental result evidences the effectiveness of RNAS-CL and shows that RNAS-CL produces small and robust neural architecture.