Improving Differentiable Architecture Search via Self-Distillation

TL;DR

This paper introduces SD-DARTS, a novel NAS method that employs self-distillation and voting teachers to reduce sharp minima convergence, thereby improving the performance of the derived architectures.

Contribution

The paper proposes a self-distillation approach with voting teachers to enhance DARTS, addressing the sharp minima issue and improving NAS results.

Findings

SD-DARTS reduces the sharpness of the loss landscape.

SD-DARTS outperforms state-of-the-art NAS methods.

Voting teachers improve the stability and accuracy of architecture search.

Abstract

Differentiable Architecture Search (DARTS) is a simple yet efficient Neural Architecture Search (NAS) method. During the search stage, DARTS trains a supernet by jointly optimizing architecture parameters and network parameters. During the evaluation stage, DARTS discretizes the supernet to derive the optimal architecture based on architecture parameters. However, recent research has shown that during the training process, the supernet tends to converge towards sharp minima rather than flat minima. This is evidenced by the higher sharpness of the loss landscape of the supernet, which ultimately leads to a performance gap between the supernet and the optimal architecture. In this paper, we propose Self-Distillation Differentiable Neural Architecture Search (SD-DARTS) to alleviate the discretization gap. We utilize self-distillation to distill knowledge from previous steps of the supernet to guide its training in the current step, effectively reducing the sharpness of the supernet's loss and bridging the performance gap between the supernet and the optimal architecture. Furthermore, we introduce the concept of voting teachers, where multiple previous supernets are selected as teachers, and their output probabilities are aggregated through voting to obtain the final teacher prediction. Experimental results on real datasets demonstrate the advantages of our novel self-distillation-based NAS method compared to state-of-the-art alternatives.