The devil is in discretization discrepancy. Robustifying Differentiable NAS with Single-Stage Searching Protocol

TL;DR

This paper addresses the discretization error in differentiable neural architecture search (NAS) by proposing a robust single-stage searching protocol that improves performance and efficiency, reducing the impact of regularization issues and enhancing search stability.

Contribution

The authors introduce a novel single-stage NAS protocol that does not rely on decoding continuous architectures, improving robustness and efficiency over existing differentiable NAS methods.

Findings

Achieves 75.3% accuracy on Cityscapes validation in the search stage.

Outperforms other DNAS methods by 1.1% on a non-dense search space.

Training completes in only 5.5 GPU days due to weight reuse.

Abstract

Neural Architecture Search (NAS) has been widely adopted to design neural networks for various computer vision tasks. One of its most promising subdomains is differentiable NAS (DNAS), where the optimal architecture is found in a differentiable manner. However, gradient-based methods suffer from the discretization error, which can severely damage the process of obtaining the final architecture. In our work, we first study the risk of discretization error and show how it affects an unregularized supernet. Then, we present that penalizing high entropy, a common technique of architecture regularization, can hinder the supernet's performance. Therefore, to robustify the DNAS framework, we introduce a novel single-stage searching protocol, which is not reliant on decoding a continuous architecture. Our results demonstrate that this approach outperforms other DNAS methods by achieving 75.3% in the searching stage on the Cityscapes validation dataset and attains performance 1.1% higher than the optimal network of DCNAS on the non-dense search space comprising short connections. The entire training process takes only 5.5 GPU days due to the weight reuse, and yields a computationally efficient architecture. Additionally, we propose a new dataset split procedure, which substantially improves results and prevents architecture degeneration in DARTS.