ICDARTS: Improving the Stability and Performance of Cyclic DARTS

TL;DR

This paper proposes ICDARTS, an improved neural architecture search method based on Cyclic DARTS, enhancing stability, generalizability, and flexibility by removing dependency issues and expanding search space capabilities.

Contribution

ICDARTS introduces a new approach that eliminates dependency of evaluation network weights on the search network and allows zero operations to be retained, improving NAS stability and generalizability.

Findings

ICDARTS achieves more stable search processes.

Networks found by ICDARTS generalize better across tasks.

Expanded search space improves architecture diversity.

Abstract

This work introduces improvements to the stability and generalizability of Cyclic DARTS (CDARTS). CDARTS is a Differentiable Architecture Search (DARTS)-based approach to neural architecture search (NAS) that uses a cyclic feedback mechanism to train search and evaluation networks concurrently. This training protocol aims to optimize the search process by enforcing that the search and evaluation networks produce similar outputs. However, CDARTS introduces a loss function for the evaluation network that is dependent on the search network. The dissimilarity between the loss functions used by the evaluation networks during the search and retraining phases results in a search-phase evaluation network that is a sub-optimal proxy for the final evaluation network that is utilized during retraining. We present ICDARTS, a revised approach that eliminates the dependency of the evaluation network weights upon those of the search network, along with a modified process for discretizing the search network's \textit{zero} operations that allows these operations to be retained in the final evaluation networks. We pair the results of these changes with ablation studies on ICDARTS' algorithm and network template. Finally, we explore methods for expanding the search space of ICDARTS by expanding its operation set and exploring alternate methods for discretizing its continuous search cells. These experiments resulted in networks with improved generalizability and the implementation of a novel method for incorporating a dynamic search space into ICDARTS.