DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS

TL;DR

DARTS-PRIME enhances differentiable neural architecture search by introducing dynamic scheduling and regularization techniques, leading to improved performance and reliability across multiple domains.

Contribution

It presents DARTS-PRIME, a novel variant of DARTS that incorporates informed update scheduling and proximity regularization for better discretization and results.

Findings

Improves NAS performance and reliability

Achieves results comparable to state-of-the-art methods

Effective across multiple domains

Abstract

Differentiable Architecture Search (DARTS) is a recent neural architecture search (NAS) method based on a differentiable relaxation. Due to its success, numerous variants analyzing and improving parts of the DARTS framework have recently been proposed. By considering the problem as a constrained bilevel optimization, we present and analyze DARTS-PRIME, a variant including improvements to architectural weight update scheduling and regularization towards discretization. We propose a dynamic schedule based on per-minibatch network information to make architecture updates more informed, as well as proximity regularization to promote well-separated discretization. Our results in multiple domains show that DARTS-PRIME improves both performance and reliability, comparable to state-of-the-art in differentiable NAS.