Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search

TL;DR

This paper introduces a landmark regularization method that improves neural architecture search by aligning shared-weight network rankings with standalone architecture performance, enhancing effectiveness across various algorithms and tasks.

Contribution

It proposes a novel regularization term that enhances ranking correlation in weight-sharing NAS, applicable across multiple algorithms and search spaces.

Findings

Consistently improves NAS performance across algorithms

Enhances ranking correlation between shared and standalone architectures

Applicable to various search spaces and tasks

Abstract

Weight sharing has become a de facto standard in neural architecture search because it enables the search to be done on commodity hardware. However, recent works have empirically shown a ranking disorder between the performance of stand-alone architectures and that of the corresponding shared-weight networks. This violates the main assumption of weight-sharing NAS algorithms, thus limiting their effectiveness. We tackle this issue by proposing a regularization term that aims to maximize the correlation between the performance rankings of the shared-weight network and that of the standalone architectures using a small set of landmark architectures. We incorporate our regularization term into three different NAS algorithms and show that it consistently improves performance across algorithms, search-spaces, and tasks.