Exploring Complicated Search Spaces with Interleaving-Free Sampling

TL;DR

This paper introduces IF-NAS, a novel neural architecture search algorithm designed for complex search spaces with long-distance connections, effectively avoiding interleaved connections that hinder existing methods.

Contribution

The paper proposes IF-NAS, a simple periodic sampling strategy that improves search in complex spaces with long-distance connections by avoiding interleaved connections.

Findings

IF-NAS outperforms random sampling and previous weight-sharing algorithms

The method generalizes well to micro cell-based spaces

Emphasizes the importance of macro structure in neural architecture search

Abstract

The existing neural architecture search algorithms are mostly working on search spaces with short-distance connections. We argue that such designs, though safe and stable, obstacles the search algorithms from exploring more complicated scenarios. In this paper, we build the search algorithm upon a complicated search space with long-distance connections, and show that existing weight-sharing search algorithms mostly fail due to the existence of \textbf{interleaved connections}. Based on the observation, we present a simple yet effective algorithm named \textbf{IF-NAS}, where we perform a periodic sampling strategy to construct different sub-networks during the search procedure, avoiding the interleaved connections to emerge in any of them. In the proposed search space, IF-NAS outperform both random sampling and previous weight-sharing search algorithms by a significant margin. IF-NAS also generalizes to the micro cell-based spaces which are much easier. Our research emphasizes the importance of macro structure and we look forward to further efforts along this direction.