Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variation

TL;DR

This paper addresses the variability in zero-shot NAS ranking functions by modeling their outputs as random variables and using stochastic ordering to improve architecture selection, leading to better search performance.

Contribution

It introduces a novel approach that accounts for ranking function variation through stochastic ordering, enhancing zero-shot NAS effectiveness.

Findings

Stochastic ordering improves NAS search outcomes.

Modeling ranking outputs as random variables is beneficial.

Proposed method outperforms traditional averaging approaches.

Abstract

Neural Architecture Search (NAS) is a powerful automatic alternative to manual design of a neural network. In the zero-shot version, a fast ranking function is used to compare architectures without training them. The outputs of the ranking functions often vary significantly due to different sources of randomness, including the evaluated architecture's weights' initialization or the batch of data used for calculations. A common approach to addressing the variation is to average a ranking function output over several evaluations. We propose taking into account the variation in a different manner, by viewing the ranking function output as a random variable representing a proxy performance metric. During the search process, we strive to construct a stochastic ordering of the performance metrics to determine the best architecture. Our experiments show that the proposed stochastic ordering can effectively boost performance of a search on standard benchmark search spaces.