Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS

TL;DR

This paper introduces BONAS, a hybrid neural architecture search method combining sample-based and weight-sharing techniques to improve efficiency and reliability in designing neural networks.

Contribution

It proposes a Bayesian optimization framework with a Graph Convolutional Network predictor that accelerates sample-based NAS while maintaining its reliability.

Findings

Significantly accelerates sample-based NAS

Maintains higher reliability than traditional one-shot NAS

Outperforms competing algorithms in experiments

Abstract

Neural Architecture Search (NAS) has shown great potentials in finding better neural network designs. Sample-based NAS is the most reliable approach which aims at exploring the search space and evaluating the most promising architectures. However, it is computationally very costly. As a remedy, the one-shot approach has emerged as a popular technique for accelerating NAS using weight-sharing. However, due to the weight-sharing of vastly different networks, the one-shot approach is less reliable than the sample-based approach. In this work, we propose BONAS (Bayesian Optimized Neural Architecture Search), a sample-based NAS framework which is accelerated using weight-sharing to evaluate multiple related architectures simultaneously. Specifically, we apply Graph Convolutional Network predictor as a surrogate model for Bayesian Optimization to select multiple related candidate models in each iteration. We then apply weight-sharing to train multiple candidate models simultaneously. This approach not only accelerates the traditional sample-based approach significantly, but also keeps its reliability. This is because weight-sharing among related architectures are more reliable than those in the one-shot approach. Extensive experiments are conducted to verify the effectiveness of our method over many competing algorithms.