Neural Ensemble Search via Bayesian Sampling

TL;DR

This paper introduces NESBS, a neural ensemble search algorithm using Bayesian sampling, which effectively finds diverse neural network ensembles that outperform single architectures in NAS, with comparable search costs.

Contribution

The paper proposes a novel ensemble-based NAS method, NESBS, that improves performance by selecting diverse neural network ensembles rather than single architectures.

Findings

NESBS achieves better performance than state-of-the-art NAS algorithms.

NESBS maintains comparable search costs to existing NAS methods.

Ensemble selection enhances neural network performance in NAS.

Abstract

Recently, neural architecture search (NAS) has been applied to automate the design of neural networks in real-world applications. A large number of algorithms have been developed to improve the search cost or the performance of the final selected architectures in NAS. Unfortunately, these NAS algorithms aim to select only one single well-performing architecture from their search spaces and thus have overlooked the capability of neural network ensemble (i.e., an ensemble of neural networks with diverse architectures) in achieving improved performance over a single final selected architecture. To this end, we introduce a novel neural ensemble search algorithm, called neural ensemble search via Bayesian sampling (NESBS), to effectively and efficiently select well-performing neural network ensembles from a NAS search space. In our extensive experiments, NESBS algorithm is shown to be able to achieve improved performance over state-of-the-art NAS algorithms while incurring a comparable search cost, thus indicating the superior performance of our NESBS algorithm over these NAS algorithms in practice.