VINNAS: Variational Inference-based Neural Network Architecture Search

TL;DR

VINNAS introduces a variational inference-based neural architecture search method that efficiently finds sparse, high-performing convolutional neural networks while avoiding mode collapse, leading to state-of-the-art accuracy with fewer parameters.

Contribution

The paper proposes a novel NAS approach using variational dropout with automatic relevance determination to prevent mode collapse and promote sparsity in neural architectures.

Findings

Achieves state-of-the-art accuracy on image classification tasks.

Finds highly sparse neural network architectures with fewer parameters.

Effectively avoids mode collapse in NAS process.

Abstract

In recent years, neural architecture search (NAS) has received intensive scientific and industrial interest due to its capability of finding a neural architecture with high accuracy for various artificial intelligence tasks such as image classification or object detection. In particular, gradient-based NAS approaches have become one of the more popular approaches thanks to their computational efficiency during the search. However, these methods often experience a mode collapse, where the quality of the found architectures is poor due to the algorithm resorting to choosing a single operation type for the entire network, or stagnating at a local minima for various datasets or search spaces. To address these defects, we present a differentiable variational inference-based NAS method for searching sparse convolutional neural networks. Our approach finds the optimal neural architecture by dropping out candidate operations in an over-parameterised supergraph using variational dropout with automatic relevance determination prior, which makes the algorithm gradually remove unnecessary operations and connections without risking mode collapse. The evaluation is conducted through searching two types of convolutional cells that shape the neural network for classifying different image datasets. Our method finds diverse network cells, while showing state-of-the-art accuracy with up to almost 2 times fewer non-zero parameters.