Tackling Neural Architecture Search With Quality Diversity Optimization

TL;DR

This paper introduces a novel approach to neural architecture search by framing it as a quality diversity optimization problem, leading to more diverse and application-specific architectures than traditional multi-objective methods.

Contribution

The paper formulates multi-objective NAS as a quality diversity optimization problem and proposes three new QDO-based NAS optimizers that outperform existing multi-objective NAS methods.

Findings

QDO-based NAS optimizers find more diverse architectures.

QDO approaches outperform multi-objective NAS in quality and efficiency.

Application-specific niches are better addressed with QDO methods.

Abstract

Neural architecture search (NAS) has been studied extensively and has grown to become a research field with substantial impact. While classical single-objective NAS searches for the architecture with the best performance, multi-objective NAS considers multiple objectives that should be optimized simultaneously, e.g., minimizing resource usage along the validation error. Although considerable progress has been made in the field of multi-objective NAS, we argue that there is some discrepancy between the actual optimization problem of practical interest and the optimization problem that multi-objective NAS tries to solve. We resolve this discrepancy by formulating the multi-objective NAS problem as a quality diversity optimization (QDO) problem and introduce three quality diversity NAS optimizers (two of them belonging to the group of multifidelity optimizers), which search for high-performing yet diverse architectures that are optimal for application-specific niches, e.g., hardware constraints. By comparing these optimizers to their multi-objective counterparts, we demonstrate that quality diversity NAS in general outperforms multi-objective NAS with respect to quality of solutions and efficiency. We further show how applications and future NAS research can thrive on QDO.