A Survey on Multi-Objective Neural Architecture Search

TL;DR

This survey reviews multi-objective neural architecture search (MONAS), highlighting recent advances, taxonomy, objectives, and future directions, emphasizing the importance of balancing accuracy with computational and resource constraints.

Contribution

It provides a comprehensive taxonomy, corrects previous categorizations, and introduces new objectives for MONAS, along with analysis of their stochastic properties.

Findings

Most objectives require stochastic optimization approaches

New objectives have been proposed for better trade-offs

Analysis shows importance of stochastic vs deterministic properties

Abstract

Recently, the expert-crafted neural architectures is increasing overtaken by the utilization of neural architecture search (NAS) and automatic generation (and tuning) of network structures which has a close relation to the Hyperparameter Optimization and Auto Machine Learning (AutoML). After the earlier NAS attempts to optimize only the prediction accuracy, Multi-Objective Neural architecture Search (MONAS) has been attracting attentions which considers more goals such as computational complexity, power consumption, and size of the network for optimization, reaching a trade-off between the accuracy and other features like the computational cost. In this paper, we present an overview of principal and state-of-the-art works in the field of MONAS. Starting from a well-categorized taxonomy and formulation for the NAS, we address and correct some miscategorizations in previous surveys of the NAS field. We also provide a list of all known objectives used and add a number of new ones and elaborate their specifications. We have provides analyses about the most important objectives and shown that the stochastic properties of some the them should be differed from deterministic ones in the multi-objective optimization procedure of NAS. We finalize this paper with a number of future directions and topics in the field of MONAS.