Multi-objective Differentiable Neural Architecture Search

TL;DR

This paper introduces a novel neural architecture search method that efficiently profiles Pareto fronts across multiple devices and objectives using a hypernetwork, enabling zero-shot transfer and outperforming existing methods.

Contribution

The authors propose a hypernetwork-based NAS algorithm that encodes user preferences and hardware features, allowing single-run Pareto front profiling and zero-shot transfer to new devices.

Findings

Effective profiling of Pareto fronts across 19 devices.

Outperforms existing MOO NAS methods on various datasets.

Scalable to diverse search spaces and objectives.

Abstract

Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that require training a neural network. Typically, in MOO for neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics, yielding representative and diverse architectures across multiple devices in just a single search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments involving up to 19 hardware devices and 3 different objectives demonstrate the effectiveness and scalability of our method. Finally, we show that, without any additional costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only space for language modelling.