Pareto-aware Neural Architecture Generation for Diverse Computational Budgets

TL;DR

This paper introduces PNAG, a Pareto-aware neural architecture generator that efficiently produces optimal architectures for various computational budgets, reducing search costs and improving results across multiple hardware platforms.

Contribution

We propose a single trained generator that produces Pareto optimal architectures for any budget, enabling efficient multi-budget neural architecture search.

Findings

Outperforms existing methods on mobile, CPU, and GPU platforms.

Reduces search cost by jointly learning multiple Pareto optimal architectures.

Achieves better architecture quality compared to independent search methods.

Abstract

Designing feasible and effective architectures under diverse computational budgets, incurred by different applications/devices, is essential for deploying deep models in real-world applications. To achieve this goal, existing methods often perform an independent architecture search process for each target budget, which is very inefficient yet unnecessary. More critically, these independent search processes cannot share their learned knowledge (i.e., the distribution of good architectures) with each other and thus often result in limited search results. To address these issues, we propose a Pareto-aware Neural Architecture Generator (PNAG) which only needs to be trained once and dynamically produces the Pareto optimal architecture for any given budget via inference. To train our PNAG, we learn the whole Pareto frontier by jointly finding multiple Pareto optimal architectures under diverse budgets. Such a joint search algorithm not only greatly reduces the overall search cost but also improves the search results. Extensive experiments on three hardware platforms (i.e., mobile device, CPU, and GPU) show the superiority of our method over existing methods.