FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining

TL;DR

This paper introduces NARS, a method that jointly searches for neural network architectures and their training recipes, significantly improving efficiency and performance of compact models like FBNetV3 across multiple tasks.

Contribution

The paper proposes a novel joint architecture-recipe search method with predictor pretraining, enabling fast, resource-efficient generation of high-performing neural networks.

Findings

FBNetV3 outperforms manually-designed models on ImageNet with fewer FLOPs.

NARS achieves high prediction accuracy and sample efficiency through predictor pretraining.

FBNetV3 improves object detection performance with fewer parameters and FLOPs.

Abstract

Neural Architecture Search (NAS) yields state-of-the-art neural networks that outperform their best manually-designed counterparts. However, previous NAS methods search for architectures under one set of training hyper-parameters (i.e., a training recipe), overlooking superior architecture-recipe combinations. To address this, we present Neural Architecture-Recipe Search (NARS) to search both (a) architectures and (b) their corresponding training recipes, simultaneously. NARS utilizes an accuracy predictor that scores architecture and training recipes jointly, guiding both sample selection and ranking. Furthermore, to compensate for the enlarged search space, we leverage "free" architecture statistics (e.g., FLOP count) to pretrain the predictor, significantly improving its sample efficiency and prediction reliability. After training the predictor via constrained iterative optimization, we run fast evolutionary searches in just CPU minutes to generate architecture-recipe pairs for a variety of resource constraints, called FBNetV3. FBNetV3 makes up a family of state-of-the-art compact neural networks that outperform both automatically and manually-designed competitors. For example, FBNetV3 matches both EfficientNet and ResNeSt accuracy on ImageNet with up to 2.0x and 7.1x fewer FLOPs, respectively. Furthermore, FBNetV3 yields significant performance gains for downstream object detection tasks, improving mAP despite 18% fewer FLOPs and 34% fewer parameters than EfficientNet-based equivalents.