Combining Neural Architecture Search and Automatic Code Optimization: A Survey

TL;DR

This survey reviews the integration of Hardware-aware Neural Architecture Search and Automatic Code Optimization, highlighting their combined potential to improve deep learning model efficiency on resource-limited devices.

Contribution

It introduces the NACOS framework that jointly optimizes neural architectures and compiler settings, addressing limitations of independent approaches.

Findings

Joint optimization outperforms separate methods in efficiency.

NACOS framework demonstrates improved hardware compatibility.

Combining techniques reduces resource demands of deep models.

Abstract

Deep Learning models have experienced exponential growth in complexity and resource demands in recent years. Accelerating these models for efficient execution on resource-constrained devices has become more crucial than ever. Two notable techniques employed to achieve this goal are Hardware-aware Neural Architecture Search (HW-NAS) and Automatic Code Optimization (ACO). HW-NAS automatically designs accurate yet hardware-friendly neural networks, while ACO involves searching for the best compiler optimizations to apply on neural networks for efficient mapping and inference on the target hardware. This survey explores recent works that combine these two techniques within a single framework. We present the fundamental principles of both domains and demonstrate their sub-optimality when performed independently. We then investigate their integration into a joint optimization process that we call Hardware Aware-Neural Architecture and Compiler Optimizations co-Search (NACOS).