SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Search

TL;DR

SONATA introduces a self-adaptive evolutionary framework for hardware-aware neural architecture search, leveraging learned importance of design parameters and surrogate models to improve search efficiency and optimize neural network performance on IoT devices.

Contribution

It proposes a novel self-adaptive evolutionary algorithm that uses surrogate models and reinforcement learning to guide neural architecture search for IoT hardware.

Findings

Achieved up to 0.25% accuracy improvement on ImageNet-1k.

Gained up to 2.42x reductions in latency and energy consumption.

Outperformed NSGA-II with 93.6% Pareto dominance.

Abstract

Recent advancements in Artificial Intelligence (AI), driven by Neural Networks (NN), demand innovative neural architecture designs, particularly within the constrained environments of Internet of Things (IoT) systems, to balance performance and efficiency. HW-aware Neural Architecture Search (HW-aware NAS) emerges as an attractive strategy to automate the design of NN using multi-objective optimization approaches, such as evolutionary algorithms. However, the intricate relationship between NN design parameters and HW-aware NAS optimization objectives remains an underexplored research area, overlooking opportunities to effectively leverage this knowledge to guide the search process accordingly. Furthermore, the large amount of evaluation data produced during the search holds untapped potential for refining the optimization strategy and improving the approximation of the Pareto front. Addressing these issues, we propose SONATA, a self-adaptive evolutionary algorithm for HW-aware NAS. Our method leverages adaptive evolutionary operators guided by the learned importance of NN design parameters. Specifically, through tree-based surrogate models and a Reinforcement Learning agent, we aspire to gather knowledge on 'How' and 'When' to evolve NN architectures. Comprehensive evaluations across various NAS search spaces and hardware devices on the ImageNet-1k dataset have shown the merit of SONATA with up to 0.25% improvement in accuracy and up to 2.42x gains in latency and energy. Our SONATA has seen up to sim$93.6% Pareto dominance over the native NSGA-II, further stipulating the importance of self-adaptive evolution operators in HW-aware NAS.