AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture

TL;DR

AutoShrink introduces a topology-aware neural architecture search method that efficiently discovers flexible, resource-efficient neural network structures by progressively shrinking edges in DAGs, significantly reducing search time and model size.

Contribution

The paper presents a novel node-based NAS approach that uses edge shrinking in DAGs to explore flexible network topologies with less search time and better resource efficiency.

Findings

Achieves up to 48% parameter reduction on ImageNet-1K.

Reduces search time by over 6 times compared to SOTA methods.

Maintains comparable accuracy with state-of-the-art models.

Abstract

Resource is an important constraint when deploying Deep Neural Networks (DNNs) on mobile and edge devices. Existing works commonly adopt the cell-based search approach, which limits the flexibility of network patterns in learned cell structures. Moreover, due to the topology-agnostic nature of existing works, including both cell-based and node-based approaches, the search process is time consuming and the performance of found architecture may be sub-optimal. To address these problems, we propose AutoShrink, a topology-aware Neural Architecture Search(NAS) for searching efficient building blocks of neural architectures. Our method is node-based and thus can learn flexible network patterns in cell structures within a topological search space. Directed Acyclic Graphs (DAGs) are used to abstract DNN architectures and progressively optimize the cell structure through edge shrinking. As the search space intrinsically reduces as the edges are progressively shrunk, AutoShrink explores more flexible search space with even less search time. We evaluate AutoShrink on image classification and language tasks by crafting ShrinkCNN and ShrinkRNN models. ShrinkCNN is able to achieve up to 48% parameter reduction and save 34% Multiply-Accumulates (MACs) on ImageNet-1K with comparable accuracy of state-of-the-art (SOTA) models. Specifically, both ShrinkCNN and ShrinkRNN are crafted within 1.5 GPU hours, which is 7.2x and 6.7x faster than the crafting time of SOTA CNN and RNN models, respectively.