Search-time Efficient Device Constraints-Aware Neural Architecture Search

TL;DR

This paper introduces DCA-NAS, a fast neural architecture search method that efficiently generates device-constraint-aware models suitable for edge computing, outperforming manual designs and matching mobile architectures.

Contribution

The paper presents DCA-NAS, a novel NAS approach that incorporates device constraints and uses weight sharing and channel bottlenecks for rapid search.

Findings

DCA-NAS outperforms manual architectures on image classification tasks.

It achieves comparable performance to popular mobile architectures.

It generalizes well across different search spaces and hardware benchmarks.

Abstract

Edge computing aims to enable edge devices, such as IoT devices, to process data locally instead of relying on the cloud. However, deep learning techniques like computer vision and natural language processing can be computationally expensive and memory-intensive. Creating manual architectures specialized for each device is infeasible due to their varying memory and computational constraints. To address these concerns, we automate the construction of task-specific deep learning architectures optimized for device constraints through Neural Architecture Search (NAS). We present DCA-NAS, a principled method of fast neural network architecture search that incorporates edge-device constraints such as model size and floating-point operations. It incorporates weight sharing and channel bottleneck techniques to speed up the search time. Based on our experiments, we see that DCA-NAS outperforms manual architectures for similar sized models and is comparable to popular mobile architectures on various image classification datasets like CIFAR-10, CIFAR-100, and Imagenet-1k. Experiments with search spaces -- DARTS and NAS-Bench-201 show the generalization capabilities of DCA-NAS. On further evaluating our approach on Hardware-NAS-Bench, device-specific architectures with low inference latency and state-of-the-art performance were discovered.