DANCE: Resource-Efficient Neural Architecture Search with Data-Aware and Continuous Adaptation

TL;DR

DANCE introduces a continuous, adaptable neural architecture search method that reduces costs and improves robustness across diverse deployment scenarios by learning distributions over architectural components.

Contribution

It presents a novel continuous evolution framework for NAS, enabling smooth adaptation and efficient sampling within a unified architecture space.

Findings

Outperforms state-of-the-art NAS in accuracy.

Reduces search costs significantly.

Maintains robust performance across hardware constraints.

Abstract

Neural Architecture Search (NAS) has emerged as a powerful approach for automating neural network design. However, existing NAS methods face critical limitations in real-world deployments: architectures lack adaptability across scenarios, each deployment context requires costly separate searches, and performance consistency across diverse platforms remains challenging. We propose DANCE (Dynamic Architectures with Neural Continuous Evolution), which reformulates architecture search as a continuous evolution problem through learning distributions over architectural components. DANCE introduces three key innovations: a continuous architecture distribution enabling smooth adaptation, a unified architecture space with learned selection gates for efficient sampling, and a multi-stage training strategy for effective deployment optimization. Extensive experiments across five datasets demonstrate DANCE's effectiveness. Our method consistently outperforms state-of-the-art NAS approaches in terms of accuracy while significantly reducing search costs. Under varying computational constraints, DANCE maintains robust performance while smoothly adapting architectures to different hardware requirements. The code and appendix can be found at https://github.com/Applied-Machine-Learning-Lab/DANCE.