Warm-starting DARTS using meta-learning

TL;DR

This paper introduces a meta-learning approach to warm-start DARTS for neural architecture search, enabling quick adaptation to new tasks with reduced search costs by leveraging task similarity and learned transfer architectures.

Contribution

It proposes a novel meta-learning framework for warm-starting DARTS, incorporating task similarity and meta-transfer architectures to improve efficiency and performance across multiple tasks.

Findings

Achieves 60% reduction in search costs.

Finds competitive architectures on new tasks.

Utilizes task similarity for better transfer performance.

Abstract

Neural architecture search (NAS) has shown great promise in the field of automated machine learning (AutoML). NAS has outperformed hand-designed networks and made a significant step forward in the field of automating the design of deep neural networks, thus further reducing the need for human expertise. However, most research is done targeting a single specific task, leaving research of NAS methods over multiple tasks mostly overlooked. Generally, there exist two popular ways to find an architecture for some novel task. Either searching from scratch, which is ineffective by design, or transferring discovered architectures from other tasks, which provides no performance guarantees and is probably not optimal. In this work, we present a meta-learning framework to warm-start Differentiable architecture search (DARTS). DARTS is a NAS method that can be initialized with a transferred architecture and is able to quickly adapt to new tasks. A task similarity measure is used to determine which transfer architecture is selected, as transfer architectures found on similar tasks will likely perform better. Additionally, we employ a simple meta-transfer architecture that was learned over multiple tasks. Experiments show that warm-started DARTS is able to find competitive performing architectures while reducing searching costs on average by 60%.