Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search

TL;DR

Arch-Graph introduces a novel acyclic architecture relation predictor for transfer learning in neural architecture search, effectively modeling task correlations and ranking candidate models with high efficiency and transferability.

Contribution

It formulates NAS as an acyclic architecture relation graph prediction problem, incorporating task embeddings and a Maximal Weighted Acyclic Subgraph approach for improved transferability.

Findings

Outperforms existing NAS methods on TransNAS-Bench-101.

Achieves top 0.16 ext{ and }0.29 ext{ extbackslash extbackslash} architectures on average.

Requires only 50 models for effective search.

Abstract

Neural Architecture Search (NAS) aims to find efficient models for multiple tasks. Beyond seeking solutions for a single task, there are surging interests in transferring network design knowledge across multiple tasks. In this line of research, effectively modeling task correlations is vital yet highly neglected. Therefore, we propose \textbf{Arch-Graph}, a transferable NAS method that predicts task-specific optimal architectures with respect to given task embeddings. It leverages correlations across multiple tasks by using their embeddings as a part of the predictor's input for fast adaptation. We also formulate NAS as an architecture relation graph prediction problem, with the relational graph constructed by treating candidate architectures as nodes and their pairwise relations as edges. To enforce some basic properties such as acyclicity in the relational graph, we add additional constraints to the optimization process, converting NAS into the problem of finding a Maximal Weighted Acyclic Subgraph (MWAS). Our algorithm then strives to eliminate cycles and only establish edges in the graph if the rank results can be trusted. Through MWAS, Arch-Graph can effectively rank candidate models for each task with only a small budget to finetune the predictor. With extensive experiments on TransNAS-Bench-101, we show Arch-Graph's transferability and high sample efficiency across numerous tasks, beating many NAS methods designed for both single-task and multi-task search. It is able to find top 0.16\% and 0.29\% architectures on average on two search spaces under the budget of only 50 models.