Inductive Transfer for Neural Architecture Optimization

TL;DR

This paper introduces transfer learning techniques to neural architecture search, significantly reducing computational costs by leveraging previous knowledge for architecture selection and early stopping across multiple image classification datasets.

Contribution

The paper presents two novel transfer learning methods for neural architecture search: one for architecture selection and another for learning curve extrapolation, improving efficiency without accuracy loss.

Findings

Achieved faster neural architecture search on five image benchmarks.

Reduced computational cost by transferring knowledge across datasets.

Maintained competitive accuracy while accelerating the search process.

Abstract

The recent advent of automated neural network architecture search led to several methods that outperform state-of-the-art human-designed architectures. However, these approaches are computationally expensive, in extreme cases consuming GPU years. We propose two novel methods which aim to expedite this optimization problem by transferring knowledge acquired from previous tasks to new ones. First, we propose a novel neural architecture selection method which employs this knowledge to identify strong and weak characteristics of neural architectures across datasets. Thus, these characteristics do not need to be rediscovered in every search, a strong weakness of current state-of-the-art searches. Second, we propose a method for learning curve extrapolation to determine if a training process can be terminated early. In contrast to existing work, we propose to learn from learning curves of architectures trained on other datasets to improve the prediction accuracy for novel datasets. On five different image classification benchmarks, we empirically demonstrate that both of our orthogonal contributions independently lead to an acceleration, without any significant loss in accuracy.