On Supernet Transfer Learning for Effective Task Adaptation

TL;DR

This paper introduces a supernet transfer learning method that efficiently adapts both weights and architectures for new tasks, outperforming traditional NAS and transfer learning in speed and model quality.

Contribution

The authors propose a novel supernet transfer learning approach that jointly fine-tunes weights and architectures, enabling faster and more effective task adaptation.

Findings

Speeds up model discovery by 3 to 5 times on average.

Finds better models than NAS from scratch.

Increases robustness and positive transfer across diverse datasets.

Abstract

Neural Architecture Search (NAS) methods have been shown to outperform hand-designed models and help to democratize AI. However, NAS methods often start from scratch with each new task, making them computationally expensive and limiting their applicability. Transfer learning is a practical alternative with the rise of ever-larger pretrained models. However, it is also bound to the architecture of the pretrained model, which inhibits proper adaptation of the architecture to different tasks, leading to suboptimal (and excessively large) models. We address both challenges at once by introducing a novel and practical method to \textit{transfer supernets}, which parameterize both weight and architecture priors, and efficiently finetune both to new tasks. This enables supernet transfer learning as a replacement for traditional transfer learning that also finetunes model architectures to new tasks. Through extensive experiments across multiple image classification tasks, we demonstrate that supernet transfer learning does not only drastically speed up the discovery of optimal models (3 to 5 times faster on average), but will also find better models than running NAS from scratch. The added model flexibility also increases the robustness of transfer learning, yielding positive transfer to even very different target datasets, especially with multi-dataset pretraining.