Neural Inheritance Relation Guided One-Shot Layer Assignment Search

TL;DR

This paper introduces a novel neural inheritance relation in layer assignment, enabling an efficient one-shot search method that significantly improves neural architecture search performance across multiple datasets.

Contribution

It uncovers a neural inheritance relation in layer assignment and proposes an inheritance-guided one-shot search approach that reduces search space and enhances efficiency.

Findings

The inheritance relation holds across different network depths.

The proposed method achieves superior results compared to handcrafted architectures.

Experiments on CIFAR-100, Tiny-ImageNet, and ImageNet validate effectiveness.

Abstract

Layer assignment is seldom picked out as an independent research topic in neural architecture search. In this paper, for the first time, we systematically investigate the impact of different layer assignments to the network performance by building an architecture dataset of layer assignment on CIFAR-100. Through analyzing this dataset, we discover a neural inheritance relation among the networks with different layer assignments, that is, the optimal layer assignments for deeper networks always inherit from those for shallow networks. Inspired by this neural inheritance relation, we propose an efficient one-shot layer assignment search approach via inherited sampling. Specifically, the optimal layer assignment searched in the shallow network can be provided as a strong sampling priori to train and search the deeper ones in supernet, which extremely reduces the network search space. Comprehensive experiments carried out on CIFAR-100 illustrate the efficiency of our proposed method. Our search results are strongly consistent with the optimal ones directly selected from the architecture dataset. To further confirm the generalization of our proposed method, we also conduct experiments on Tiny-ImageNet and ImageNet. Our searched results are remarkably superior to the handcrafted ones under the unchanged computational budgets. The neural inheritance relation discovered in this paper can provide insights to the universal neural architecture search.