Path-Level Network Transformation for Efficient Architecture Search

TL;DR

This paper presents a novel path-level network transformation method that enhances neural architecture search by enabling topology modifications while reusing trained weights, leading to more efficient and effective model design.

Contribution

The authors introduce a path-level transformation operation and a bidirectional tree-structured reinforcement learning controller for flexible architecture search.

Findings

Achieved 97.70% accuracy on CIFAR-10 with 14.3M parameters.

Obtained 74.6% top-1 accuracy on ImageNet in mobile setting.

Demonstrated improved parameter efficiency and transferability.

Abstract

We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that can only perform layer-level architecture modifications, such as adding (pruning) filters or inserting (removing) a layer, which fails to change the topology of connection paths. Our proposed path-level transformation operations enable the meta-controller to modify the path topology of the given network while keeping the merits of reusing weights, and thus allow efficiently designing effective structures with complex path topologies like Inception models. We further propose a bidirectional tree-structured reinforcement learning meta-controller to explore a simple yet highly expressive tree-structured architecture space that can be viewed as a generalization of multi-branch architectures. We experimented on the image classification datasets with limited computational resources (about 200 GPU-hours), where we observed improved parameter efficiency and better test results (97.70% test accuracy on CIFAR-10 with 14.3M parameters and 74.6% top-1 accuracy on ImageNet in the mobile setting), demonstrating the effectiveness and transferability of our designed architectures.