Differentiable Network Adaption with Elastic Search Space

TL;DR

This paper introduces Differentiable Network Adaption (DNA), a gradient-based method for automatically adjusting network width and depth to meet specific computational budgets, outperforming heuristic approaches.

Contribution

The paper presents a novel differentiable approach with an elastic search space for bi-directional network architecture adaptation, improving efficiency and performance.

Findings

DNA outperforms previous methods on ImageNet.

DNA enhances high-accuracy networks like EfficientNet and MobileNet-v3.

The elastic search space enables flexible network optimization.

Abstract

In this paper we propose a novel network adaption method called Differentiable Network Adaption (DNA), which can adapt an existing network to a specific computation budget by adjusting the width and depth in a differentiable manner. The gradient-based optimization allows DNA to achieve an automatic optimization of width and depth rather than previous heuristic methods that heavily rely on human priors. Moreover, we propose a new elastic search space that can flexibly condense or expand during the optimization process, allowing the network optimization of width and depth in a bi-direction manner. By DNA, we successfully achieve network architecture optimization by condensing and expanding in both width and depth dimensions. Extensive experiments on ImageNet demonstrate that DNA can adapt the existing network to meet different targeted computation requirements with better performance than previous methods. What's more, DNA can further improve the performance of high-accuracy networks obtained by state-of-the-art neural architecture search methods such as EfficientNet and MobileNet-v3.