MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks

TL;DR

MorphNet is an efficient method for automatically designing neural network structures by iteratively shrinking and expanding networks, optimizing for resource constraints and performance across various datasets.

Contribution

It introduces a scalable, resource-aware structure learning approach that outperforms previous methods in designing efficient neural networks.

Findings

Achieves higher performance under resource constraints.

Discovers novel network structures for different datasets.

Scales effectively to large networks.

Abstract

We present MorphNet, an approach to automate the design of neural network structures. MorphNet iteratively shrinks and expands a network, shrinking via a resource-weighted sparsifying regularizer on activations and expanding via a uniform multiplicative factor on all layers. In contrast to previous approaches, our method is scalable to large networks, adaptable to specific resource constraints (e.g. the number of floating-point operations per inference), and capable of increasing the network's performance. When applied to standard network architectures on a wide variety of datasets, our approach discovers novel structures in each domain, obtaining higher performance while respecting the resource constraint.