Compact Neural Networks via Stacking Designed Basic Units

TL;DR

This paper introduces TissueNet, a novel method for constructing compact neural networks by stacking designed basic units, achieving significant parameter and FLOPs reduction without complex pruning criteria.

Contribution

The paper proposes TissueNet, a new approach that builds compact networks through stacking units, eliminating the need for pruning criteria and demonstrating competitive performance.

Findings

Achieves up to 80% FLOPs reduction

Saves up to 89.7% parameters

Maintains comparable accuracy

Abstract

Unstructured pruning has the limitation of dealing with the sparse and irregular weights. By contrast, structured pruning can help eliminate this drawback but it requires complex criterion to determine which components to be pruned. To this end, this paper presents a new method termed TissueNet, which directly constructs compact neural networks with fewer weight parameters by independently stacking designed basic units, without requiring additional judgement criteria anymore. Given the basic units of various architectures, they are combined and stacked in a certain form to build up compact neural networks. We formulate TissueNet in diverse popular backbones for comparison with the state-of-the-art pruning methods on different benchmark datasets. Moreover, two new metrics are proposed to evaluate compression performance. Experiment results show that TissueNet can achieve comparable classification accuracy while saving up to around 80% FLOPs and 89.7% parameters. That is, stacking basic units provides a new promising way for network compression.