Structural Pruning in Deep Neural Networks: A Small-World Approach

TL;DR

This paper introduces a novel structural pruning method inspired by brain network properties, which reduces model size and interconnection costs in deep neural networks by pre-training network trimming into a Small-World structure.

Contribution

The paper proposes a hierarchical pruning scheme that creates a Small-World network structure before training, improving efficiency and reducing parameters significantly.

Findings

Reduces parameters to 2.3% of baseline on LeNet-5 for MNIST.

Decreases parameters to 9.02% of baseline on VGG-16 for CIFAR-10.

Achieves a locally clustered and globally sparse network structure.

Abstract

Deep Neural Networks (DNNs) are usually over-parameterized, causing excessive memory and interconnection cost on the hardware platform. Existing pruning approaches remove secondary parameters at the end of training to reduce the model size; but without exploiting the intrinsic network property, they still require the full interconnection to prepare the network. Inspired by the observation that brain networks follow the Small-World model, we propose a novel structural pruning scheme, which includes (1) hierarchically trimming the network into a Small-World model before training, (2) training the network for a given dataset, and (3) optimizing the network for accuracy. The new scheme effectively reduces both the model size and the interconnection needed before training, achieving a locally clustered and globally sparse model. We demonstrate our approach on LeNet-5 for MNIST and VGG-16 for CIFAR-10, decreasing the number of parameters to 2.3% and 9.02% of the baseline model, respectively.