Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks

TL;DR

This paper introduces a novel method using a switcher neural network to automatically learn sparse neural network structures by exploring the knowledge in the weights, improving efficiency and performance.

Contribution

A new approach employing a switcher neural network to optimize neural network sparsity by leveraging weight information, learned alternately with the task network.

Findings

Achieves sparse, well-performing fully connected layers.

Effective across various network architectures and datasets.

Stable convergence to optimal sparse structures.

Abstract

Sparse neural networks are important for achieving better generalization and enhancing computation efficiency. This paper proposes a novel learning approach to obtain sparse fully connected layers in neural networks (NNs) automatically. We design a switcher neural network (SNN) to optimize the structure of the task neural network (TNN). The SNN takes the weights of the TNN as the inputs and its outputs are used to switch the connections of TNN. In this way, the knowledge contained in the weights of TNN is explored to determine the importance of each connection and the structure of TNN consequently. The SNN and TNN are learned alternately with stochastic gradient descent (SGD) optimization, targeting at a common objective. After learning, we achieve the optimal structure and the optimal parameters of the TNN simultaneously. In order to evaluate the proposed approach, we conduct image classification experiments on various network structures and datasets. The network structures include LeNet, ResNet18, ResNet34, VggNet16 and MobileNet. The datasets include MNIST, CIFAR10 and CIFAR100. The experimental results show that our approach can stably lead to sparse and well-performing fully connected layers in NNs.