Neural Network Pruning as Spectrum Preserving Process

TL;DR

This paper presents a spectrum-preserving perspective on neural network pruning, linking it to matrix spectrum learning, and proposes a new sparsification algorithm to improve lightweight neural network deployment.

Contribution

It introduces a theoretical foundation connecting matrix spectrum preservation with neural network pruning and develops a tailored sparsification algorithm for better pruning results.

Findings

Spectrum preservation is crucial for effective pruning.

The proposed algorithm outperforms existing pruning methods.

Enhanced interpretability of neural networks through spectrum analysis.

Abstract

Neural networks have achieved remarkable performance in various application domains. Nevertheless, a large number of weights in pre-trained deep neural networks prohibit them from being deployed on smartphones and embedded systems. It is highly desirable to obtain lightweight versions of neural networks for inference in edge devices. Many cost-effective approaches were proposed to prune dense and convolutional layers that are common in deep neural networks and dominant in the parameter space. However, a unified theoretical foundation for the problem mostly is missing. In this paper, we identify the close connection between matrix spectrum learning and neural network training for dense and convolutional layers and argue that weight pruning is essentially a matrix sparsification process to preserve the spectrum. Based on the analysis, we also propose a matrix sparsification algorithm tailored for neural network pruning that yields better pruning result. We carefully design and conduct experiments to support our arguments. Hence we provide a consolidated viewpoint for neural network pruning and enhance the interpretability of deep neural networks by identifying and preserving the critical neural weights.