Spectral pruning of fully connected layers: ranking the nodes based on the eigenvalues

TL;DR

This paper introduces a spectral space approach to neural network pruning, using eigenvalues to rank node importance, enabling effective pruning strategies that significantly reduce network size without performance loss.

Contribution

It proposes a novel spectral method for ranking nodes in neural networks based on eigenvalues, facilitating efficient pruning with minimal accuracy impact.

Findings

Eigenvalue-based ranking effectively identifies important nodes.

Spectral pruning achieves high compression with unchanged performance.

Applicable across various architectures and tasks.

Abstract

Training of neural networks can be reformulated in spectral space, by allowing eigenvalues and eigenvectors of the network to act as target of the optimization instead of the individual weights. Working in this setting, we show that the eigenvalues can be used to rank the nodes' importance within the ensemble. Indeed, we will prove that sorting the nodes based on their associated eigenvalues, enables effective pre- and post-processing pruning strategies to yield massively compacted networks (in terms of the number of composing neurons) with virtually unchanged performance. The proposed methods are tested for different architectures, with just a single or multiple hidden layers, and against distinct classification tasks of general interest.