Spectral Pruning for Recurrent Neural Networks

TL;DR

This paper introduces a spectral pruning algorithm for RNNs, providing theoretical generalization bounds and demonstrating its effectiveness through numerical experiments, aiming to reduce computational costs.

Contribution

It proposes a novel spectral pruning method for RNNs with theoretical generalization error bounds and empirical validation, advancing pruning techniques beyond heuristics.

Findings

Spectral pruning effectively reduces RNN parameters.

Theoretical generalization error bounds are established.

Numerical experiments confirm improved efficiency.

Abstract

Recurrent neural networks (RNNs) are a class of neural networks used in sequential tasks. However, in general, RNNs have a large number of parameters and involve enormous computational costs by repeating the recurrent structures in many time steps. As a method to overcome this difficulty, RNN pruning has attracted increasing attention in recent years, and it brings us benefits in terms of the reduction of computational cost as the time step progresses. However, most existing methods of RNN pruning are heuristic. The purpose of this paper is to study the theoretical scheme for RNN pruning method. We propose an appropriate pruning algorithm for RNNs inspired by "spectral pruning", and provide the generalization error bounds for compressed RNNs. We also provide numerical experiments to demonstrate our theoretical results and show the effectiveness of our pruning method compared with existing methods.