Hankel Singular Value Regularization for Highly Compressible State Space Models

TL;DR

This paper introduces Hankel singular value regularization for state space models in neural networks, enabling highly compressible models that retain accuracy, by efficiently computing singular values during training.

Contribution

It proposes a scalable algorithm for regularizing Hankel singular values in state space models, improving compressibility without sacrificing performance.

Findings

Regularized models are up to 10× more compressible.

Maintains high accuracy on Long Range Arena benchmarks.

Efficient computation exploits block-diagonal structure.

Abstract

Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10$\times$ more compressible than standard state space layers while maintaining high accuracy.