Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks

TL;DR

This paper provides a theoretical framework for data-driven low-rank compression of deep neural networks, explaining why such methods outperform data-agnostic approaches and offering guarantees for maintaining accuracy after compression.

Contribution

It develops recovery theorems under weak assumptions, offering the first theoretical guarantees for data-driven low-rank neural network compression.

Findings

Proves three recovery theorems for low-rank approximation

Shows data-driven methods outperform data-agnostic approaches

Provides theoretical guarantees for maintaining accuracy

Abstract

Deep neural networks have achieved state-of-the-art performance across numerous applications, but their high memory and computational demands present significant challenges, particularly in resource-constrained environments. Model compression techniques, such as low-rank approximation, offer a promising solution by reducing the size and complexity of these networks while only minimally sacrificing accuracy. In this paper, we develop an analytical framework for data-driven post-training low-rank compression. We prove three recovery theorems under progressively weaker assumptions about the approximate low-rank structure of activations, modeling deviations via noise. Our results represent a step toward explaining why data-driven low-rank compression methods outperform data-agnostic approaches and towards theoretically grounded compression algorithms that reduce inference costs while maintaining performance.