A General Error-Theoretical Analysis Framework for Constructing Compression Strategies

TL;DR

This paper introduces a theoretical framework for optimizing layer-wise compression in deep models, using geometric error analysis to minimize performance loss while maximizing compression.

Contribution

It proposes a novel Compression Error Theory (CET) framework that leverages algebraic geometry to determine optimal compression levels across layers.

Findings

Achieves nearly 11× parameter compression on ResNet-34.

Maintains or surpasses original model performance after compression.

Provides a geometric interpretation of quantization error for better optimization.

Abstract

The exponential growth in parameter size and computational complexity of deep models poses significant challenges for efficient deployment. The core problem of existing compression methods is that different layers of the model have significant differences in their tolerance to compression levels. For instance, the first layer of a model can typically sustain a higher compression level compared to the last layer without compromising performance. Thus, the key challenge lies in how to allocate compression levels across layers in a way that minimizes performance loss while maximizing parameter reduction. To address this challenge, we propose a Compression Error Theory (CET) framework, designed to determine the optimal compression level for each layer. Taking quantization as an example, CET leverages differential expansion and algebraic geometry to reconstruct the quadratic form of quantization error as ellipsoids and hyperbolic paraboloids, and utilizes their geometric structures to define an error subspace. To identify the error subspace with minimal performance loss, by performing orthogonal decomposition of the geometric space, CET transforms the optimization process of the error subspace into a complementary problem. The final theoretical analysis shows that constructing the quantization subspace along the major axis results in minimal performance degradation. Through experimental verification of the theory, CET can greatly retain performance while compressing. Specifically, on the ResNet-34 model, CET achieves nearly 11$\times$ parameter compression while even surpassing performance comparable to the original model.