GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks

TL;DR

GRAIL is a post-hoc linear reconstruction method that restores accuracy in compressed neural networks without additional training, using a small calibration set to improve performance across various models.

Contribution

It introduces a simple, data-aware, zero-finetuning post-compression technique that enhances accuracy of compressed models by linearly reconstructing hidden representations.

Findings

Consistently improves accuracy or perplexity across ResNets, ViTs, and LLMs.

Requires only a few forward passes without gradients or labels.

Operates with manageable computational overhead.

Abstract

Structured deep model compression methods are hardware-friendly and substantially reduce memory and inference costs. However, under aggressive compression, the resulting accuracy degradation often necessitates post-compression finetuning, which can be impractical due to missing labeled data or high training cost. We propose post-hoc blockwise compensation, called GRAIL, a simple zero-finetuning step applied after model compression that restores each block's input-output behavior using a small calibration set. The method summarizes hidden activations via a Gram matrix and applies ridge regression to linearly reconstruct the original hidden representation from the reduced one. The resulting reconstruction map is absorbed into the downstream projection weights, while the upstream layer is compressed. The approach is selector-agnostic (Magnitude, Wanda, Gram-based selection, or folding), data-aware (requiring only a few forward passes without gradients or labels), and recovers classic pruning or folding when the Gram matrix is near identity, indicating weak inter-channel correlations. Across ResNets, ViTs, and decoder-only LLMs, GRAIL consistently improves accuracy or perplexity over data-free and data-aware pruning or folding baselines in practical compression regimes, with manageable overhead and no backpropagation. The code is available at https://github.com/TWWinde/GRAIL_Compensation.