CompSRT: Quantization and Pruning for Image Super Resolution Transformers

TL;DR

This paper introduces CompSRT, a novel compression method for image super resolution transformers that combines Hadamard-based quantization and pruning, significantly improving performance and reducing model size.

Contribution

The paper presents a new compression technique for SR transformers that leverages Hadamard transforms and scalar decomposition, outperforming existing methods in accuracy and efficiency.

Findings

Achieves up to 1.53 dB improvement over SOTA

Reduces blurriness in images at various bitwidths

Prunes 40% of weights with minimal performance loss

Abstract

Model compression has become an important tool for making image super resolution models more efficient. However, the gap between the best compressed models and the full precision model still remains large and a need for deeper understanding of compression theory on more performant models remains. Prior research on quantization of LLMs has shown that Hadamard transformations lead to weights and activations with reduced outliers, which leads to improved performance. We argue that while the Hadamard transform does reduce the effect of outliers, an empirical analysis on how the transform functions remains needed. By studying the distributions of weights and activations of SwinIR-light, we show with statistical analysis that lower errors is caused by the Hadamard transforms ability to reduce the ranges, and increase the proportion of values around $0$. Based on these findings, we introduce CompSRT, a more performant way to compress the image super resolution transformer network SwinIR-light. We perform Hadamard-based quantization, and we also perform scalar decomposition to introduce two additional trainable parameters. Our quantization performance statistically significantly surpasses the SOTA in metrics with gains as large as 1.53 dB, and visibly improves visual quality by reducing blurriness at all bitwidths. At $3$-$4$ bits, to show our method is compatible with pruning for increased compression, we also prune $40\%$ of weights and show that we can achieve $6.67$-$15\%$ reduction in bits per parameter with comparable performance to SOTA.