Extreme Model Compression with Structured Sparsity at Low Precision

TL;DR

This paper introduces SLOPE, a unified framework combining structured sparsity and low-precision quantization for neural networks, achieving significant size reduction with minimal accuracy loss.

Contribution

SLOPE is the first method to effectively integrate structured sparsity and low-bit quantization using a novel training regularization strategy.

Findings

Achieves ~20x model size reduction on ResNet-18 with 99% accuracy retention.

Outperforms existing methods in classification, detection, and segmentation tasks.

Applicable to various models like ResNet-18, ViT-Small, and Mask R-CNN.

Abstract

Deep neural networks (DNNs) are used in many applications, but their large size and high computational cost make them hard to run on devices with limited resources. Two widely used techniques to address this challenge are weight quantization, which lowers the precision of all weights, and structured sparsity, which removes unimportant weights while retaining the important ones at full precision. Although both are effective individually, they are typically studied in isolation due to their compounded negative impact on model accuracy when combined. In this work, we introduce SLOPE Structured Sparsity at Low Precision), a unified framework, to effectively combine structured sparsity and low-bit quantization in a principled way. We show that naively combining sparsity and quantization severely harms performance due to the compounded impact of both techniques. To address this, we propose a training-time regularization strategy that minimizes the discrepancy between full-precision weights and their sparse, quantized counterparts by promoting angular alignment rather than direct matching. On ResNet-18, SLOPE achieves $\sim20\times$ model size reduction while retaining $\sim$99% of the original accuracy. It consistently outperforms state-of-the-art quantization and structured sparsity methods across classification, detection, and segmentation tasks on models such as ResNet-18, ViT-Small, and Mask R-CNN.