Model compression as constrained optimization, with application to neural nets. Part I: general framework

TL;DR

This paper introduces a unified constrained optimization framework for neural network compression, encompassing various techniques like quantization and pruning, and proposes an efficient alternating algorithm to optimize compressed models.

Contribution

It formulates model compression as a general constrained optimization problem and develops a versatile learning-compression algorithm applicable across different compression methods.

Findings

Algorithm guarantees local optimality under standard assumptions.

Framework unifies multiple compression techniques within a single optimization approach.

Efficient alternating optimization method demonstrated for neural network compression.

Abstract

Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quantization, low-rank decomposition, pruning, lossless compression and others. Then, we give a general algorithm to optimize this nonconvex problem based on the augmented Lagrangian and alternating optimization. This results in a "learning-compression" algorithm, which alternates a learning step of the uncompressed model, independent of the compression type, with a compression step of the model parameters, independent of the learning task. This simple, efficient algorithm is guaranteed to find the best compressed model for the task in a local sense under standard assumptions. We present separately in several companion papers the development of this general framework into specific algorithms for model compression based on quantization, pruning and other variations, including experimental results on compressing neural nets and other models.