Quantization through Piecewise-Affine Regularization: Optimization and Statistical Guarantees

TL;DR

This paper introduces Piecewise-Affine Regularization (PAR) for quantization in supervised learning, providing optimization methods and statistical guarantees for quantized solutions, especially in overparameterized models.

Contribution

It develops a theoretical framework for PAR, derives closed-form proximal mappings, and demonstrates statistical guarantees comparable to classical regularizations.

Findings

Critical points in overparameterized PAR are highly quantized.

Closed-form proximal mappings for various PARs are derived.

PAR can approximate classical regularizations with similar statistical guarantees.

Abstract

Optimization problems over discrete or quantized variables are very challenging in general due to the combinatorial nature of their search space. Piecewise-affine regularization (PAR) provides a flexible modeling and computational framework for quantization based on continuous optimization. In this work, we focus on the setting of supervised learning and investigate the theoretical foundations of PAR from optimization and statistical perspectives. First, we show that in the overparameterized regime, where the number of parameters exceeds the number of samples, every critical point of the PAR-regularized loss function exhibits a high degree of quantization. Second, we derive closed-form proximal mappings for various (convex, quasi-convex, and non-convex) PARs and show how to solve PAR-regularized problems using the proximal gradient method, its accelerated variant, and the Alternating Direction Method of Multipliers. Third, we study statistical guarantees of PAR-regularized linear regression problems; specifically, we can approximate classical formulations of $\ell_1$-, squared $\ell_2$-, and nonconvex regularizations using PAR and obtain similar statistical guarantees with quantized solutions.