Learning In Practice: Reasoning About Quantization

TL;DR

This paper introduces a framework for understanding learning algorithms under various quantizations, proving convergence for certain algorithms and empirically analyzing the effects of quantization across datasets.

Contribution

It formalizes reasoning about quantized learning, proving convergence of quantization-aware algorithms and providing extensive empirical insights.

Findings

Quantization-aware Perceptron and Frank-Wolfe algorithms converge.

Quantization impacts model performance variably across datasets.

The framework bridges theoretical analysis and practical quantization effects.

Abstract

There is a mismatch between the standard theoretical analyses of statistical machine learning and how learning is used in practice. The foundational assumption supporting the theory is that we can represent features and models using real-valued parameters. In practice, however, we do not use real numbers at any point during training or deployment. Instead, we rely on discrete and finite quantizations of the reals, typically floating points. In this paper, we propose a framework for reasoning about learning under arbitrary quantizations. Using this formalization, we prove the convergence of quantization-aware versions of the Perceptron and Frank-Wolfe algorithms. Finally, we report the results of an extensive empirical study of the impact of quantization using a broad spectrum of datasets.