Provable Quantization with Randomized Hadamard Transform

TL;DR

This paper introduces a provable, efficient quantization method using a randomized Hadamard transform combined with dithering, achieving near-optimal error bounds with reduced computational cost.

Contribution

It provides the first rigorous analysis of dithered quantization with the Hadamard transform, matching the performance of random rotations.

Findings

Achieves mean squared error close to that of truly random rotations.

Reduces computational complexity from quadratic to near-linear in dimension.

Provides theoretical guarantees for a variant of TurboQuant with Hadamard transform.

Abstract

Vector quantization via random projection followed by scalar quantization is a fundamental primitive in machine learning, with applications ranging from similarity search to federated learning and KV cache compression. While dense random rotations yield clean theoretical guarantees, they require $\Theta(d^2)$ time. The randomized Hadamard transform $HD$ reduces this cost to $O(d \log d)$, but its discrete structure complicates analysis and leads to weaker or purely empirical compression guarantees. In this work, we study a variant of this approach: dithered quantization with a single randomized Hadamard transform. Specifically, the quantizer applies $HD$ to the input vector and subtracts a random scalar offset before quantizing, injecting additional randomness at negligible cost. We prove that this approach is unbiased and provides mean squared error bounds that asymptotically match those achievable with truly random rotation matrices. In particular, we prove that a dithered version of TurboQuant achieves mean squared error $\bigl(\pi\sqrt{3}/2 + o(1)\bigr) \cdot 4^{-b}$ at $b$ bits per coordinate, where the $o(1)$ term vanishes uniformly over all unit vectors and all dimensions as the number of quantization levels grows.