Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory

TL;DR

This paper explores how high-capacity kernel associative memories maintain robustness under quantization by using dense representations of sparse functions, offering insights for hardware-efficient neural systems.

Contribution

It introduces a geometric interpretation of KLR networks, demonstrating robustness to quantization and sensitivity to pruning, based on a sparse-to-dense representation principle.

Findings

Networks are robust under low-precision quantization.

Networks are sensitive to pruning.

Sparse input mappings are implemented via dense bimodal parameters.

Abstract

High-capacity associative memories based on Kernel Logistic Regression (KLR) achieve strong retrieval performance but typically require substantial computational resources. This paper investigates the compressibility of KLR Hopfield networks to clarify the geometric principles underlying their robust representations. We present a geometric interpretation based on spontaneous symmetry breaking and Walsh analysis, and examine it through compression experiments involving quantization and pruning. The experiments reveal a clear asymmetry: the network remains robust under low-precision quantization while exhibiting strong sensitivity to pruning. We interpret this behavior through a "sparse function, dense representation" principle, in which a sparse input mapping is implemented through a dense bimodal parameterization. These findings suggest a practical route toward hardware-efficient kernel associative memories and provide insight into the geometric principles underlying robust representation in neural systems.