Binarized Neural Networks Converge Toward Algorithmic Simplicity: Empirical Support for the Learning-as-Compression Hypothesis

TL;DR

This paper provides empirical evidence that training Binarized Neural Networks aligns with the concept of learning as a process of algorithmic compression, using algorithmic complexity measures to better understand neural network dynamics.

Contribution

It introduces a novel application of algorithmic information theory, specifically the Block Decomposition Method, to analyze neural network training and supports the learning-as-compression hypothesis.

Findings

BDM correlates more strongly with training loss than entropy.

Training involves progressive internalization of structured regularities.

Supports viewing learning as algorithmic compression.

Abstract

Understanding and controlling the informational complexity of neural networks is a central challenge in machine learning, with implications for generalization, optimization, and model capacity. While most approaches rely on entropy-based loss functions and statistical metrics, these measures often fail to capture deeper, causally relevant algorithmic regularities embedded in network structure. We propose a shift toward algorithmic information theory, using Binarized Neural Networks (BNNs) as a first proxy. Grounded in algorithmic probability (AP) and the universal distribution it defines, our approach characterizes learning dynamics through a formal, causally grounded lens. We apply the Block Decomposition Method (BDM) -- a scalable approximation of algorithmic complexity based on AP -- and demonstrate that it more closely tracks structural changes during training than entropy, consistently exhibiting stronger correlations with training loss across varying model sizes and randomized training runs. These results support the view of training as a process of algorithmic compression, where learning corresponds to the progressive internalization of structured regularities. In doing so, our work offers a principled estimate of learning progression and suggests a framework for complexity-aware learning and regularization, grounded in first principles from information theory, complexity, and computability.