Loss-Complexity Landscape and Model Structure Functions

TL;DR

This paper introduces a duality framework linking the Kolmogorov structure function with statistical mechanics concepts, revealing phase transitions in model complexity and generalization through theoretical analysis and practical experiments.

Contribution

It develops a novel dualization framework for the Kolmogorov structure function using computable proxies and establishes a mathematical analogy with statistical mechanics.

Findings

Loss-complexity trade-offs exhibit phase transitions.

Model complexity correlates with generalization and overfitting thresholds.

Theoretical predictions are validated with linear and tree-based regression models.

Abstract

We develop a framework for dualizing the Kolmogorov structure function $h_x(\alpha)$, which then allows using computable complexity proxies. We establish a mathematical analogy between information-theoretic constructs and statistical mechanics, introducing a suitable partition function and free energy functional. We explicitly prove the Legendre-Fenchel duality between the structure function and free energy, showing detailed balance of the Metropolis kernel, and interpret acceptance probabilities as information-theoretic scattering amplitudes. A susceptibility-like variance of model complexity is shown to peak precisely at loss-complexity trade-offs interpreted as phase transitions. Practical experiments with linear and tree-based regression models verify these theoretical predictions, explicitly demonstrating the interplay between the model complexity, generalization, and overfitting threshold.