Neural Weight Norm = Kolmogorov Complexity

TL;DR

This paper establishes a theoretical link between neural network weight norms in fixed-precision regimes and Kolmogorov complexity, explaining why weight decay favors simpler, more compressible functions.

Contribution

It proves that the minimal weight norm of a fixed-precision neural network outputting a binary string equals the string's Kolmogorov complexity, up to a logarithmic factor.

Findings

Weight decay induces a prior matching Solomonoff's universal prior.

In fixed precision, all weight norms are proportional to the number of non-zero parameters.

The bounds are tight up to constants, with permutation encodings realizing the logarithmic factor.

Abstract

Why does weight decay work? We prove that, in any fixed-precision regime, the smallest weight norm of a looped neural network outputting a binary string equals the Kolmogorov complexity of that string, up to a logarithmic factor. This implies that weight decay induces a prior matching Solomonoff's universal prior, the optimal prior over computable functions, up to a polynomial factor. The result is norm-agnostic: in fixed precision, every weight norm collapses to the non-zero parameter count up to constants, so the same sandwich bound holds for any norm used as a regulariser. The proof has two short reductions: any program for a universal Turing machine can be encoded into neural weights at unit cost per program bit, and any fixed-precision network can be described by enumerating its non-zero parameters with logarithmic addressing overhead. Both bounds are tight up to constants, with the logarithmic factor realised by permutation encodings: a network whose parameters encode a permutation produces a string whose Kolmogorov complexity is the non-zero parameter count times its logarithm. The fixed-precision assumption is essential: with infinite precision, neural networks can encode non-computable functions and the weight norm loses its relevance.